Video decoding method, computer device, equipment and storage medium

Abstract

Aspects of the present disclosure provide an apparatus and method including a processing circuit for video decoding. The processing circuit determines a transform candidate for a block from a grouped transform set based on one of a feature vector or feature scalar S extracted from reconstructed samples in one or more neighboring blocks of the block of a current picture. Each transform set in the grouped transform set may include one or more transform candidates for the block. The one or more neighboring blocks may be in the current picture or in a reconstructed picture different from the current picture. The processing circuit reconstructs samples of the block based on the determined transform candidates. The processing circuit may select a transform set subset from the grouped transform set based on a prediction mode of the block indicated in encoded information of the block, and determine a transform candidate from the transform set subset based on the reconstructed samples.

Description

[0001] Reference Merge

[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 490,967, filed on September 30, 2021, entitled “METHOD AND APPARATUS FOR VIDEO CODING,” which claims the benefit of priority to U.S. Provisional Application No. 63 / 130,249, filed on December 23, 2020, entitled “Feature based transform selection.” The entire disclosure of the prior application is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to video encoding / decoding embodiments, and more particularly to a video decoding method, a computer device, a video decoding apparatus, and a non-transitory computer-readable storage medium. Background Art

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent that the work is described in this background section, and in all aspects of the specification that may not constitute prior art at the time of filing, is neither explicitly nor implicitly admitted to be prior art to the present disclosure.

[0005] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having a spatial dimension of, for example, 1920×1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (also informally referred to as a frame rate) of, for example, 60 pictures per second or 60Hz. Uncompressed video has specific bit rate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a 60Hz frame rate) requires close to 1.5Gbit / s bandwidth. One hour of such video requires more than 600 GBytes of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression and their combinations can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be the same as the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely used. The amount of distortion allowed depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that higher allowable / tolerable distortion can produce higher compression ratios.

[0007] Video encoders and decoders may utilize techniques from several broad categories including, for example, motion compensation, transforms, quantization, and entropy coding.

[0008] Video coding techniques may include techniques known as intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture may be an intra-frame picture. Intra-frame pictures and their derivatives (e.g., independent decoder refresh pictures) may be used to reset decoder states, and may therefore be used as the first picture in an encoded video bitstream and video session, or as a still picture. Samples of intra-frame blocks may be subjected to transformation, and transform coefficients may be quantized before entropy coding. Intra-frame prediction may be a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transformation, and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0009] Conventional intra-frame coding, such as that known from, for example, MPEG-2 generation coding techniques, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to use surrounding sample data and / or metadata obtained during encoding / decoding of, for example, spatially adjacent and preceding data blocks in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, not from reference pictures.

[0010] There can be many different forms of intra-frame prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be encoded in an intra-frame prediction mode. In some cases, a mode can have sub-modes and / or parameters, and these sub-modes and / or parameters can be encoded separately or included in a mode codeword. Which codewords to use for a given mode / sub-mode / parameter combination may have an impact on the coding efficiency gain through intra-frame prediction, and so does the entropy coding technique used to convert the codewords into a bitstream. The prior art lacks video coding techniques related to a transform set or transform kernel selection scheme based on reconstructed samples of one or more neighboring blocks of a block (e.g., feature indicators (e.g., feature vectors or feature scalars of reconstructed samples of one or more neighboring blocks of a block)). Summary of the invention

[0011] According to an embodiment, a method for video decoding in a decoder is provided. The method comprises: determining a transform candidate for the block from a group of transform sets based on one of a feature vector or a feature scalar S extracted from reconstructed samples in one or more neighboring blocks of the block in a current picture, each transform set in the group of transform sets comprising one or more transform candidates for the block, one or more neighboring blocks in the current picture or in a reconstructed picture different from the current picture; and reconstructing samples of the block based on the determined transform candidates.

[0012] According to an embodiment, a computer device is provided, which includes: one or more computer-readable non-transitory storage media configured to store computer program codes; and one or more computer processors configured to access the computer program codes and execute the above-mentioned method for video decoding in a decoder according to the instructions of the computer program codes.

[0013] According to an embodiment, a video decoding device is provided, comprising a processing circuit configured to perform the above-mentioned method for video decoding in a decoder.

[0014] According to an embodiment, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores a computer program code, and when the computer program code is executed by at least one processor, the at least one processor performs the above-mentioned method for video decoding in a decoder.

[0015] The video decoding method, computer device, video decoding apparatus and non-transitory computer-readable storage medium of the present invention provide a video encoding technology related to a transform set or transform kernel selection scheme based on reconstructed samples of one or more neighboring blocks of a block (e.g., feature indicators (e.g., feature vectors or feature scalars of reconstructed samples of one or more neighboring blocks of a block)). BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:

[0017] Figure 1A is a schematic illustration of an exemplary subset of intra prediction modes;

[0018] Figure 1B is a diagram of exemplary intra prediction directions;

[0019] Figure 2 is a schematic illustration of a current block and its surrounding spatial merging candidates in an example;

[0020] Figure 3 is a schematic illustration of a simplified block diagram of a communication system according to an embodiment;

[0021] Figure 4 is a schematic illustration of a simplified block diagram of a communication system according to another embodiment;

[0022] Figure 5 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment;

[0023] Figure 6 is a schematic illustration of a simplified block diagram of an encoder according to an embodiment;

[0024] Figure 7 shows a block diagram of an encoder according to another embodiment;

[0025] Figure 8 shows a block diagram of a decoder according to another embodiment;

[0026] Fig. 9 An example of a nominal mode for encoding a block according to an embodiment of the present disclosure is shown.

[0027] Fig.10 An example of non-directional smooth intra prediction according to aspects of the present disclosure is shown.

[0028] Fig.11 An example of a recursive filtering based intra predictor according to an embodiment of the present disclosure is shown.

[0029] Fig.12 An example of multiple reference lines for encoding a block according to an embodiment of the present disclosure is shown.

[0030] Fig.13 Examples of primary transform basis functions according to embodiments of the present disclosure are shown.

[0031] Fig.14A An exemplary correlation of the availability of various transform kernels based on transform block size and prediction mode according to an embodiment of the present disclosure is shown.

[0032] Fig. 14B An exemplary transform type selection based on intra prediction mode according to an embodiment of the present disclosure is shown.

[0033] Figure 15 to Figure 16 An example of two transform encoding processes using 16x64 transform and 16x48 transform, respectively, according to an embodiment of the present disclosure is shown.

[0034] FIG. 17A to FIG. 17D An exemplary residual pattern (in grayscale) observed for intra prediction mode is shown according to an embodiment of the present disclosure.

[0035] Fig.18 Exemplary spatial neighboring samples of a block according to an embodiment of the present disclosure are shown.

[0036] Fig.19 A flowchart outlining a process according to an implementation of the present disclosure is shown.

[0037] Fig. 20 is a schematic illustration of a computer system according to an embodiment. DETAILED DESCRIPTION

[0038] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further improved in newer coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). The predictor block can be formed using neighboring sample values ​​belonging to samples that are already available. Sample values ​​of neighboring samples are copied into the predictor block according to the direction. A reference to the used direction can be encoded in the bitstream or can be predicted by itself.

[0039] Reference Figure 1A, a subset of nine known predictor directions from the 33 possible predictor directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes) is depicted at the bottom right. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees to the horizontal.

[0040] Still refer to Figure 1A , at the top left, a block (104) of 4×4 samples is depicted (indicated by the dashed bold line). Block (104) includes 16 samples, each of which is labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the Y dimension and the X dimension in block (104). Since the size of the block is 4×4 samples, S44 is at the bottom right. Reference samples are further shown that follow a similar numbering scheme. The reference samples are labeled with R, their Y position (e.g., row index), and X position (column index) relative to block (104). In both H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; therefore, there is no need to use negative values.

[0041] Intra-picture prediction may be performed by copying reference sample values ​​from neighboring samples delimited by a signaled prediction direction. For example, assume that the encoded video bitstream includes the following signaling: For this block, the signaling indicates a prediction direction consistent with arrow (102), that is, the sample is predicted from one or more prediction samples to the upper right at a 45 degree angle to the horizontal. In this case, sample S41, sample S32, sample S23, and sample S14 are predicted from the same reference sample R05. Sample S44 is then predicted from reference sample R08.

[0042] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to compute the reference sample; in particular when the directions do not divide evenly at 45 degrees.

[0043] As video coding technology has developed, the number of possible directions has also increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and when disclosed JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those possible directions with a small number of bits, thereby accepting certain penalties for less likely directions. In addition, the direction itself can sometimes be predicted from neighboring directions used in adjacent decoded blocks.

[0044] Figure 1B A schematic diagram (180) depicting 65 intra prediction directions according to JEM is shown to illustrate the increase in the number of prediction directions over time.

[0045] The mapping of the intra-prediction direction bits representing directions in the coded video bitstream can vary with different video coding techniques; and can range, for example, from a simple direct mapping of prediction directions to intra-prediction modes, to codewords, to complex adaptive schemes involving most probable modes, and similar techniques. However, in all cases, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in well-working video coding techniques, those less likely directions will be represented by more bits than more likely directions.

[0046] Motion compensation may be a lossy compression technique, and may involve a technique where a block of sample data from a previously reconstructed picture or portion thereof (reference picture) is used to predict a newly reconstructed picture or picture portion after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third dimension being an indication of the reference picture in use (the latter may indirectly be a temporal dimension).

[0047] In some video compression techniques, the MV applicable to a region of sample data can be predicted from other MVs, such as an MV associated with another region of sample data that is spatially adjacent to the region being reconstructed and precedes the MV in decoding order. Doing so can significantly reduce the amount of data required to encode the MV, thereby eliminating redundancy and increasing compression. MV prediction can work effectively, for example, because when encoding an input video signal obtained from a camera (called natural video), there is a statistical probability that a larger region than the region to which a single MV applies moves in a similar direction, and therefore in some cases similar motion vectors obtained from MVs of adjacent regions can be used to predict. This results in the MV found for a given region being similar or identical to the MV predicted from the surrounding MVs, and in turn can be represented after entropy coding with fewer bits than would be used to directly encode the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example, because of rounding errors when calculating the predictor from several surrounding MVs.

[0048] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, this article describes a technique that is hereinafter referred to as "spatial merging".

[0049] Reference Figure 2 , the current block (201) includes samples that were found by the encoder during a motion search process to be predictable from a previous block of the same size that has been spatially shifted. Instead of encoding the MV directly, the MV associated with any of the five surrounding samples denoted A0, A1 and B0, B1, B2 (202 to 206, respectively) may be derived from metadata associated with one or more reference pictures, such as from the most recent (in decoding order) reference picture. In H.265, MV prediction may use a predictor from the same reference picture being used by a neighboring block.

[0050] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3In the example of , a first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via a network (350). The encoded video data can be sent in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to recover the video picture, and display the video picture based on the recovered video data. Unidirectional data transmission may be common in media service applications and the like.

[0051] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data that may occur, for example, during a video conference. For the bidirectional transmission of data, in the example, each of the terminal devices (330) and (340) can encode video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), and can decode the encoded video data to restore the video pictures, and can display the video pictures at an accessible display device based on the restored video data.

[0052] exist Figure 3 In the example of , terminal devices (310), (320), (330) and (40) may be illustrated as servers, personal computers and smart phones, but the principles of the present disclosure may not be limited to this. Implementations of the present disclosure may be applied to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. Communication network (350) may exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purposes of the present discussion, unless otherwise described herein below, the architecture and topology of network (350) may be unimportant to the operation of the present disclosure.

[0053] Figure 4The arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter is shown. The disclosed subject matter can also be applied to other applications with video functions, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0054] The streaming system may include a capture subsystem (413) that may include a video source (401), such as a digital camera, that creates, for example, an uncompressed video picture stream (402). In an example, the video picture stream (402) includes samples captured by the digital camera. The video picture stream (402), depicted as thick lines to emphasize the high amount of data when compared to the encoded video data (404) (or encoded video bitstream), can be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to implement or realize various aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as thin lines to emphasize the lower amount of data when compared to the video picture stream (402), can be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4 The client subsystems (406) and (408) in the video transmission system can access the streaming server (405) to retrieve the copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and creates an output stream of video pictures (411) that can be presented on a display (412) (e.g., a display screen) or other presentation device (not depicted). In some streaming transmission systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In the example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC.

[0055] Note that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0056] Figure 5A block diagram of a video decoder (510) according to an embodiment of the present disclosure is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) is shown in an example.

[0057] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data. The receiver (531) can receive the encoded video data with other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective consuming entities (not depicted). The receiver (531) can separate the encoded video sequence from the other data. To combat network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other embodiments, the buffer memory can be external to the video decoder (510) (not depicted). In other embodiments, there may be a buffer memory (not depicted) external to the video decoder (510), for example to combat network jitter, and in addition there may be another buffer memory (515) internal to the video decoder (510), for example to handle playout timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory may be very small. For use over a best-effort packet network such as the Internet, a buffer memory (515) may be required, which may be relatively large and may advantageously have an adaptive size, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0058] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510) and potentially for controlling a rendering device, such as a rendering device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5As shown. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI message) or a video usability information (VUI) parameter set fragment (not depicted). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be based on a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (Groups of Picture, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (520) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0059] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0060] Depending on the type of coded video picture or part thereof (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol (521) may involve a number of different units. Which units are involved and how they are involved can be controlled by sub-group control information parsed by the parser (520) from the coded video sequence. For clarity, the flow of such sub-group control information between the parser (520) and the following units is not depicted.

[0061] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated into each other. However, for the purpose of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0062] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients from the parser (520) as symbols (521) and control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) can output blocks including sample values ​​that can be input into an aggregator (555).

[0063] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks; that is, blocks that do not use predictive information from a previously reconstructed picture, but instead may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding, already reconstructed information obtained from a current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information already generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0064] In other cases, the output samples of the scaler / inverse transform unit (551) may relate to an inter-coded and possibly motion compensated block. In such a case, the motion compensated prediction unit (553) may access a reference picture memory (557) to obtain samples for prediction. After motion compensation of the obtained samples according to the signs (521) associated with the block, these samples may be added by an aggregator (555) to the output of the scaler / inverse transform unit (551), in this case referred to as residual samples or residual signals, to generate output sample information. The address within the reference picture memory (557) from which the motion compensated prediction unit (553) obtains the predicted samples may be controlled by a motion vector, which may be used by the motion compensated prediction unit (553) in the form of a sign (521) that may have, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory (557) when sub-sample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0065] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques that are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0066] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0067] Certain coded pictures, once fully reconstructed, can be used as reference pictures for future predictions. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

[0068] The video decoder (510) may perform decoding operations according to a predetermined video compression technique in a standard (e.g., ITU-T Recommendation H.265). The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence complies with both the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available according to the profile. It is also necessary for compliance that the complexity of the encoded video sequence is within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (e.g., measured in millions of samples per second), the maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0069] In an embodiment, a receiver (531) may receive additional (redundant) data with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0070] Figure 6 A block diagram of a video encoder (603) according to an embodiment of the present disclosure is shown. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (403) in an example.

[0071] The video encoder (603) can be used to obtain the video source (601) (which is not Figure 6 In an example, the video source (601) is a part of the electronic device (620) that receives the video samples, and the video source can capture the video images to be encoded by the video encoder (603). In another example, the video source (601) is a part of the electronic device (620).

[0072] The video source (601) can provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream, which can have any appropriate bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any appropriate sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) can be a storage device that stores pre-prepared videos. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. The video data can be provided as multiple individual pictures that are given motion when viewed sequentially. The picture itself can be organized as a spatial array of pixels, where each pixel can include one or more samples, depending on the sampling structure, color space, etc. in use. The relationship between pixels and samples can be easily understood by those skilled in the art. The following description focuses on samples.

[0073] According to an embodiment, the video encoder (603) can encode and compress the pictures of the source video sequence into a coded video sequence (643) in real time or under any other time constraints required by the application. Enforcing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to the other functional units. For the sake of clarity, the coupling is not depicted. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology...), picture size, picture group (group of picture, GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions related to the video encoder (603) optimized for a specific system design.

[0074] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As an oversimplified description, in an example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols such as a symbol stream based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner to what the (remote) decoder will also create (because any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream results in a bit-exact result that is independent of the decoder location (local or remote), the contents in the reference picture memory (634) are also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the exact same sample values ​​that the decoder will "see" as reference picture samples when using prediction during decoding. This basic principle of reference picture synchronization (and the drift that results if synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0075] The operation of the "local" decoder (633) may be identical to that of a "remote" decoder, which is a decoder that has been combined with Figure 5 The video decoder (510) is described in detail above. However, referring briefly to Figure 5 , since the symbols are available and encoding / decoding the symbols into an encoded video sequence can be lossless through the entropy encoder (645) and the parser (520), the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0076] It can be observed at this point that any decoder technology other than parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on the decoder operation. The description of encoder technology can be simplified because they are the opposite technology of the fully described decoder technology. A more detailed description is provided below only in certain areas.

[0077] During operation, in some examples, the source encoder (630) may perform motion compensated predictive encoding, which predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as “reference pictures.” In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

[0078] The local video decoder (633) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the video decoder ( Figure 6 When the encoded video data is decoded at a remote end (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture, and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content with the reconstructed reference picture that will be obtained by the remote video decoder (absent transmission errors).

[0079] The predictor (635) may perform a prediction search of the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may serve as appropriate prediction references for the new picture. The predictor (635) may operate on a sample block by pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references obtained from multiple reference pictures stored in the reference picture memory (634).

[0080] The controller (650) may manage encoding operations of the source encoder (630), including, for example, the setting of parameters and sub-group parameters for encoding video data.

[0081] The outputs of all the aforementioned functional units may be subjected to entropy coding in an entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0082] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which can be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0083] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a certain coded picture type to each coded picture, which can affect the coding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned one of the following picture types:

[0084] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of those variations of I pictures and their respective applications and features.

[0085] A predictive picture (P picture) may be a picture that is encoded and decoded using intra prediction or inter prediction, where intra prediction or inter prediction uses at most one motion vector and a reference index to predict sample values ​​of each block.

[0086] Bidirectional predictive pictures (B pictures) can be pictures that are encoded and decoded using intra prediction or inter prediction, where intra prediction or inter prediction uses up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0087] A source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Blocks of a P picture may be predictively coded with reference to one previously coded reference picture via spatial prediction or via temporal prediction. Blocks of a B picture may be predictively coded with reference to one or two previously coded reference pictures via spatial prediction or via temporal prediction.

[0088] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard (e.g., ITU-T Recommendation H.265). In its operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0089] In an embodiment, the transmitter (640) may send additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0090] Video may be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlations in a given picture, and inter-picture prediction exploits (temporal or other) correlations between pictures. In an example, a particular picture being encoded / decoded, referred to as the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a previously encoded and still buffered reference picture in the video, the block in the current picture may be encoded by a vector referred to as a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension identifying the reference picture when multiple reference pictures are used.

[0091] In some embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be in the past and future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be predicted by a combination of the first reference block and the second reference block.

[0092] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0093] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Each CTU can be recursively partitioned into one or more coding units (CUs) with a quadtree. For example, a CTU of 64×64 pixels can be partitioned into a CU of 64×64 pixels, or 4 CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In the example, each CU is analyzed to determine the prediction type of the CU, such as an inter-prediction type or an intra-prediction type. Depending on temporal and / or spatial predictability, the CU is partitioned into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) of pixels (e.g., 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.).

[0094] Figure 7 A diagram of a video encoder (703) according to another embodiment of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures, and encode the processed block into an encoded picture that is part of an encoded video sequence. In an example, the video encoder (703) is used instead of Figure 4 A video encoder (403) in an example.

[0095] In the HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block (e.g., a prediction block of 8×8 samples, etc.). The video encoder (703) determines, for example, using rate-distortion optimization, whether to best encode the processing block using intra mode, inter mode, or bidirectional prediction mode. When the processing block is to be encoded in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when the processing block is to be encoded in inter mode or bidirectional prediction mode, the video encoder (703) may encode the processing block into an encoded picture using inter prediction or bidirectional prediction techniques, respectively. In some video coding techniques, the merge mode may be an inter-picture prediction submode, in which motion vectors are obtained from one or more motion vector predictors without the aid of encoded motion vector components external to the predictor. In some other video coding techniques, there may be motion vector components applicable to the subject block. In the example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0096] exist Figure 7 In the example of FIG. 7 , the video encoder ( 703 ) includes Figure 7 An inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) are shown coupled together.

[0097] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., description of redundant information according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0098] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with an already encoded block in the same picture in some cases, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In an example, the intra encoder (722) also calculates an intra prediction result (e.g., a predicted block) based on a reference block in the same picture and the intra prediction information.

[0099] The general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select the intra prediction information and include the intra prediction information in the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select the inter prediction information and include the inter prediction information in the bitstream.

[0100] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate a transform coefficient. In an example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate a transform coefficient. Then, the transform coefficient is quantized to obtain a quantized transform coefficient. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the inter prediction information. In some examples, the decoded blocks are appropriately processed to generate decoded pictures, and the decoded pictures may be buffered in memory circuitry (not shown) and used as reference pictures.

[0101] The entropy encoder (725) is configured to format the bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information according to a suitable standard, such as the HEVC standard. In an example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Note that according to the disclosed subject matter, when the block is encoded in inter-frame mode or the merge sub-mode of the bidirectional prediction mode, there is no residual information.

[0102] Figure 8A diagram of a video decoder (810) according to another embodiment of the present disclosure is shown. The video decoder (810) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In an example, the video decoder (810) is used instead of Figure 4 A video decoder (410) in an example of FIG.

[0103] exist Figure 8 In the example of FIG. 8 , the video decoder ( 810 ) includes Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) are shown coupled together.

[0104] The entropy decoder (871) may be configured to reconstruct certain symbols representing syntax elements constituting the encoded picture from the encoded picture. Such symbols may include, for example, a mode for encoding a block (such as, intra mode, inter mode, bidirectional prediction mode, a merged sub-mode or another sub-mode of the latter two), prediction information (such as, intra prediction information or inter prediction information) of certain samples or metadata that can be identified for prediction by an intra decoder (872) or an inter decoder (880), for example, residual information in the form of quantized transform coefficients, etc. In an example, when the prediction mode is an inter mode or a bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0105] The inter-frame decoder (880) is configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0106] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0107] The residual decoder (873) is configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (Quantizer Parameter, QP)), and this information can be provided by the entropy decoder (871) (the data path is not indicated because this is only low-volume control information).

[0108] The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) and the prediction result (output by the inter-frame prediction module or the intra-frame prediction module as appropriate) in the spatial domain to form a reconstructed block, which can be part of a reconstructed picture, which can be part of a reconstructed video. It should be noted that other suitable operations, such as deblocking operations, etc., can be performed to improve visual quality.

[0109] Note that the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using any suitable technology. In an embodiment, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more processors that execute software instructions.

[0110] Video coding techniques related to a transform set or a transform kernel selection scheme based on reconstructed samples of one or more neighboring blocks of a block (e.g., feature indicators (e.g., feature vectors or feature scalars of reconstructed samples of one or more neighboring blocks of a block)) are disclosed. The video coding format may include an open video coding format designed for video transmission over the Internet, such as AOMedia Video 1 (AV1) or a next-generation AOMedia Video format that exceeds AV1. The video coding standard may include the High Efficiency Video Coding (HEVC) standard, the next-generation video coding that exceeds HEVC (e.g., Versatile Video Coding (VVC)), etc.

[0111] Various intra prediction modes may be used in intra prediction (e.g., AV1, VVC, etc.). In an embodiment, for example, in AV1, directional intra prediction is used. In directional intra prediction, prediction samples of a block may be generated by extrapolating from adjacent reconstructed samples along a direction. The direction corresponds to an angle. A mode used for directional intra prediction to predict prediction samples of a block may be referred to as a directional mode (also referred to as a directional prediction mode, a directional intra mode, a directional intra prediction mode, an angle mode). Each directional mode may correspond to a different angle or a different direction. In an example, for example, in the open video coding format VP9, ​​eight directional modes correspond to eight angles from 45° to 207°, such as Fig. 9As shown. The eight directional modes may also be referred to as nominal modes (e.g., V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED). In order to exploit a wider variety of spatial redundancies in directional textures (e.g., in AV1), the directional modes may be extended, for example, beyond the eight nominal modes to reach an angle set with finer granularity and more angles (or directions), such as Fig. 9 shown.

[0112] Fig. 9 An example of a nominal mode of a coding block (coding block, CB) (910) according to an embodiment of the present disclosure is shown. Certain angles (referred to as nominal angles) may correspond to nominal modes. In the example, eight nominal angles (or nominal intra-frame angles) (901) to (908) correspond to eight nominal modes (e.g., V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRE, D157_PRED, D203_PRED, and D67_PRED), respectively. The eight nominal angles (901) to (908) and the eight nominal modes may be referred to as V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, respectively. A nominal mode index may indicate a nominal mode (e.g., one of the eight nominal modes). In the example, a signal is used to indicate the nominal mode index.

[0113] In addition, each nominal angle may correspond to a plurality of finer angles (e.g., seven finer angles), and thus 56 angles (or prediction angles) or 56 directional modes (or angle modes, directional intra prediction modes) may be used, for example, in AV1. Each prediction angle may be represented by a nominal angle and an angle offset (or angle increment (δ)). The angle offset may be obtained by multiplying an offset integer I (e.g., -3, -2, -1, 0, 1, 2, or 3) by a step size (e.g., 3°). In an example, the prediction angle is equal to the sum of the nominal angle and the angle offset. In an example, for example, in AV1, a nominal mode (e.g., eight nominal modes (901) to (908)) and certain non-angular smoothing modes (e.g., five non-angular smoothing modes as described below, such as DC mode, PAETH mode, SMOOTH mode, vertical SMOOTH mode, and horizontal SMOOTH mode) may be signaled. Subsequently, if the current prediction mode is a directional mode (or angle mode), an index may be further signaled to indicate an angle offset (e.g., offset integer I) corresponding to the nominal angle. In an example, a directional mode (e.g., one of 56 directional modes) may be determined based on a nominal mode index and an index indicating an angular offset relative to the nominal mode. In an example, in order to implement the directional prediction mode in a generic manner, a unified directional predictor is used to implement the 56 directional modes used in AV1, for example, and the unified directional predictor may project each pixel to a reference sub-pixel position and interpolate the reference pixel through a 2-tap bilinear filter.

[0114] A non-directional smooth intra predictor (also referred to as a non-directional smooth intra prediction mode, a non-directional smooth mode, a non-angular smooth mode) may be used in intra prediction of a block (e.g., a CB). In some examples (e.g., in AV1), five non-directional smooth intra prediction modes include a DC mode or a DC predictor (e.g., DC), a PAETH mode or a PAETH predictor (e.g., PAETH), a SMOOTH mode or a SMOOTH predictor (e.g., SMOOTH), a vertical SMOOTH mode (referred to as a SMOOTH_V mode, a SMOOTH_V predictor, SMOOTH_V), and a horizontal SMOOTH mode (referred to as a SMOOTH_H mode, a SMOOTH_H predictor, or SMOOTH_H).

[0115] Fig.10An example of a non-directional smooth intra prediction mode (e.g., DC mode, PAETH mode, SMOOTH mode, SMOOTH_V mode, and SMOOTH_H mode) according to aspects of the present disclosure is shown. To predict a sample (1001) in a CB (1000) based on a DC predictor, an average of a first value of a left neighboring sample (1012) and a second value of an upper neighboring sample (or top neighboring sample) (1011) may be used as a predictor.

[0116] To predict sample (1001) based on the PAETH predictor, the first value of the left neighboring sample (1012), the second value of the top neighboring sample (1011), and the third value of the top left neighboring sample (1013) can be obtained. Then, the reference value is obtained using Equation 1.

[0117] Reference value = first value + second value - third value (Formula 1)

[0118] One of the first value, the second value, and the third value that is closest to the reference value may be set as a predicted value of the sample (1001).

[0119] SMOOTH_V mode, SMOOTH_H mode, and SMOOTH mode can predict CB (1000) using quadratic interpolation of the vertical direction, the horizontal direction, and the average value of the vertical direction and the horizontal direction, respectively. In order to predict sample (1001) based on the SMOOTH predictor, the average (e.g., weighted combination) of the first value, the second value, the value of the right sample (1014), and the value of the bottom sample (1016) can be used. In various examples, the right sample (1014) and the bottom sample (1016) are not reconstructed, and therefore, the value of the upper right adjacent sample (1015) and the value of the lower left adjacent sample (1017) can replace the values ​​of the right sample (1014) and the bottom sample (1016), respectively. Therefore, the average (e.g., weighted combination) of the first value, the second value, the value of the upper right adjacent sample (1015), and the value of the lower left adjacent sample (1017) can be used as the SMOOTH predictor. To predict sample (1001) based on the SMOOTH_V predictor, an average (e.g., a weighted combination) of the second value of the top neighboring sample (1011) and the value of the bottom left neighboring sample (1017) may be used. To predict sample (1001) based on the SMOOTH_H predictor, an average (e.g., a weighted combination) of the first value of the left neighboring sample (1012) and the value of the top right neighboring sample (1015) may be used.

[0120] Fig.11An example of a recursive filtering based intra predictor (also referred to as filter intra mode or recursive filtering mode) according to an embodiment of the present disclosure is shown. To capture the attenuated spatial correlation with reference objects on edges, the filter intra mode can be used for a CB

[0121] (1100). In the example, CB (1100) is a luma block. The luma block (1100) can be divided into a plurality of slices (e.g., eight (4×2) slices B0-B7). Each of the slices B0-B7 can have a plurality of adjacent samples. For example, slice B0 has seven adjacent samples (or seven adjacent elements) R00-R06, including four top adjacent samples R01-R04, two left adjacent samples R05-R06, and an upper left adjacent sample R00. Similarly, slice B7 has seven adjacent samples R70-R76, including four top adjacent samples R71-R74, two left adjacent samples R75-R76, and an upper left adjacent sample R70.

[0122] In some examples, multiple (e.g., five) filter intra modes (or multiple recursive filtering modes) are pre-designed, for example, for AV1. Each filter intra mode can be represented by a set of eight 7-tap filters that reflect the correlation between the sample (or pixel) in the corresponding 4×2 tile (e.g., B0) and the seven neighboring elements (e.g., R00-R06) of the adjacent 4×2 tile B0. The weighting factors of the 7-tap filters can be position-dependent. For each of the tiles B0-B7, seven neighboring elements (e.g., R00-R06 for B0 and R70-R76 for B7) can be used to predict the sample in the corresponding tile. In the example, the neighboring elements R00-R06 are used to predict the sample in the tile B0. In the example, the neighboring elements R70-R76 are used to predict the sample in the tile B7. For some tiles in CB (1100), such as tile B0, all seven neighboring elements (e.g., R00-R06) have been reconstructed. For other slices in CB (1100), at least one of the seven neighboring elements is not reconstructed, and therefore the predicted value of the direct neighboring element (or the predicted sample of the direct neighboring element) can be used as a reference. For example, the seven neighboring elements R70-R76 of slice B7 are not reconstructed, so the predicted samples of the direct neighboring elements can be used.

[0123] Chroma samples may be predicted from luma samples. In an embodiment, a luma-to-chroma mode (e.g., CfL mode, CfL predictor) is a chroma-only intra-frame predictor that may model chroma samples (or pixels) as linear functions of coincident reconstructed luma samples (or pixels). For example, CfL prediction may be expressed using the following equation (2).

[0124] CfL(α)=αLA +D (Formula 2)

[0125] Among them, L A represents the AC contribution of the luma component, α represents the scaling parameter of the linear model, and D represents the DC contribution of the chroma component. In an example, the reconstructed luma pixels are subsampled based on the chroma resolution, and the average value is subtracted to form the AC contribution (e.g., L A ). To approximate the chroma AC components from the AC contributions, the decoder is not required to calculate the scaling parameter α, but in some examples, such as in AV1, the CfL mode determines the scaling parameter α based on the original chroma pixels and signals the scaling parameter α in the bitstream, thereby reducing decoder complexity and producing more accurate predictions. The DC contribution of the chroma components can be calculated using the intra DC mode. The intra DC mode can be sufficient for most chroma content and has a mature and fast implementation.

[0126] Multi-line intra prediction can use more reference lines for intra prediction. The reference line can include multiple samples in a picture. In an example, the reference line includes samples in rows and samples in columns. In an example, the encoder can determine and signal the reference line used to generate the intra predictor. The index indicating the reference line (also called the reference line index) can be signaled before the intra prediction mode. In an example, only MPM is allowed when a non-zero reference line index is signaled. Fig.12 An example of four reference lines for CB (1210) is shown. Fig.12 , a reference line may include up to six segments, such as segments A to F, and an upper left reference sample. For example, reference line 0 includes segments B and E, and an upper left reference sample. For example, reference line 3 includes segments A to F, and an upper left reference sample. Segments A and F may be filled with the closest samples from segments B and E, respectively. In some examples, such as in HEVC, only one reference line (e.g., reference line 0 adjacent to CB (1210)) is used for intra prediction. In some examples, such as in VVC, multiple reference lines (e.g., reference line 0, reference line 1, and reference line 3) are used for intra prediction.

[0127] Typically, one can use, for example, the above reference Figures 9 to 12 The block is predicted using one or a suitable combination of the various intra prediction modes described.

[0128] Implementations of primary transforms such as those used in AOMedia Video 1 (AV1) are described below. A forward transform (e.g., in an encoder) may be performed on a transform block (TB) including a residual (e.g., a residual in the spatial domain) to obtain a TB including transform coefficients in the frequency domain (or spatial frequency domain). The TB including the residual in the spatial domain may be referred to as a residual TB, and the TB including the transform coefficients in the frequency domain may be referred to as a coefficient TB. In an example, the forward transform includes a forward primary transform that may transform the residual TB into a coefficient TB. In an example, the forward transform includes a forward primary transform and a forward secondary transform, wherein the forward primary transform may transform the residual TB into an intermediate coefficient TB, and the forward secondary transform may transform the intermediate coefficient TB into a coefficient TB.

[0129] An inverse transform may be performed on the coefficient TB in the frequency domain (e.g., in an encoder or decoder) to obtain a residual TB in the spatial domain. In an example, the inverse transform includes an inverse primary transform that may transform the coefficient TB into a residual TB. In an example, the inverse transform includes an inverse primary transform and an inverse secondary transform, wherein the inverse secondary transform may transform the coefficient TB into an intermediate coefficient TB, and the inverse primary transform may transform the intermediate coefficient TB into a residual TB.

[0130] Typically, the primary transform may refer to a forward primary transform or an inverse primary transform, where the primary transform is performed between the residual TB and the coefficient TB. In some embodiments, the primary transform may be a separable transform, where the 2D primary transform may include a horizontal primary transform (also referred to as a horizontal transform) and a vertical primary transform (also referred to as a vertical transform). The secondary transform may refer to a forward secondary transform or an inverse secondary transform, where the secondary transform is performed between the intermediate coefficient TB and the coefficient TB.

[0131] To support extended coding block partitioning, as described in this disclosure, multiple transform sizes (e.g., ranging from 4 points to 64 points for each dimension) and transform shapes (e.g., rectangular, square with a width to height ratio of 2:1, 1:2, 4:1, or 1:4) may be used, for example in AV1.

[0132] The 2D transform process may use a hybrid transform kernel that may include a different 1D transform for each dimension of the encoded residual block. The primary 1D transform may include (a) 4-point, 8-point, 16-point, 32-point, 64-point DCT-2; (b) 4-point, 8-point, 16-point asymmetric DST (ADST) (e.g., DST-4, DST-7) and corresponding flipped versions (e.g., a flipped version of ADST or FlipADST may apply ADST in reverse order); and / or (c) 4-point, 8-point, 16-point, 32-point identity transform (IDTX). Fig.13 Examples of primary transform basis functions according to embodiments of the present disclosure are shown. Fig.13 The primary transform basis functions in the example include basis functions of DCT-2 and asymmetric DST (DST-4 and DST-7) with N-point input. Fig.13 The primary transform basis functions shown can be used for AV1.

[0133] The availability of hybrid transform kernels may depend on the transform block size and the prediction mode. Fig.14A An exemplary correlation of the availability of various transform kernels (e.g., the transform types shown in the first column and described in the second column) based on transform block size (e.g., the size shown in the third column) and prediction mode (e.g., intra prediction and inter prediction shown in the third column) is shown. An exemplary hybrid transform kernel and availability based on prediction mode and transform block size can be used in AV1. Fig.14A , the symbols “→” and “↓” represent the horizontal dimension (also referred to as the horizontal direction) and the vertical dimension (also referred to as the vertical direction), respectively. The symbols “√” and “×” represent the availability of the transform kernel for the corresponding block size and prediction mode. For example, the symbol “√” indicates that the transform kernel is available, and the symbol “×” indicates that the transform kernel is not available.

[0134] In the example, the transformation type (1410) is composed of Fig.14A The ADST_DCT representation is shown in the first column of Fig.14A As shown in the second column of , the transform type (1410) includes ADST in the vertical direction and DCT in the horizontal direction. Fig.14A The third column, when the block size is less than or equal to 16×16

[0135] (e.g., 16×16 samples, 16×16 luma samples), the transform type (1410) can be used for intra prediction and inter prediction.

[0136] In the example, Fig.14A As shown in the first column of , the transform type (1420) is represented by V_ADST. Fig.14A As shown in the second column of , the transform type (1420) includes ADST in the vertical direction and IDTX (i.e., identity matrix) in the horizontal direction. Therefore, the transform type (1420) (e.g., V_ADST) is performed in the vertical direction but not in the horizontal direction. Fig.14A In the third column, regardless of the block size, the transform type (1420) is not available for intra prediction. When the block size is less than 16×16 (e.g., 16×16 samples, 16×16 luma samples), the transform type (1420) is available for inter prediction.

[0137] In the example, Fig.14A Applies to luma components. For chroma components, the transform type (or transform kernel) selection can be performed implicitly. In the example, Fig. 14B As shown, for the intra-frame prediction residual, the transform type can be selected according to the intra-frame prediction mode. In the example, Fig. 14B The transform type selection shown is applicable to chroma components. For inter prediction residuals, the transform type may be selected according to the transform type selection of the co-located luma block. Therefore, in the example, the transform type for chroma components is not signaled in the bitstream.

[0138] A transform such as a primary transform, a secondary transform may be applied to a block such as a CB. In an example, the transform includes a combination of a primary transform, a secondary transform. The transform may be a non-separable transform, a separable transform, or a combination of a non-separable transform and a separable transform.

[0139] The secondary transform may be performed, for example, in VVC. Figure 15 to Figure 16 As shown in, for example in VVC, a low-frequency non-separable transform (LFNST) (also called a reduced secondary transform (RST)) can be applied between the forward primary transform and quantization at the encoder side and between dequantization and the inverse primary transform at the decoder side to further decorrelate the primary transform coefficients.

[0140] Using a 4×4 input block (or input matrix) X as an example (as shown in Equation 3), the application of a non-separable transform that can be used in LFNST can be described as follows. As shown in Equations 3 to 4, in order to apply a 4×4 non-separable transform (e.g., LFNST), the 4×4 input block X can be represented by the vector express.

[0141]

[0142]

[0143] The non-separable transformation can be calculated as in represents the transform coefficient vector, and T is the 16×16 transform matrix. The 16×1 coefficient vector may then be transformed using the scanning order of the 4×4 input block (e.g., horizontal scanning order, vertical scanning order, zigzag scanning order, or diagonal scanning order). Reorganize into 4×4 output blocks (or output matrices, coefficient blocks). Transform coefficients with smaller indices can be placed in the 4×4 coefficient blocks with smaller scan indices.

[0144] A non-separable secondary transform may be applied to a block (e.g., CB). In some examples, such as Figure 15 to Figure 16 As shown, for example, in VVC, LFNST is applied between the forward primary transform and quantization (eg, at the encoder side) and between dequantization and the inverse primary transform (eg, at the decoder side).

[0145] Figure 15 to Figure 16 Examples of two transform coding processes (1700) and (1800) using a 16x64 transform (or a 64x16 transform, depending on whether the transform is a forward or inverse sub-transform) and a 16x48 transform (or a 48x16 transform, depending on whether the transform is a forward or inverse sub-transform) are shown, respectively. Fig.15 , in the process (1700), at the encoder side, a forward primary transform (1710) may be first performed on a block (e.g., a residual block) to obtain a coefficient block (1713). Subsequently, a forward secondary transform (or forward LFNST) (1712) may be applied to the coefficient block (1713). In the forward secondary transform (1712), the 64 coefficients of the 4×4 sub-block AD at the upper left corner of the coefficient block (1713) may be represented by a vector of length 64, and the vector of length 64 may be multiplied by a 64×16 transform matrix (i.e., a width of 64 and a height of 16) to obtain a vector of length 16. The elements in the vector of length 16 are filled back into the upper left 4×4 sub-block A of the coefficient block (1713). The coefficients in the sub-block BD may be zero. The coefficients resulting after the forward sub-transform (1712) are then quantized at a quantization step (1714) and entropy encoded to generate coded bits in a bitstream (1716).

[0146] The coded bits may be received at the decoder side and entropy decoded followed by a dequantization step (1724) to generate coefficient blocks (1723). An inverse secondary transform (or inverse LFNST) may be performed.

[0147] (1722), such as an inverse RST 8×8, to obtain 64 coefficients from, for example, 16 coefficients at the upper left 4×4 sub-block E. The 64 coefficients can be filled back into the 4×4 sub-blocks E-H. In addition, the coefficients in the coefficient block (1723) after the inverse secondary transform (1722) can be processed with an inverse primary transform (1720) to obtain a restored residual block.

[0148] Fig.16 The example process (1800) is similar to the process (1700), except that fewer (i.e., 48) coefficients are processed during the forward secondary transform (1712). Specifically, 48 coefficients in sub-blocks A-C are processed with a smaller transform matrix of size 48×16. Using the smaller 48×16 transform matrix can reduce the memory size for storing the transform matrix and the number of computations (e.g., multiplications, additions, subtractions, etc.), and thus can reduce the computational complexity.

[0149] In the example, a 4×4 non-separable transform (e.g., 4×4 LFNST) or an 8×8 non-separable transform (e.g., 8×8 LFNST) is applied according to the block size of a block (e.g., CB). The block size of the block can include width, height, etc. For example, in the case where the minimum value of the width and height is less than a threshold (e.g., 8) (e.g., min(width, height) < 8), the 4×4 LFNST is applied to the block. For example, in the case where the minimum value of the width and height is greater than a threshold (e.g., 4) (e.g., min(width, height) > 4), the 8×8 LFNST is applied to the block.

[0150] The non-separable transform (e.g., LFNST) can be based on a direct matrix multiplication method and can thus be implemented in a single pass without iteration. To reduce the dimension of the non-separable transform matrix and minimize the storage space for storing the transform coefficients and the computational complexity, a simplified non-separable transform method (or RST) can be used in the LFNST. Thus, in the simplified non-separable transform, an N-dimensional vector (e.g., for an 8×8 non-separable secondary transform (NSST), N is 64) can be mapped to an R-dimensional vector in a different space, where N / R (R < N) is a reduction factor. Thus, instead of an N×N matrix, the RST matrix is an R×N matrix as described in Equation 5.

[0151]

[0152] In Equation 5, the R rows of the R×N transform matrix are R bases of the N-dimensional space. The inverse transform matrix can be the transform matrix used in the forward transform (e.g., T RxN). For 8×8 LFNST, a simplification factor of 4 can be applied, and the 64×64 direct matrix used in the 8×8 non-separable transform can be simplified to a 16×64 direct matrix, as Fig.15 Alternatively, a simplification factor greater than 4 can be applied, and the 64×64 direct matrix used in the 8×8 non-separable transform can be simplified to a 16×48 direct matrix, as shown in Fig.16 Therefore, the 48×16 inverse RST matrix can be used at the decoder side to generate the core (primary) transform coefficients in the 8×8 upper left region.

[0153] Reference Fig.16 , when a 16×48 matrix is ​​applied instead of a 16×64 matrix with the same transform set configuration, the input of the 16×48 matrix includes 48 input data from three 4×4 blocks A, B, and C in the upper left 8×8 block except for the lower right 4×4 block D. By reducing the dimension, the memory usage for storing the LFNST matrix can be reduced, for example, from 10KB to 8KB with minimal performance degradation.

[0154] To reduce complexity, LFNST can be restricted to be applicable only when the coefficients outside the first coefficient subgroup are insignificant. In an example, LFNST can be restricted to be applicable only when all coefficients outside the first coefficient subgroup are insignificant. Figure 15 to Figure 16 , the first coefficient subgroup corresponds to the upper left block E, and therefore the coefficients outside block E are not significant.

[0155] In an example, when LFNST is applied, only the primary transform coefficients are non-significant (e.g., zero). In an example, when LFNST is applied, all only primary transform coefficients are zero. Only primary transform coefficients may refer to transform coefficients obtained from the primary transform without a secondary transform. Therefore, LFNST index signaling may be conditioned on the last valid position, thereby avoiding additional coefficient scanning in LFNST. In some examples, additional coefficient scanning is used to check the valid transform coefficients at a specific position. In an example, in terms of per-pixel multiplication, the worst-case processing of LFNST, for example, limits the non-separable transforms of 4×4 blocks and 8×8 blocks to 8×16 transforms and 8×48 transforms, respectively. In the above case, when LFNST is applied, the last valid scan position may be less than 8. For other sizes, when LFNST is applied, the last valid scan position may be less than 16. For 4×N and N×4 CBs, and N is greater than 8, the restriction may mean that LFNST is applied to the upper left 4×4 region in the CB. In an example, the restriction means that LFNST is applied only once to the upper left 4×4 region in the CB. In an example, when LFNST is applied, all primary coefficients are non-significant (e.g., zero), and the number of operations for the primary transform is reduced. From the encoder's perspective, when testing the LFNST transform, the quantization of the transform coefficients can be significantly simplified. For example, for the first 16 coefficients in the scan order, the rate-distortion optimized quantization can be performed to the greatest extent, and the remaining coefficients can be set to zero.

[0156] The LFNST transform (e.g., transform kernel, transform core, or transform matrix) may be selected as follows. In an embodiment, multiple transform sets may be used, and one or more non-separable transform matrices (or kernels) may be included in each of the multiple transform sets in the LFNST. According to aspects of the present disclosure, a transform set may be selected from multiple transform sets, and a non-separable transform matrix may be selected from one or more non-separable transform matrices in the transform set.

[0157] Table 1 shows an exemplary mapping from intra prediction modes to multiple transform sets according to an embodiment of the present disclosure. The mapping indicates the relationship between intra prediction modes and multiple transform sets. Relationships such as those indicated in Table 1 may be predefined and may be stored in encoders and decoders.

[0158] Table 1: Transformation set selection table

[0159] IntraPredMode Tr.set index IntraPredMode<c 1 0<=IntraPredMode<=1 0 2<=IntraPredMode<=12 1 13<=IntraPredMode<=23 2 24<=IntraPredMode<=44 3 45<=IntraPredMode<=55 2 56<=IntraPredMode<=80 1 81<=IntraPredMode<=83 0

[0160] Referring to Table 1, the plurality of transform sets include four transform sets, for example, transform set 0 to transform set 3, respectively represented by transform set indices (e.g., Tr.set index) from 0 to 3. The index (e.g., IntraPredMode) may indicate an intra prediction mode, and the transform set index may be obtained based on the index and Table 1. Thus, the transform set may be determined based on the intra prediction mode. In an example, if one of three cross component linear model (CCLM) modes (e.g., INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for a CB (e.g., 81≤IntraPredMode≤83), transform set 0 is selected for the CB.

[0161] As described above, each transform set may include one or more non-separable transform matrices. One of the one or more non-separable transform matrices may be selected by, for example, an LFNST index explicitly indicated by a signal. The LFNST index may be indicated by a signal once in the bitstream of each intra-coded CU (e.g., CB), for example, after the transform coefficient is indicated by a signal. In an embodiment, each transform set includes two non-separable transform matrices (kernels), and the selected non-separable secondary transform candidate may be one of the two non-separable transform matrices. In some examples, LFNST is not applied to the CB (e.g., the number of non-zero coefficients of the CB or CB encoded with a transform skip mode is less than a threshold). In an example, when LFNST is not applied to the CB, the LFNST index is not indicated by a signal for the CB. The default value of the LFNST index may be zero and not indicated by a signal, thereby indicating that LFNST is not applied to the CB.

[0162] In an embodiment, LFNST is restricted to be applicable only when all coefficients outside the first coefficient subgroup are non-significant, and the encoding of the LFNST index may depend on the position of the last significant coefficient. The LFNST index may be context-coded. In an example, the context encoding of the LFNST index does not depend on the intra prediction mode, and only the first binary bit is context-coded. LFNST may be applied to intra-coded CUs in intra slices or inter slices, and may be applied to both luma and chroma components. If dual tree is enabled, the LFNST indexes for luma and chroma components may be signaled separately. For inter slices (e.g., dual tree is disabled), a single LFNST index may be signaled and the LFNST index may be used for both luma and chroma components.

[0163] Intra sub-partition (ISP) coding mode can be used. In ISP coding mode, the luma intra prediction block can be divided into 2 or 4 sub-partitions vertically or horizontally according to the block size. In some examples, when RST is applied to each feasible sub-partition, the performance improvement is negligible. Therefore, in some examples, when ISP mode is selected, LFNST is disabled and the LFNST index (or RST index) is not signaled. Disabling RST or LFNST for ISP predicted residuals can reduce coding complexity. In some examples, when matrix-based intra predication mode (MIP) is selected, LFNST is disabled and the LFNST index is not signaled.

[0164] In some examples, due to the implicit partitioning (TU tiling) of CUs larger than 64×64 due to the maximum transform size limit (e.g., 64×64), LFNST index search can increase data buffering by four times for a certain number of decoding pipeline stages. Therefore, the maximum size allowed for LFNST can be limited to 64×64. In an example, LFNST is enabled only with discrete cosine transform (DCT) type 2 (DCT-2) transforms.

[0165] In some examples, the separable transform scheme may not be efficient for capturing directional texture patterns (e.g., edges along 45° or 135° directions). For example, in the above scenario, a non-separable transform scheme may improve coding efficiency. In order to reduce computational complexity and memory usage, a non-separable transform scheme may be used as a secondary transform that is applied to the low-frequency transform coefficients obtained from the primary transform.

[0166] In some implementations, the selection of the transform kernel to be used, for example from the set of transform kernels, is based on the prediction mode information. The prediction mode information may indicate a prediction mode.

[0167] In some examples, the prediction mode information alone provides only a coarse representation of the entire space of residual patterns observed for the prediction mode. FIG. 17A to FIG. 17DAn exemplary residual pattern (in grayscale) observed for directional mode D45_PRED (45°) as an intra prediction mode in AV1 according to an embodiment of the present disclosure is shown. According to aspects of the present disclosure, neighboring reconstructed samples can provide additional information for more efficiently representing the residual pattern. Therefore, methods for transform set selection and / or transform kernel selection based on neighboring reconstructed samples in addition to prediction mode information are disclosed. For example, feature indicators (e.g., feature vectors) of neighboring reconstructed samples are used in addition to prediction mode information. Or feature scalar S) as input to the selection process.

[0168] In the present disclosure, the term block may refer to PB, CB, coded block, coding unit (CU), transform block (TB), transform unit (TU), luminance block (e.g., luminance CB), chrominance block (e.g., chrominance CB), etc.

[0169] In the present disclosure, the size of a block may refer to block width, block height, block aspect ratio (e.g., the ratio of block width to block height, the ratio of block height to block width), block area size or block area (e.g., block width×block height), minimum value of block width and block height, maximum value of block width and block height, etc.

[0170] In the present disclosure, the transform kernel of a block can be used for a primary transform, a secondary transform, a tertiary transform, or a transform scheme beyond the tertiary transform. The transform kernel can be used for a separable transform or a non-separable transform. The transform kernel can be used for a luminance block, a chrominance block, an inter-frame prediction, an intra-frame prediction, etc. The transform kernel can also be referred to as a transform core, a transform candidate, a transform kernel option, etc. In an example, the transform kernel is a transform matrix. The block can be transformed based on at least the transform kernel. Therefore, the method in the present disclosure can be applied to a primary transform, a secondary transform, a tertiary transform, any transform scheme beyond the tertiary transform, a separable transform, a non-separable transform, a luminance block, a chrominance block, an inter-frame prediction, an intra-frame prediction, etc.

[0171] In the present disclosure, a transform set may refer to a group of transform kernels or transform kernel options. A transform set may include one or more transform kernels or transform kernel options. In an example, a transform kernel for a block may be selected from a transform set.

[0172] According to aspects of the present disclosure, neighboring reconstructed samples may be used to determine a transform kernel from a grouped transform set. Neighboring reconstructed samples of a block being reconstructed in a current picture may be used to determine a transform kernel for the block from the grouped transform set. The grouped transform set may be predetermined. In an example, the grouped transform set is pre-stored in an encoder and / or a decoder.

[0173] The neighboring reconstructed samples (e.g., a set of neighboring reconstructed samples) of a block may refer to reconstructed samples (e.g., groups of reconstructed samples) from a previously decoded neighboring block in a current picture or reconstructed samples in a previously decoded picture. The previously decoded neighboring block may include one or more neighboring blocks of the block, and the neighboring reconstructed samples are also referred to as reconstructed samples in one or more neighboring blocks of the block. The one or more neighboring blocks of the block may be in the current picture or in a reconstructed picture different from the current picture (e.g., a reference picture). The neighboring reconstructed samples of the block may include spatial neighboring samples of the block in the current picture and / or temporal neighboring samples of the block in another picture different from the current picture (e.g., a previously decoded picture). Fig.18 An exemplary spatial neighboring sample of a block according to an embodiment of the present disclosure is shown. The block is a current block (1851) in a current picture. Samples 1-4 and AX are spatial neighboring samples of the current block (1851) and have been reconstructed. In the example, samples 1-4 and AX are in one or more reconstructed neighboring blocks of the current block (1851). According to various aspects of the present disclosure, one or more of samples 1-4 and AX can be used to determine the transform kernel of the current block (1851). In the example, the neighboring reconstructed samples of the current block (1851) include the upper left neighboring sample 4, the left neighboring column (1852) including the sample AF, and the upper neighboring row (1854) including the sample GL, and are used to determine the transform kernel of the current block (1851). In the example, the adjacent reconstructed samples of the current block (1851) include upper left adjacent samples 1-4, left adjacent columns (1852) to (1853) including samples AF and MR, and upper adjacent rows (1854)-(1855) including samples GL and SX, and are used to determine the transform kernel of the current block (1851).

[0174] The grouped transform set may include one or more transform sets. Each of the one or more transform sets may include any suitable one or more transform kernels. Thus, the grouped transform set may include transform kernels used in a primary transform, a secondary transform, a tertiary transform, or a transform scheme beyond a tertiary transform. The grouped transform set may include transform kernels that are separable transforms and / or non-separable transforms. The grouped transform set may include transform kernels for luma blocks, chroma blocks, inter prediction, intra prediction, and the like.

[0175] According to aspects of the present disclosure, a transform kernel (or transform candidate) for a block in a current picture may be determined from a grouped transform set based on reconstructed samples in one or more neighboring blocks of the block. The transform kernel (or transform candidate) may be determined based on feature indicators (e.g., feature vectors) extracted from reconstructed samples in one or more neighboring blocks of the block in the current picture. A transform kernel (or transform candidate) for the block is determined from a grouped transform set using a feature scalar (or feature scalar S). Each set in the grouped transform set may include one or more transform candidates for the block. One or more neighboring blocks may be in a current picture or a reconstructed picture different from the current picture. In addition, samples of the block may be reconstructed based on the determined transform candidates.

[0176] In an embodiment, feature indicators (e.g., feature vectors) of adjacent reconstructed samples may be extracted from the adjacent reconstructed samples. or feature scalar S). Therefore, the transform kernel of a block (e.g., current block (1851)) can be determined based on the feature indicator. The feature indicator can be a vector (referred to as a feature vector ) or a scalar (referred to as a feature scalar S). The feature indicator may be extracted based on statistical information or statistical analysis of the neighboring reconstructed samples, for example, based on one or more variables associated with the neighboring reconstructed samples. The one or more variables may indicate one or more distributions of sample values ​​of the neighboring reconstructed samples.

[0177] In an embodiment, a transform kernel of a block (e.g., a current block (1851)) may be determined based on neighboring reconstructed samples of the block (e.g., a feature indicator) and prediction mode information. The prediction mode information of the block may indicate information used to predict the block (e.g., inter-frame prediction or intra-frame prediction of the block). According to aspects of the present disclosure, the prediction mode information of the block may indicate a prediction mode of the block (e.g., intra-frame prediction mode, inter-frame prediction mode). In an example, the transform kernel of the block is determined based on neighboring reconstructed samples of the block (e.g., a feature indicator) and the prediction mode.

[0178] In an example, the block is intra-coded or intra-predicted, and the prediction mode information is referred to as intra-prediction mode information. The prediction mode information (e.g., intra-prediction mode information) of the block may indicate an intra-prediction mode used for the block. The intra-prediction mode may refer to a prediction mode used for intra-prediction of the block, such as Fig. 9 The directional mode (or directional prediction mode) described in Fig.10 A non-directional prediction mode described in (e.g., DC mode, PAETH mode, SMOOTH mode, SMOOTH_V mode, or SMOOTH_H mode) or Fig.11 The intra-frame prediction mode may also refer to the prediction mode described in the present disclosure, a suitable variation of the prediction mode described in the present disclosure, or a suitable combination of the prediction modes described in the present disclosure. For example, the intra-frame prediction mode may be combined with Fig.12 The multi-line intra prediction combination described in .

[0179] More specifically, for example, a transform set subset may be selected from the grouped transform set based on encoded information of a block from the encoded video bitstream. The encoded information of the block may include prediction mode information indicating a prediction mode (e.g., an intra-prediction mode or an inter-prediction mode) of the block. In an embodiment, a transform set subset may be selected from the grouped transform set based on the prediction mode information (e.g., a prediction mode).

[0180] According to aspects of the present disclosure, a subset of transform sets may be selected from a grouped transform set based on a prediction mode of a block indicated in encoded information of the block. Furthermore, transform candidates from the subset of transform sets may be determined based on reconstructed samples in one or more neighboring blocks of the block. For example, a transform candidate may be determined based on feature indicators (e.g., feature vectors) extracted from reconstructed samples in one or more neighboring blocks of the block. or feature scalar S) to determine transform candidates from a subset of the transform set.

[0181] After selecting a transform set subset from the grouped transform sets, a transform kernel for the block may be further determined based at least on neighboring reconstructed samples (eg, feature indicators) using any suitable method as described below.

[0182] In an embodiment, the transform set in the selected transform set subset is identified or selected from the selected transform set subset using neighboring reconstructed samples of the block (or reconstructed samples in one or more neighboring blocks of the block). In an example, feature indicators (e.g., feature vectors) extracted from the reconstructed samples in one or more neighboring blocks of the block are used. The information may include a first index or a feature scalar S (e.g., a feature scalar S) to identify or select a transform set in the selected transform set subset. In addition, a transform kernel (or transform candidate) for the block may be determined (e.g., selected) from the identified (or selected) transform set based on the first index. The encoded information may indicate (e.g., include) the first index. The first index may be signaled in the encoded video bitstream.

[0183] In an embodiment, a transform set in the selected transform set subset is selected from the selected transform set subset based on the second index. The encoded information may indicate (e.g., include) the second index. The second index may be signaled in the encoded video bitstream. In addition, a transform kernel (or transform candidate) of the block from the selected transform set may be determined (e.g., selected) based on neighboring reconstructed samples of the block (or reconstructed samples in one or more neighboring blocks of the block) (e.g., feature indicators). In an example, a feature indicator of the block (e.g., a feature vector) may be used. or feature scalar S) to determine the transform kernel (or transform candidate) in the selected transform set.

[0184] In an embodiment, a transform kernel (or transform candidate) of a block is implicitly determined (or identified) from a selected subset of transform sets based on neighboring reconstructed samples of the block. In an example, a feature indicator (e.g., a feature vector) of the block from the selected subset of transform sets is used. or feature scalar S) and does not use an index indicated by a signal in the coded video bitstream to implicitly determine (or identify) the transform kernel (or transform candidate).

[0185] A feature indicator (eg, a feature vector) of the block may be determined (eg, obtained) based on neighboring reconstructed samples. According to various aspects of the present disclosure, a feature indicator of a block (e.g., a feature vector S) may be determined or obtained based on a statistical analysis of neighboring reconstructed samples of the block (or reconstructed samples in one or more neighboring blocks of the block). As described above, one of the following operations may be performed: (i) based on the feature indicator (e.g., feature vector or feature scalar S) to determine a transform set from a subset of the transform sets, (ii) to determine a transform set based on a feature indicator (e.g., a feature vector or feature scalar S) to determine the transform candidates from the transform set subset, or (iii) the transform set can be selected from the transform set subset based on an index in the encoded information and based on a feature indicator (e.g., feature vector Or feature scalar S) determines transform candidates from the selected transform set.

[0186] In an embodiment, a single variable X may be used to represent adjacent reconstructed samples, and the characteristic indicator of a block is a characteristic scalar S of the block. The variable X may indicate a sample value of an adjacent reconstructed sample. In an example, the variable X is an array of sample values ​​of adjacent reconstructed samples, and reflects the distribution of sample values ​​over adjacent reconstructed samples. The characteristic scalar S may indicate statistical information of the sample value. The characteristic scalar S may include, but is not limited to, a scalar quantitative measurement of the variable X that can be obtained from adjacent reconstructed samples, such as the mean (or first-order moment) of the variable X, the variance (or second-order moment) of the variable X, and the skewness (or third-order moment) of the variable X. In an example, the variable X is referred to as a random variable X.

[0187] In an example, the feature indicator is a feature scalar S. The feature scalar S is determined as a moment (eg, first order moment, second order moment, third order moment, etc.) of the variable X, which indicates sample values ​​of reconstructed samples in one or more neighboring blocks of the block.

[0188] An example of a variable X is Fig.18As shown. In the example, variable X includes reconstructed neighboring samples adjacent to the current block (1851), including sample 4, left adjacent column (1852) and upper adjacent row (1854). In the example, variable X includes reconstructed neighboring samples 1 to 4 and A to X.

[0189] In an embodiment, multiple variables (e.g., two variables Y and Z) may be used to represent different sets of adjacent reconstructed samples (if available) (e.g., a first set corresponding to Y and a second set corresponding to Z), and the feature indicator of a block is a feature vector of the block. Multiple variables can respectively indicate sample values ​​of different sets of adjacent reconstructed samples. In an example, variable Y is an array of first sample values ​​of a first set of adjacent reconstructed samples, and reflects the distribution of the first sample values ​​of the first set of adjacent reconstructed samples. Variable Z is an array of second sample values ​​of a second set of adjacent reconstructed samples, and reflects the distribution of the second sample values ​​of the second set of adjacent reconstructed samples. Feature vector Statistics for multiple variables can be indicated.

[0190] Eigenvector The statistical analysis of multiple variables that can be obtained from adjacent reconstructed samples may include, but is not limited to, joint variability measures of multiple variables (e.g., covariance or second-order moments of multiple variables (e.g., two variables Y and Z)). In an example, the multiple variables (e.g., two variables Y and Z) are referred to as random variables.

[0191] Fig.18 1850 shows an example of multiple variables Y and Z. In the example, variable Y includes a left adjacent column (1852), and variable Z includes an upper adjacent row (1854). In the example, one of variable Y and variable Z includes an upper left sample 4.

[0192] In the example, the feature indicator is the feature vector Eigenvector Determined as the joint variability of variables (e.g., variable Y and variable Z), which respectively indicate the sample values ​​of reconstructed samples of the adjacent column (e.g., left adjacent column (1852)) on the left side of a block (e.g., current block (1851)) and the sample values ​​of reconstructed samples of the adjacent row (e.g., upper adjacent row (1854)) on the upper side of the block (e.g., current block (1851)).

[0193] In the example, variable Y includes left adjacent columns (1852) to (1853), and variable Z includes upper adjacent rows (1854) to (1855). In the example, one of variable Y and variable Z includes upper left samples 1 to 4.

[0194] According to various aspects of the present disclosure, when a feature indicator is used to determine a transform kernel of a block or to determine a transform set including a transform kernel of a block, the transform kernel or the transform set can be determined based on the feature indicator and a threshold. The threshold can be selected from a predefined threshold set. The transform kernel or the transform set can be determined based on the feature indicator and the threshold using any suitable method. Some exemplary methods are described below. As described above, according to various aspects of the present disclosure, a transform set subset is selected from a grouped transform set, for example, based on prediction mode information. Subsequently, in an example, a transform set is identified from the selected transform set subset using the feature indicator of the block and the threshold. In another example, a transform set in the selected transform set subset is selected using an index (e.g., a second index) indicated by a signal in the coded video bitstream, and a transform kernel of a block from the selected transform set can be determined (e.g., selected) using the feature indicator of the block and the threshold. In another example, a transform kernel is implicitly identified using a feature indicator and a threshold from the selected transform set subset.

[0195] For example, for classification purposes, a threshold set (denoted as K s ). In the example, one of the following operations may be performed: (i) based on the moment of variable X and from a predefined threshold set K s (ii) determining transform candidates from the transform set subset based on the moment of variable X and the threshold, or (iii) selecting a transform set from the transform set subset based on an index in the encoded information, and determining transform candidates from the selected transform set based on the moment of variable X and the threshold.

[0196] In an embodiment, the feature scalar S is used to identify a transform kernel for a block, or to identify a transform set comprising a transform kernel for a block, for example, from a selected subset of transform sets as described above. s One or more first thresholds (or first thresholds) may be included.

[0197] In an example, the feature scalar S is a quantitative measure of the variable X, such as the mean, variance, skewness, etc. of the variable X. For example, the feature scalar S is a moment of the variable X. The moment of the variable may be one of a first moment of the variable X (or mean), a second moment of the variable X (or variance), or a third moment of the variable X (e.g., skewness).

[0198] Each prediction mode (eg, intra prediction mode or inter prediction mode) may correspond to a threshold set K s The unique threshold subset K in s ', for example, this indicates that multiple prediction modes are related to the threshold set K sThe prediction mode of a block can be one of multiple prediction modes. Each prediction mode is associated with a corresponding threshold subset K. s ' is an injective mapping. For example, the threshold set K s The threshold subset K corresponding to the first prediction mode is included s1 ' and the threshold subset K corresponding to the second prediction mode s2 ', and the threshold subset K s1 There is no element or no threshold with threshold subset K s2 ' have the same elements or thresholds.

[0199] In the example, the feature scalar S is a quantitative measure of the variable X, such as the mean, variance, skewness, etc. of the variable X. Each prediction mode may correspond to a threshold set K s Any threshold subset K in s '. Each prediction mode is related to the threshold set K s The corresponding threshold subset K in s The mapping between ' is a non-injective mapping. In an example, multiple prediction modes (eg, a first prediction mode and a second prediction mode) may correspond to a threshold set K s The same threshold subset K in s '. In this example, the threshold set K s The threshold subset K corresponding to the first prediction mode is included s3 ' and the threshold subset K corresponding to the second prediction mode s4 ', and the threshold subset K s3 The elements (or thresholds) in ' and the threshold subset K s4 ' are the same as the elements (or thresholds) in . In this example, the threshold set K s Is a single element collection.

[0200] In an embodiment, the threshold set K s The threshold subset K in s The elements of ' are thresholds that depend on the quantization index (or the corresponding quantization step associated with the quantization index). In general, each quantization index can correspond to a unique quantization step. s The corresponding threshold subset K in s The above description is applicable when the mapping between ' is non-injective or injective.

[0201] In an embodiment, only the threshold set K is defined s The threshold subset K in s ' corresponds to an element of a specific quantization index (or a corresponding quantization step size). The threshold subset K can be obtained using a mapping function sThe remaining elements in ' (if any). The mapping function can be linear or nonlinear. The mapping function can be predefined and used by the encoder and / or decoder. The threshold subset K s The remaining elements in ' may correspond to different quantization indices or (or corresponding quantization step sizes). s The corresponding threshold subset K in s The above description is applicable when the mapping between ' is non-injective or injective.

[0202] In an embodiment, a lookup table may be used to determine the threshold subset K. s '. For example, a lookup table is used to set the threshold subset K based on the prediction mode and / or quantization index. s 'Select element. In an example, the lookup table includes a relationship between a prediction mode, a quantization index (or a corresponding quantization step size) and a threshold. The lookup table can be traversed to determine the threshold subset K using the prediction mode (e.g., intra-frame prediction mode or inter-frame prediction mode) and / or the quantization index (or the corresponding quantization step size). s '(For example, for the threshold subset K s 'Select elements). In each prediction mode and threshold set K s The corresponding threshold subset K in s The above description is applicable when the mapping between ' is non-injective or injective.

[0203] In an embodiment, the mapping from the quantization index (or the corresponding quantization step) to the threshold is a linear mapping. Parameters used in the linear mapping, such as the slope and the intercept, may be predefined or derived using the encoded information of the block. The encoded information may include, but is not limited to, block size, quantization index (or the corresponding quantization step), prediction mode (e.g., intra-frame prediction mode or inter-frame prediction mode). The linear mapping (e.g., the parameters used in the linear mapping) may be predefined or derived based on one or a combination of the block size, quantization index (or the corresponding quantization step), prediction mode. If the mapping from the quantization index (or the corresponding quantization step) to the threshold is a nonlinear mapping, the description may be adjusted appropriately. In each prediction mode and threshold set K s The corresponding threshold subset K in s The above description is applicable when the mapping between ' is non-injective or injective.

[0204] As described above, the feature scalar S can be used to identify a transform set including a transform kernel of a block or to identify a transform kernel of a block, and the threshold set K s One or more first thresholds may be included. In an embodiment, the first threshold is selected from a threshold set K, for example, depending on the block size of the block. s In an embodiment, for example, a threshold value is selected from a threshold set Ks The choice of the threshold in depends on the quantization parameter used in the quantization, such as the quantization index (or the corresponding quantization step size). s The selection of the threshold in depends on the prediction mode (eg, intra prediction mode and / or inter prediction mode).

[0205] In an embodiment, the feature vector A transform kernel for identifying a block or a transform set for identifying a transform kernel comprising a block. For classification purposes, a classification vector set may be predefined Sum and classification vector set The associated threshold set (denoted as K v ). Threshold set K v One or more second thresholds or a plurality of second thresholds may be included.

[0206] In the example, for the feature vector Each prediction mode (e.g., intra prediction mode or inter prediction mode) may correspond to a classification vector set The unique categorical vector subset in and threshold set K v The unique threshold subset K in v '. Unique classification vector subset Corresponding to the unique threshold subset K v '. Each prediction mode is associated with a corresponding subset of classification vectors (Together with the associated threshold subset K v ') is an injective mapping. For example, the threshold set K v Including threshold subset K v1 ' and threshold subset K v2 '. Classification vector set Include categorical vector subsets and the classification vector subset One prediction mode corresponds to the threshold subset K v1 ' and classification vector subset And the other prediction mode corresponds to the threshold subset K v2 ' and classification vector subset Threshold subset K v1 There is no threshold and threshold subset K in ' v2 ', and the classification vector subset There are no categorical vectors or categorical vector subsets in The classification vectors in are the same.

[0207] In an example, each prediction mode (eg, intra prediction mode or inter prediction mode) may correspond to a threshold set K v Any threshold subset K inv 'And the classification vector set Any corresponding subset of classification vectors in Each prediction mode is related to the threshold set K v The corresponding threshold subset K in v The mapping between them is non-injective. Each prediction mode is associated with a classification vector set The corresponding classification vector subset in The mapping between is a non-injective mapping. In the example, each prediction mode is associated with a corresponding threshold subset K v ' and the corresponding classification vector subset The mapping between is a non-injective mapping. In the example, multiple prediction modes can correspond to the same threshold subset K v '(and threshold subset K v 'The corresponding classification vector subset associated ).

[0208] In an embodiment, the threshold set K v The threshold subset K in v The elements of 'are thresholds that depend on the quantization index (or the corresponding quantization step size). Similarly, the classification vector set The corresponding classification vector subset in The elements of are classification vectors depending on the quantization index (or the corresponding quantization step size). In (i) each prediction mode and (ii) the corresponding threshold subset K v ' and threshold subset K v 'The corresponding classification vector subset The above description is applicable when the mapping between is non-injective or injective.

[0209] In an embodiment, only the threshold set K is defined s The threshold subset K in v ' an element and a classification vector set The corresponding classification vector subset in The threshold subset K can be obtained by using a mapping function: v The remaining elements (if any) in ' and the categorical vector subset The remaining elements in (if any). The mapping function can be linear or nonlinear. The mapping function can be predefined and used by the encoder and / or decoder. The threshold subset K v The remaining elements in ' and the corresponding classification vector subset The remaining elements in may correspond to different quantization indices (or quantization step sizes corresponding to quantization indices). v' and threshold subset K v 'The corresponding classification vector subset The above description is applicable when the mapping between is non-injective or injective.

[0210] In an embodiment, a lookup table may be used to obtain the threshold subset K by v 'Select elements to determine the threshold subset K v ', and by subsetting the classification vector Select elements to determine the corresponding classification vector subset In an example, the lookup table includes a prediction mode, a quantization index (or a quantization step corresponding to the quantization index), a threshold value, and a classification vector. The lookup table can be traversed to determine the threshold subset K using the prediction mode (e.g., intra-frame prediction mode or inter-frame prediction mode) and / or the quantization index (or the quantization step corresponding to the quantization index). v '(For example, for the threshold subset K v 'Select elements) and corresponding classification vector subsets (For example, for a subset of categorical vectors Select elements). In (i) each prediction mode and (ii) the threshold subset K corresponding to v ' and threshold subset K v 'The corresponding classification vector subset The above description is applicable when the mapping between is non-injective or injective.

[0211] In an embodiment, the feature vector is obtained from the neighboring reconstructed samples of the block In addition, using the prediction mode information and the quantization index (or the corresponding quantization step size), from the classification vector subset Get the classification vector (for example, subset Elements in ), and from the threshold subset K v 'Get the threshold value corresponding to the classification vector (for example, the threshold subset K v ' elements {Kv′}). Note that {}

[0212] (For example, ) represents the corresponding set or subset (for example, a subset of classification vectors ). The prediction mode information may indicate a prediction mode (eg, an intra-frame prediction mode or an inter-frame prediction mode). Two vectors (ie, feature vectors) may be calculated. and classification vector ), and then compare the distance with the threshold {Kv′}. Therefore, it can be based on the threshold {Kv′} and the feature vector With classification vector The distances between the two are compared to identify transformation sets or transformation candidates.

[0213] In an example, one of the following operations may be performed: (i) based on the distance and the number of cells included in the threshold set K v The threshold subset K in v ', determine the transform set from the transform set subset based on the comparison of the distance with the threshold value (e.g., {Kv'}), (ii) determine the transform candidate from the transform set subset based on the comparison of the distance with the threshold value (e.g., {Kv'}), or (iii) select a transform set from the transform set subset based on the index in the encoded information, and determine the transform candidate from the selected transform set based on the comparison of the distance with the threshold value (e.g., {Kv'}). The feature vector can be calculated (e.g., the joint variability of variables) and from a predefined set of classification vectors The classification vector selected in In this example, the feature vector is calculated (e.g., the joint variability of the variables) and the vectors included in the classification The subset of categorical vectors in The classification vector selected in The distance between.

[0214] In an example, the distance may be determined as, but not limited to, two vectors (i.e., feature vectors and classification vector ). Alternatively, the distance can be determined as where parameter α is a predefined constant and parameter N indicates the eigenvector and classification vector The dimension (e.g., size) of .

[0215] In one embodiment, the comparison may determine, but is not limited to, (i) whether the distance is less than or equal to a threshold value (e.g., ) or (ii) whether the distance is greater than or equal to a threshold (e.g., ) where {} represents the elements of the corresponding set.

[0216] In some embodiments, the use of neighboring reconstructed samples (e.g., feature indicators of neighboring reconstructed samples) to select transform candidates from a subset of transform sets may be restricted. For certain subsets of transform sets selected using coded information, such as prediction modes of blocks (e.g., intra prediction mode, inter prediction mode), the transform candidate or transform kernel identification process does not use feature indicators of neighboring reconstructed samples (e.g., feature vectors Or feature scalar S). Alternatively, the transform candidates may be identified by one or more associated indices signaled in the coded video bitstream. In an example, each of certain transform set subsets has a reduced number of available transform sets and / or transform kernels. In an example, the above restrictions apply but are not limited to certain prediction modes, such as vertical SMOOTH mode SMOOTH_V mode and horizontal SMOOTH mode SMOOTH_H. Therefore, for certain transform set subsets selected based on certain prediction modes (e.g., SMOOTH_V and SMOOTH_H), the transform candidates for the block may be identified by one or more associated indices signaled in the coded video bitstream and not determined based on the feature indicator of the block.

[0217] Fig.19 A flowchart outlining a process (1900) according to an embodiment of the present disclosure is shown. The process (1900) can be used to reconstruct blocks, such as CBs, CUs, PBs, TBs, TUs, luma blocks (e.g., luma CBs or luma TBs), chroma blocks (e.g., chroma CBs or chroma TBs), etc. In various embodiments, the process (1900) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330), and (340), a processing circuit that performs the functions of a video encoder (403), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (510), a processing circuit that performs the functions of a video encoder (603), etc. In some embodiments, the process (1900) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (1900). The process starts at (S1901) and proceeds to (S1910).

[0218] At (S1910), the reconstructed samples in one or more neighboring blocks of the block of the current picture may be based on, for example, feature indicators (e.g., feature vectors) extracted from the reconstructed samples in one or more neighboring blocks of the block. or one of the feature scalars S), determine a transform candidate for the block from the grouped transform sets. Each transform set in the grouped transform sets may include one or more transform candidates for the block. The one or more neighboring blocks may be in the current picture or in a reconstructed picture different from the current picture.

[0219] In an embodiment, a subset of transform sets may be selected from the grouped transform sets based on a prediction mode (e.g., intra prediction mode, inter prediction mode) of the block indicated in the encoded information of the block. The subset of transform sets may be selected based on feature indicators (e.g., feature vectors) extracted from reconstructed samples in one or more neighboring blocks of the block. or one of the feature scalars S) to determine transform candidates from a subset of the transform set.

[0220] In an example, the block may be reconstructed based on feature indicators (e.g., feature vectors) extracted from reconstructed samples in one or more neighboring blocks of the block. The method may select one of the transform sets in the transform set subset (or one of the feature scalars S). A transform candidate for the block may be determined from the selected one of the transform sets in the transform set subset based on an index indicated by a signal in the encoded information.

[0221] In an example, one of the transform sets of the subset of transform sets may be selected based on a second index signaled in the encoded information. The transform set may be selected based on a feature indicator (e.g., a feature vector) extracted from reconstructed samples in one or more neighboring blocks of the block. or one of the feature scalars S) to determine a transform candidate for the block from a selected one of the transform sets of the transform set subset.

[0222] In an example, the block may be reconstructed based on feature indicators (e.g., feature vectors) extracted from reconstructed samples in one or more neighboring blocks of the block. Or one of the feature scalars S) implicitly determines the transform candidates from a subset of the transform set.

[0223] In an example, a feature indicator (eg, a feature vector) is determined based on a statistical analysis of reconstructed samples in one or more neighboring blocks of the block. or feature scalar S). In addition, one of the following operations may be performed: (i) based on the feature indicator (e.g., feature vector or feature scalar S) to determine a transform set from a subset of the transform sets, (ii) to determine a transform set based on a feature indicator (e.g., a feature vector or feature scalar S) to determine a transform candidate from the transform set subset, (iii) selecting a transform set from the transform set subset based on a second index in the encoded information, and selecting a transform set based on a feature indicator (e.g., feature vector Or feature scalar S) determines transform candidates from the selected transform set.

[0224] In an example, the feature indicator is a feature scalar S and is determined as a moment of a variable indicating sample values ​​of reconstructed samples in one or more neighboring blocks of a block. A threshold set K may be predefined s Therefore, one of the following operations can be performed: (i) Based on the moment of the variable and from the threshold set K s The method comprises the steps of: (i) determining a transform set from a subset of transform sets based on a selected threshold, (ii) determining transform candidates from the subset of transform sets based on a moment of the variable and the threshold, or (iii) selecting a transform set from the subset of transform sets based on a second index in the encoded information, and determining transform candidates from the selected transform set based on a moment of the variable and the threshold.

[0225] In an example, the moment of the variable is one of a first-order moment of the variable, a second-order moment of the variable, or a third-order moment of the variable. The prediction mode of the block is one of a plurality of prediction modes. Each of the plurality of prediction modes corresponds to a threshold set K s The unique threshold subset K in s ', which indicates multiple prediction modes and threshold set K s An injective mapping between multiple threshold subsets in .

[0226] Based on one of (i) the block size, (ii) the quantization parameter, or (iii) the prediction mode of the block, a threshold value K is set. s Select a threshold in .

[0227] In the example, the feature indicator is the feature vector And is determined as the joint variability of the variables indicating the sample values ​​of the reconstructed samples of the adjacent columns on the left side of the block and the sample values ​​of the reconstructed samples of the adjacent rows on the upper side of the block, respectively. Predefine the classification vector set Sum and classification vector set The associated threshold set K v . Calculate the joint variability of the variables with the vectors included in the classification set The subset of categorical vectors in The distance between the selected classification vectors. Therefore, you can do one of the following:

[0228] (i) Based on the distance and the number of v The subset K in v ', determine the transformation set from the transformation set subset, (ii) based on the distance and threshold K v and (iii) selecting a transform set from the transform set subset based on a second index in the encoded information, and determining a transform candidate from the selected transform set based on a comparison of the distance with a threshold.

[0229] At (S1920), samples of the block may be reconstructed based on the determined transform candidates.

[0230] The process (1900) may be adjusted appropriately. The step(s) in the process (1900) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of implementation may be used.

[0231] The embodiments of the present disclosure may be used alone or in any order. In addition, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to luminance blocks or chrominance blocks.

[0232] The above techniques can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig. 20 A computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0233] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to mechanisms such as assembly, compilation, linking, etc. to create code comprising instructions that may be executed directly by one or more computer central processing units (CPU), graphics processing units (GPU), etc., or through interpretation, microcode execution, etc.

[0234] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0235] Fig. 20 The components of the computer system (2000) shown in the example are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement related to any one component or combination of components shown in the exemplary embodiment of the computer system (2000).

[0236] The computer system (2000) may include certain human interface input devices. Such human interface input devices may be responsive to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), olfactory input (not depicted). Human interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from a still image camera), video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0237] Input human interface devices may include one or more of the following (only one of each is depicted): keyboard (2001), mouse (2002), trackpad (2003), touch screen (2010), data gloves (not shown), joystick (2005), microphone (2006), scanner (2007), camera (2008).

[0238] The computer system (2000) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (2010), a data glove (not shown), or a joystick (2005), but there may also be tactile feedback devices that are not used as input devices), audio output devices (such as: speakers (2009), headphones (not depicted)), visual output devices (such as screens (2010), to include CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may output two-dimensional visual output or more than three-dimensional output through devices such as stereographic output, virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted).

[0239] The computer system (2000) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2020) including CD / DVD and other media (2021), thumb drives (2022), removable hard drives or solid-state drives (2023), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD-based devices such as security dongles (not depicted), and the like.

[0240] Those skilled in the art will also appreciate that the term "computer-readable media" used in conjunction with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0241] The computer system (2000) may also include an interface (2054) to one or more communication networks (2055). The network may be, for example, wireless, wired, optical. The network may also be local, wide area, urban, vehicle-mounted and industrial, real-time, delay-tolerant, etc. Examples of networks include: local area networks such as Ethernet; wireless LANs; cellular networks including GSM (Global System for Mobile Communication), 3G (Third Generation), 4G (Fourth Generation), 5G (Fifth Generation), LTE (Long Term Evolution), etc.; television wired connections or wireless wide area digital networks including cable television, satellite television and terrestrial broadcast television; vehicle-mounted and industrial networks including CAN buses, etc. Some networks typically require an external network interface adapter that is attached to some general data port or peripheral bus (2049) (e.g., a USB port of a computer system (2000)); other networks are typically integrated into the core of the computer system (2000) by attaching to a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). Using any of these networks, the computer system (2000) can communicate with other entities. Such communications can be one-way receive-only (e.g., broadcast television), one-way send-only (e.g., a CAN bus to some CAN bus devices), or two-way, for example, to other computer systems using local or wide area digital networks. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0242] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (2040) of the computer system (2000).

[0243] The core (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (2043), hardware accelerators for certain tasks (2044), etc. These devices, together with read-only memory (ROM) (2045), random access memory

[0244] (2046), internal mass storage devices such as internal non-user accessible hard disk drives, SSDs

[0245] (2047), etc., can be connected via a system bus (2048). In some computer systems, the system bus (2048) can be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached to the core's system bus (2048) directly or through a peripheral bus (2049). In an example, a screen (2010) can be connected to a graphics adapter (2050). The architecture of the peripheral bus includes PCI, USB, etc.

[0246] The CPU (2041), GPU (2042), FPGA (2043) and accelerator (2044) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (2045) or RAM (2046). Transition data can also be stored in RAM (2046), while permanent data can be stored in, for example, an internal mass storage device (2047). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2041), GPUs (2042), mass storage devices (2047), ROMs (2045), RAMs (2046), etc.

[0247] The computer readable medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0248] As an example and not limitation, a computer system having architecture (2000), and in particular a core (2040), can provide functionality due to a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with a user-accessible mass storage device as described above and a specific storage device of a core (2040) having non-transient properties such as a core internal mass storage device (2047) or ROM (2045). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (2040). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (2040) and specifically the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM (2046) and modifying such a data structure according to a process defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise contained in a circuit (e.g., accelerator (2044)) that may replace software or operate in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits containing logic for execution, or both circuits storing software for execution and circuits containing logic for execution. The present disclosure includes any suitable combination of hardware and software.

[0249] Appendix A: Acronyms

[0250] JEM: Joint Exploration Model

[0251] VVC: Versatile Video Coding

[0252] BMS: Benchmark Set

[0253] MV: Motion Vector

[0254] HEVC: High Efficiency Video Coding

[0255] SEI: Supplemental Enhancement Information

[0256] VUI: Video Availability Information

[0257] GOP: Group of Pictures

[0258] TU: Transform Unit

[0259] PU: prediction unit

[0260] CTU: Coding Tree Unit

[0261] CTB: Coding Tree Block

[0262] PB: prediction block

[0263] HRD: Hypothesized Reference Decoder

[0264] SNR: Signal to Noise Ratio

[0265] CPU: Central Processing Unit

[0266] GPU: Graphics Processing Unit

[0267] CRT: cathode ray tube

[0268] LCD: Liquid Crystal Display

[0269] OLED: Organic Light Emitting Diode

[0270] CD: Compact Disc

[0271] DVD: Digital Video Disc

[0272] ROM: Read Only Memory

[0273] RAM: Random Access Memory

[0274] ASIC: Application-Specific Integrated Circuit

[0275] PLD: Programmable Logic Device

[0276] LAN: Local Area Network

[0277] GSM: Global System for Mobile Communications

[0278] LTE: Long Term Evolution

[0279] CANBus: Controller Area Network Bus

[0280] USB: Universal Serial Bus

[0281] PCI: Peripheral Component Interconnect

[0282] FPGA: Field Programmable Gate Array

[0283] SSD: Solid State Drive

[0284] IC: Integrated Circuit

[0285] CU: Coding Unit

[0286] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various substitute equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method for video decoding in a decoder, characterized in that The method comprises: Based on feature vectors extracted from reconstructed samples in one or more neighboring blocks of the block in the current picture or one of the feature scalars S, determining a transform candidate for the block from a group of transform sets, each transform set in the group of transform sets comprising one or more transform candidates for the block, and one or more neighboring blocks in the current picture or in a reconstructed picture different from the current picture; and reconstructing samples of the block based on the determined transform candidates; The determining of transform candidates for the block from the grouped transform set comprises: The feature vector is determined based on a statistical analysis of the reconstructed samples in one or more neighboring blocks of the block. or one of the characteristic scalars S; selecting a subset of transform sets from the grouped transform sets based on a prediction mode of the block indicated in the encoded information of the block; and Do one of the following (i) Based on the feature vector or one of the characteristic scalars S, determining a transform set from the transform set subset, (ii) Based on the feature vector or one of the feature scalars S, determining the transform candidate from the transform set subset, or (iii) selecting the transform set from the subset of transform sets based on an index in the encoded information, and or one of the feature scalars S determines the transformation candidate from the selected transformation set; The feature vector or one of the characteristic scalars S is the characteristic vector And determine the feature vector Or one of the characteristic scalars S also includes: the characteristic vector The invention comprises the joint variability of variables respectively indicating sample values ​​of reconstructed samples of adjacent columns on the left side of the block and sample values ​​of reconstructed samples of adjacent rows on the upper side of the block, wherein the joint variability is determined by the covariance or second-order moment of variables respectively indicating sample values ​​of reconstructed samples of adjacent columns on the left side of the block and sample values ​​of reconstructed samples of adjacent rows on the upper side of the block.

2. The method according to claim 1, characterized in that The determining the transformation candidates from the transformation set subset comprises: Based on the feature vector extracted from the reconstructed samples in one or more neighboring blocks of the block or one of the feature scalars S, selecting a transform set from the transform set subset; and A transform candidate for the block is determined from a selected one of the subset of transform sets based on an index signaled in the encoded information.

3. The method according to claim 1, characterized in that The determining the transformation candidates from the transformation set subset comprises: selecting one of the subset of transform sets based on an index signaled in the encoded information; and Based on the feature vector extracted from the reconstructed samples in one or more neighboring blocks of the block or one of the feature scalars S, and determining a transform candidate for the block from a selected one of the transform sets in the transform set subset.

4. The method according to claim 1, characterized in that: The determining the transformation candidates from the transformation set subset comprises: Based on the feature vector extracted from the reconstructed samples in one or more neighboring blocks of the block or one of the feature scalars S, and implicitly determining the transform candidate from the transform set subset.

5. The method according to claim 1, characterized in that Predefined classification vector set and the classification vector set The associated threshold set K v , The method further comprises calculating the joint variability of the variables with the variables from the set of vectors included in the classification The subset of categorical vectors in The distance between the selected classification vectors, and The execution includes performing one of the following operations (i) Based on the distance and the number of v The threshold subset K in v ', determining the transform set from the transform set subset, (ii) determining the transform candidate from the transform set subset based on a comparison of the distance with the threshold, or (iii) selecting the transform set from the subset of transform sets based on an index in the encoded information, and determining the transform candidate from the selected transform set based on a comparison of the distance with the threshold.

6. A computer device, characterized in that: The computer device comprises: one or more computer-readable non-transitory storage media configured to store computer program code; and One or more computer processors configured to access the computer program code and execute the method according to any one of claims 1 to 5 as instructed by the computer program code.

7. A video decoding device comprising a processing circuit configured to perform the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer program code, which, when executed by at least one processor, causes the at least one processor to perform the method according to any one of claims 1 to 5.

9. A method for processing a video bitstream, characterized in that: The video bit stream is decoded according to the method according to any one of claims 1 to 5.

Images