Video processing method and device

Abstract

The invention provides a video processing method and device. Processing circuitry decodes information of an encoded block from a bitstream. The information indicates an intra prediction mode of the encoded block, a selection of a separable primary transform from a plurality of separable primary transform candidates, and a selection of a scan mode from a plurality of scan mode candidates. The processing circuitry determines whether a selectable secondary inseparable transform is not applied to the bitstream and a selectable separable primary transform is applied to the bitstream. Thus, the processing circuitry determines a residual for the coded block from the bitstream based on the selection of the scan mode and the selection of the separable primary transform. The processing circuitry reconstructs the coded block based on a combination of the residual of the coded block and the intra prediction. Intra prediction of a coded block is based on an intra prediction mode.

Description

[0001] This application is a divisional application of the Chinese invention patent application whose applicant is Tencent USA LLC, whose filing date is April 20, 2022, whose invention name is “Adaptive scanning with multiple transform selections”, and whose application number is 202280004088.7. Technical Field

[0002] This disclosure describes implementations generally related to video encoding. Background Art

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent the work described in this background section, the work of the presently named inventors and aspects of the description that may not otherwise be qualified as prior art at the time of filing are neither explicitly nor implicitly admitted as prior art to the present disclosure.

[0004] Uncompressed digital video can include a series of pictures, each picture having a spatial dimension of, for example, 1920×1080 luma samples and associated chroma samples. This series of pictures can have a fixed or variable picture rate (also informally referred to as a frame rate), such as 60 pictures per second or 60 Hz. Uncompressed video has specific bit rate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luma sample resolution at 60 Hz frame rate) requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0005] One purpose of video encoding and video decoding can be to reduce the redundancy of the input video signal by compression. Compression can help reduce the above-mentioned bandwidth requirements and / or storage space requirements, in some cases by two orders of magnitude or more. Both lossless compression and lossy compression and their combinations can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal based on the compressed original signal. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small enough so that the reconstructed signal can be used for the intended application. In the case of video, lossy compression is widely used. The amount of distortion that can be tolerated depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that higher allowable / tolerable distortion can produce higher compression ratios.

[0006] Video encoders and video decoders may utilize techniques from several broad categories including, for example, motion compensation, transforms, quantization, and entropy coding.

[0007] Video codec techniques may include techniques known as intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all sample blocks are encoded in intra-frame mode, the picture may be an intra-frame picture. Intra-frame pictures and their derivatives (e.g., independent decoder refresh pictures) may be used to reset decoder states, and may therefore be used as the first picture in an encoded video bitstream and video session or as a still image. Samples of intra-frame blocks may be subjected to transformation, and transform coefficients may be quantized before entropy coding. Intra-frame prediction may be a technique for minimizing sample values ​​in a pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0008] Conventional intra-frame coding, such as known from, for example, MPEG-2 generation coding techniques, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to do so based on metadata and / or surrounding sample data obtained during encoding and / or decoding of, for example, spatially adjacent and preceding data blocks in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and does not use reference data from reference pictures.

[0009] There can be many different forms of intra-frame prediction. When more than one such technique can be used in a given video coding technique, the technique used can be encoded in the intra-frame prediction mode. In some cases, a mode can have sub-modes and / or parameters, and these sub-modes and / or parameters can be encoded separately or included in the mode codeword. Which codeword is used for a given mode, sub-mode and / or parameter combination can have an impact on the coding efficiency gain through intra-frame prediction, and therefore can have an impact on the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced by H.264, refined in H.265, and further refined in newer coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). The predictor block can be formed using neighboring sample values ​​belonging to samples that are already available. Sample values ​​of neighboring samples are copied into the predictor block according to the direction. A reference to the direction used can be encoded in the bitstream or it can itself be predicted.

[0011] ReferenceFigure 1A , a subset of nine predictor directions known from the 33 possible predictor directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes) is depicted at the bottom right. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples at an angle of 45 degrees to the horizontal to the upper right. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples at an angle of 22.5 degrees to the horizontal to the lower left of sample (101).

[0012] Still refer to Figure 1A , a square block (104) of 4×4 samples is depicted at the upper left (indicated by the bold dashed line). The square block (104) includes 16 samples, each of which is labeled by "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is at the lower right. Also shown are reference samples, which follow a similar numbering scheme. The reference samples are labeled with R, their Y position (e.g., row index), and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; therefore, there is no need to use negative values.

[0013] Intra-picture prediction can work by appropriately copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the encoded video bitstream includes signaling that, for this block, indicates a prediction direction consistent with arrow (102) - that is, the sample is predicted based on one or more prediction samples to the upper right at a 45 degree angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0014] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to compute a reference sample; particularly if the directions do not divide evenly at 45 degrees.

[0015] As video coding technology has developed, the number of possible directions has also increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and when disclosed JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent these possible directions with a small number of bits, accepting certain penalties for less likely directions. In addition, the direction itself can sometimes be predicted based on neighboring directions used in neighboring decoded blocks.

[0016] Figure 1B A schematic diagram (110) depicting 65 intra prediction directions according to JEM is shown to illustrate that the number of prediction directions increases over time.

[0017] The mapping of intra-prediction direction bits representing directions in the coded video bitstream may vary from one video coding technique to another; and may range from, for example, a simple direct mapping of prediction directions to intra-prediction modes to codewords, to complex adaptive schemes involving most probable modes and similar techniques. However, in all cases, there will be some directions that are statistically less likely to appear in the video content than some other directions. Since the goal of video compression is to reduce redundancy, in a video coding technique that works well, those less probable directions will be represented by a larger number of bits than more probable directions.

[0018] Video encoding and decoding may be performed using inter-picture prediction with motion compensation. Motion compensation may be a lossy compression technique and may involve the following technique: a block of sample data from a previously reconstructed picture or a portion thereof (reference picture) is used to predict a newly reconstructed picture or picture portion after spatial shifting in a direction indicated by a motion vector (hereinafter referred to as MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. MV may have two dimensions, X and Y, or may have three dimensions, the third dimension being an indication of the reference picture in use (the third dimension may indirectly be a temporal dimension).

[0019] In some video compression techniques, an MV applicable to a particular region of sample data can be predicted based on other MVs, for example, based on an MV related to another region of sample data that is spatially adjacent to the region being reconstructed and that precedes the MV in decoding order. Doing so can significantly reduce the amount of data required to encode the MV, thereby eliminating redundancy and improving compression. For example, since when encoding an input video signal obtained from a camera (referred to as natural video), there is a statistical possibility that a region larger than the region to which a single MV applies moves in a similar direction, and thus similar motion vectors obtained from MVs of neighboring regions can be used for prediction in some cases, MV prediction can work effectively. This makes the MV found for a given region similar or identical to the MV predicted from the surrounding MVs, and can then be represented with a smaller number of bits after entropy coding than would be used if the MV was encoded directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from an original signal (i.e., a sample stream). In other cases, MV prediction itself can be lossy, for example due to rounding errors when calculating a predictor based on several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Among the multiple MV prediction mechanisms provided by H.265, this document describes a technique hereinafter referred to as "spatial merging".

[0021] Reference Figure 2 , the current block (201) includes samples that have been discovered by the encoder during the motion search process that can be predicted from a previous block of the same size that has been spatially shifted. Instead of encoding the MV directly, the MV can be derived from metadata associated with one or more reference pictures, for example, the most recent (in decoding order) reference picture, using the MV associated with any of the five surrounding samples denoted by A0, A1 and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use a predictor from the same reference picture that a neighboring block is using. Summary of the invention

[0022] Various aspects of the present disclosure provide methods and devices for video encoding / decoding. In some examples, the device for video decoding includes a receiving circuit system and a processing circuit system. For example, the processing circuit system decodes information of a coding block from a bitstream. The information indicates an intra-frame prediction mode of the coding block, a selection of a separable primary transform from a plurality of separable primary transform candidates, and a selection of a scanning mode from a plurality of scanning mode candidates. The processing circuit system determines whether an optional secondary inseparable transform is not applied to the bitstream and whether an optional separable primary transform is applied to the bitstream. In response to determining that the secondary inseparable transform is not applied to the bitstream and the optional separable primary transform is applied to the bitstream, the processing circuit system determines the residual of the coding block from the bitstream based on the selection of the scanning mode and the selection of the separable primary transform. The processing circuit system reconstructs the coding block based on a combination of the residual of the coding block and the intra-frame prediction. The intra-frame prediction of the coding block is based on the intra-frame prediction mode.

[0023] In some embodiments, the processing circuitry decodes the bitstream to determine transform coefficients in a transform domain based on the selection of the scan pattern, and applies an inverse transform associated with the separable primary transform to the transform coefficients in the transform domain to determine a residual.

[0024] In some embodiments, the information further indicates that the selectable secondary non-separable transform is excluded and the processing circuitry may determine the residual of the coding block without applying an inverse transform associated with the selectable secondary non-separable transform.

[0025] In some examples, the scanning mode is applied to a subset of the plurality of separable primary transform candidates respectively. In some examples, the scanning mode is applied to each of the plurality of separable primary transform candidates respectively.

[0026] In some examples, the processing circuitry may decode a value of the syntax from the bitstream, and the value of the syntax indicates a combination of a selection of a separable primary transform from a plurality of separable primary transform candidates and a selection of a scan mode from a plurality of scan mode candidates.

[0027] In some examples, the processing circuit system may decode a value of a first syntax and a value of a second syntax from the bitstream. The value of the first syntax indicates a selection of a separable primary transform. The value of the second syntax indicates a selection of a scan mode.

[0028] In some examples, the processing circuitry infers the selection of the scan mode. In an example, the processing circuitry infers the selection of the scan mode based on the intra prediction mode. In addition, in an example, the processing circuitry infers the selection of the separable primary transform. In another example, the processing circuitry decodes a value of a syntax from the bitstream, and the value of the syntax indicates the selection of the separable primary transform.

[0029] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method for video decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:

[0031] Figure 1A is a schematic illustration of an exemplary subset of intra prediction modes.

[0032] Figure 1B is a diagram of exemplary intra prediction directions.

[0033] Figure 2 is a schematic illustration of a current block and its surrounding spatial merging candidates in an example.

[0034] Figure 3 is a schematic illustration of a simplified block diagram of a communication system (300) according to an embodiment.

[0035] Figure 4 is a schematic illustration of a simplified block diagram of a communication system (400) according to an embodiment.

[0036] Figure 5 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment.

[0037] Figure 6 is a schematic illustration of a simplified block diagram of an encoder according to an embodiment.

[0038] Figure 7 A block diagram of an encoder according to another embodiment is shown.

[0039] Figure 8 A block diagram of a decoder according to another embodiment is shown.

[0040] Fig. 9 A set of three scanning patterns for transform coefficients within a transform sub-block according to some embodiments of the present disclosure is shown.

[0041] Fig.10 A set of two scanning patterns for transforming sub-blocks according to some embodiments of the present disclosure is shown.

[0042] Fig.11 A diagram of a residual coding module in some examples is shown.

[0043] Fig.12 Some example transform basis functions are shown.

[0044] Fig.13 Tables for multiple transform selection (MTS) in some examples are shown.

[0045] Fig.14 A diagram of a residual coding module in some examples is shown.

[0046] Fig.15 A diagram of a residual coding module according to some embodiments of the present disclosure is shown.

[0047] Fig.16 Tables are shown to illustrate transform and scan signaling mapping tables according to some examples.

[0048] Fig.17 Another table is shown for illustrating transform and scan signaling mapping tables according to some examples.

[0049] Fig.18 Another table is shown for illustrating transform and scan signaling mapping tables according to some examples.

[0050] Fig.19 A flow chart outlining a process according to an embodiment of the present disclosure is shown.

[0051] Fig. 20 A flow chart outlining a process according to an embodiment of the present disclosure is shown.

[0052] Fig.21 is a schematic illustration of a computer system according to an embodiment. DETAILED DESCRIPTION

[0053] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3 In the example of , a first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) can encode video data (e.g., a video picture stream captured by the terminal device (310)) for transmission to another terminal device (320) via a network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to recover the video picture, and display the video picture based on the recovered video data. Unidirectional data transmission may be common in media service applications, etc.

[0054] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which bidirectional transmission can occur, for example, during a video conference. For the bidirectional transmission of data, in the example, each of the terminal devices (330) and (340) can encode video data (e.g., a video picture stream captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), and can decode the encoded video data to restore the video picture, and can display the video picture at an accessible display device based on the restored video data.

[0055] exist Figure 3 In the example of , terminal devices (310), (320), (330) and (340) can be shown as servers, personal computers and smart phones, but the principles of the present disclosure may not be limited to this. Implementations of the present disclosure may be applied to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. Communication network (350) can exchange data in circuit switching channels and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purposes of this discussion, unless otherwise specified below, the architecture and topology of network (350) may be unimportant to the operation of the present disclosure.

[0056] As examples of applications of the disclosed subject matter, Figure 4 The placement of a video encoder and a video decoder in a streaming environment is shown. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc., etc.

[0057] The streaming system may include a capture subsystem (413), which may include a video source (401), such as a digital camera, that creates, for example, an uncompressed video picture stream (402). In an example, the video picture stream (402) includes samples captured by the digital camera. The video picture stream (402) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (404) (or encoded video bitstream), and the video picture stream (402) may be processed by an electronic device (420) coupled to the video source (401) including a video encoder (403). The video encoder (403) may include hardware, software, or a combination thereof to implement or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower amount of data when compared to the video picture stream (402), and the encoded video data (404) can be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4 The client subsystems (406) and (408) in the video transmission system can access the streaming server (405) to retrieve the copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). In some streaming transmission systems, the encoded video data (404), (407) and (409) (e.g., a video bitstream) can be encoded according to certain video coding standards / video compression standards. Examples of these standards include ITU-T Recommendation H.265. In the example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC.

[0058] Note that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0059] Figure 5A block diagram of a video decoder (510) according to an embodiment of the present disclosure is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used to replace Figure 4 A video decoder (410) is shown in an example.

[0060] A receiver (531) may receive one or more encoded video sequences to be decoded by a video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data with other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not depicted). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and an entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) (not depicted). In still other applications, there may be a buffer memory (not depicted) external to the video decoder (510) to, for example, prevent network jitter, and there may be another buffer memory (515) internal to the video decoder (510) to, for example, handle playout timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (515) may not be needed, or the buffer memory (515) may be small. For use over an optimal packet network such as the Internet, the buffer memory (515) may be needed, the buffer memory (515) may be relatively large and may advantageously be of an adaptable size, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0061] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include: information for managing the operation of the video decoder (510); and potential information for controlling a presentation device such as a presentation device (512) (e.g., a display screen), which is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5 As shown. The control information for (one or more) rendering devices may be in the form of a Supplemental Enhancement Information (SEI message) or a Video Usability Information (VUI) parameter set fragment (not depicted). The parser (520) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or a video coding standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Picture (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (520) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0062] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0063] Depending on the type of the coded video picture or part of the coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). For the sake of brevity, this subgroup control information flow between the parser (520) and the following multiple units is not described.

[0064] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the following functional units.

[0065] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as symbol(s) (521) from the parser (520) and control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) may output a block including sample values ​​which may be input into an aggregator (555).

[0066] In some cases, the output samples of the scaler / inverse transform (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding reconstructed information obtained from a current picture buffer (558). For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information already generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0067] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to a block that has been inter-coded and possibly motion compensated. In such a case, the motion compensated prediction unit (553) may access the reference picture memory (557) to obtain samples for prediction. After the obtained samples are motion compensated according to the symbols (521) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (551) (in this case, called residual samples or residual signal) by the aggregator (555) to generate output sample information. The address within the reference picture memory (557) from which the motion compensated prediction unit (553) obtains the predicted samples may be controlled by a motion vector, which is provided to the motion compensated prediction unit (553) in the form of a symbol (521), which may have, for example, an X component, a Y component, and a reference picture component. Motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0068] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as a coded video bitstream) that are available to the loop filter unit (556) as symbols (521) from the parser (520), but the video compression techniques may also be responsive to meta-information obtained during decoding of a previous portion (in decoding order) of a coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0069] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0070] Once fully reconstructed, certain coded pictures may be used as reference pictures for future predictions. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557) and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.

[0071] The video decoder (510) may perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows both the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as tools that can only be used under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and the metadata of the HRD buffer management signaled in the encoded video sequence.

[0072] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence (one or more). The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0073] Figure 6 A block diagram of a video encoder (603) according to an embodiment of the present disclosure is shown. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit system). The video encoder (603) can be used to replace Figure 4 The video encoder (403) in the example.

[0074] The video encoder (603) can be used to obtain the video source (601) (the video source (601) is not Figure 6 In one example, the video source (601) is a part of the electronic device (620) to receive the video samples, and the video source (601) can capture (one or more) video images to be encoded by the video encoder (603). In another example, the video source (601) is a part of the electronic device (620).

[0075] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera device that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be organized as a spatial pixel array, wherein each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be readily understood by those skilled in the art. The following description focuses on samples.

[0076] According to an embodiment, the video encoder (603) can encode and compress the pictures of the source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing a suitable encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to other functional units. For the sake of brevity, the coupling is not depicted. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology...), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions that belong to the video encoder (603) optimized for a certain system design.

[0077] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As an oversimplified description, in an example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols such as a symbol stream based on an input picture to be encoded and (one or more) reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how a (remote) decoder would create sample data (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0078] The operation of the "local" decoder (633) can be combined with the above Figure 5 The operation of the "remote" decoder described in detail is identical to that of the video decoder (510). However, reference is also briefly made to Figure 5 Since the symbols are available and the entropy encoder (645) and the parser (520) can encode / decode the symbols into the encoded video sequence without loss, the entropy decoding part of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633).

[0079] At this point it can be observed that any decoder technology other than parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on the decoder operation. Since the encoder technology is reciprocal to the decoder technology that has been fully described, the description of the encoder technology can be simplified. A more detailed description is only required in certain areas and will be provided below.

[0080] During operation, in some examples, the source encoder (630) may perform motion compensated predictive encoding that predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as “reference pictures.” In this manner, the encoding engine (632) encodes the differences between pixel blocks of an input picture and pixel blocks of reference picture(s) that may be selected as prediction reference(s) for the input picture.

[0081] The local video decoder (633) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 6 When the video encoder (603) is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture, and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture that will be obtained by the remote video decoder.

[0082] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may be used as appropriate prediction references for the new picture. The predictor (635) may operate on a pixel block by pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (634).

[0083] The controller (650) can manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0084] The outputs of all the above-mentioned functional units may be subjected to entropy coding in the entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0085] The transmitter (640) can buffer the encoded video sequence(s) created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which can be a hardware / software link to a storage device storing the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0086] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:

[0087] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of those variations of I pictures and their corresponding applications and features.

[0088] A predictive picture (P picture) may be a picture that may be encoded and decoded using intra prediction or inter prediction, which predicts sample values ​​of each block using at most one motion vector and a reference index.

[0089] Bidirectional predictive pictures (B pictures) can be pictures that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0090] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively) and encoded block by block. The blocks may be predictively encoded with reference to other (already encoded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively encoded, or may be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same picture. Pixel blocks of a P picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one previously encoded reference picture. Blocks of a B picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures.

[0091] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In its operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0092] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0093] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simply referred to as intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an example, a particular picture in encoding / decoding, which is referred to as the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that has been previously encoded in the video and is still buffered, the block in the current picture can be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case of using multiple reference pictures, the motion vector may have a third dimension that identifies the reference picture.

[0094] In some embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be predicted by a combination of the first reference block and the second reference block.

[0095] In addition, merge mode techniques can be used for inter-picture prediction to improve coding efficiency.

[0096] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), i.e., a luminance CTB and two chrominance CTBs. Each CTU can be recursively split into one or more coding units (CUs) with a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In the example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. Depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0097] Figure 7 A diagram of a video encoder (703) according to another embodiment of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a predicted block) of sample values ​​within a current video picture in a sequence of video pictures, and encode the processed block into an encoded picture that is part of an encoded video sequence. In an example, the video encoder (703) is used instead of Figure 4 The video encoder (403) in the example.

[0098] In the HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion optimization to determine whether to best encode the processing block using intra mode, inter mode, or bi-prediction mode. In the case where the processing block is to be encoded in intra mode, the video encoder (703) may encode the processing block into an encoded picture using intra prediction techniques; and in the case where the processing block is to be encoded in inter mode or bi-prediction mode, the video encoder (703) may encode the processing block into an encoded picture using inter prediction or bi-prediction techniques, respectively. In some video coding techniques, the merge mode may be an inter-picture prediction sub-mode, where the motion vector is derived from an encoded motion vector component external to one or more motion vector predictors without the aid of the predictor. In some other video coding techniques, there may be a motion vector component applicable to the subject block. In the example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0099] exist Figure 7 In this example, the video encoder (703) includes Figure 7 An inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) are shown coupled together.

[0100] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0101] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with an already encoded block in the same picture in some cases, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., generate intra prediction direction information according to one or more intra coding techniques). In an example, the intra encoder (722) also calculates an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0102] The general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select the intra prediction information and include the intra prediction information in the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select the inter prediction information and include the inter prediction information in the bitstream.

[0103] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate a transform coefficient. In an example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate a transform coefficient. Then, the transform coefficient is subjected to quantization to obtain a quantized transform coefficient. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. In some examples, the decoded blocks are processed appropriately to generate decoded pictures, and these decoded pictures can be buffered in memory circuits (not shown) and used as reference pictures.

[0104] The entropy encoder (725) is configured to format the bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information according to a suitable standard, such as the HEVC standard. In an example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Note that according to the disclosed subject matter, when the block is encoded in the inter-frame mode or the merge sub-mode of the bi-prediction mode, there is no residual information.

[0105] Figure 8A diagram of a video decoder (810) according to another embodiment of the present disclosure is shown. The video decoder (810) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In an example, the video decoder (810) is used instead of Figure 4 A video decoder (410) is shown in an example.

[0106] exist Figure 8 In this example, the video decoder (810) includes Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874) and an intra-frame decoder (872) are shown coupled together.

[0107] The entropy decoder (871) can be configured to reconstruct certain symbols representing syntax elements constituting the encoded picture from the encoded picture. Such symbols may include, for example, a mode for encoding a block (such as, for example, an intra-frame mode, an inter-frame mode, a bi-prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (such as, for example, intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata used for prediction by an intra-frame decoder (872) or an inter-frame decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an example, when the prediction mode is an inter-frame mode or a bi-prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be subjected to inverse quantization and provided to the residual decoder (873).

[0108] The inter-frame decoder (880) is configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0109] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0110] The residual decoder (873) is configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (Quantizer Parameter, QP)), and this information can be provided by the entropy decoder (871) (since this is only low-volume control information, the data path is not depicted).

[0111] The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) with the prediction result (output by the inter-frame prediction module or the intra-frame prediction module, as the case may be) in the spatial domain to form a reconstructed block, which can be part of a reconstructed picture, which in turn can be part of a reconstructed video. Note that other suitable operations such as deblocking operations can be performed to improve visual quality.

[0112] Note that the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using any suitable technology. In an embodiment, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603) and (603) and the video decoders (410), (510) and (810) may be implemented using one or more processors that execute software instructions.

[0113] Various aspects of the present disclosure provide techniques for adaptive scanning with multiple transform options. During residual coding, scanning of transform coefficients with multiple transform options can be adaptively performed. The scanned transform coefficients can be quantized and entropy encoded.

[0114] According to some aspects of the present disclosure, in order to encode the predicted residual, the CU is appropriately divided into transform units (TUs), which can be divided into a luminance transform block (e.g., including the residual after the luminance component is predicted) and a chrominance transform block (e.g., including the residual after the chrominance component is predicted) according to the luminance and chrominance components. The transform block before the transform may include the residual in the spatial domain, and the transform block in the spatial domain may be transformed into a transform block in the transform domain. The transform block in the transform domain includes transform coefficients in the transform domain. The transform block in the transform domain may be scanned according to a scanning pattern to convert the transform coefficients in a two-dimensional (2D) form into transform coefficients in a one-dimensional (1D) form. Then, the transform coefficients in the 1D form may be quantized and entropy encoded into bits, and these bits may be combined with other appropriate information into a bit stream carrying the encoded video.

[0115] According to some aspects of the present disclosure, various transform and scanning techniques may be applied to residual coding.

[0116] In some examples, for a transform block, only one transform candidate is available for transforming the residual signal into a transform domain. In an example (e.g., HEVC), a transform called type 2 discrete cosine transform (DCT-2) is used to transform the residual signal into a transform domain. For example, DCT-2 can be implemented by applying a transform basis function of DCT-2 in the horizontal and vertical directions. In some examples, DCT-2 can be implemented using a transform matrix. In some examples, the transform block can have four sizes such as 4×4, 8×8, 16×16, and 32×32, and then four integer DCT-2 transform matrices are used for transforming during residual encoding for each of the four sizes of transform blocks. The elements of the transform matrix can be represented using 8-bit integers, and therefore these transform matrices are referred to as 8-bit transform cores in some examples.

[0117] Note that in the example, residual coding of a 4x4 block for intra prediction may use a transform matrix different from the 4x4 matrix of DCT-2, such as a matrix known as a 4x4 integer DST-6 matrix.

[0118] To scan the transform coefficients of a transform block in the transform domain, in some examples, a transform block larger than 4×4 is divided into non-overlapping transform sub blocks (TSBs) each having 4×4 transform coefficients. For ease of description, a transform block of 4×4 transform coefficients may be referred to as a transform sub block. In some examples, a scan pattern of transform coefficients in a transform block in the transform domain may be determined based on a scan pattern of transform coefficients within a transform sub block and a scan pattern of transform sub blocks in a transform block.

[0119] Fig. 9 A set of three scanning patterns (910), (920), and (930) of transform coefficients within a transform sub-block are shown according to some embodiments of the present disclosure.

[0120] exist Fig. 9 In the scanning pattern (910), the scanning pattern (910) may be referred to as a reverse diagonal scanning pattern in a transform subblock. In the scanning pattern (910), each small square represents a transform coefficient in a transform subblock, and the numbers in the small squares represent a scanning order in a diagonal scan that scans diagonally from the upper left corner to the lower right corner of the transform block, for example, from 0 to 15. The reverse diagonal scanning pattern scans the transform coefficients in the reverse order of the diagonal scan, for example, from 15 to 0 as indicated by the arrow.

[0121] exist Fig. 9In the scanning pattern (920), the scanning pattern (920) may be referred to as a reverse horizontal scanning pattern in a transform sub-block. In the scanning pattern (920), each small square represents a transform coefficient in a transform sub-block, and the numbers in the small squares represent a scanning order in a horizontal raster scan that scans from left to right and from top to bottom, for example, from 0 to 15. The reverse horizontal scanning pattern scans the transform coefficients in the reverse order of the horizontal raster scan, for example, from 15 to 0 as indicated by the arrow.

[0122] exist Fig. 9 In the scanning pattern (930), the scanning pattern (930) may be referred to as a reverse vertical scanning pattern in a transform sub-block. In the scanning pattern (930), each small square represents a transform coefficient in a transform sub-block, and the numbers in the small squares represent the scanning order in a vertical raster scan that scans from top to bottom and from left to right, for example, from 0 to 15. The reverse vertical scanning pattern scans the transform coefficients in the reverse order of the vertical raster scan, for example, from 15 to 0 as indicated by the arrow.

[0123] Fig.10 A set of two scanning patterns (1010) and (1020) for transforming sub-blocks according to some embodiments of the present disclosure is shown.

[0124] exist Fig.10 , the scanning pattern (1010) may be referred to as a reverse diagonal (or vertical) scanning pattern of four transform subblocks in an 8×8 transform block. In the scanning pattern (1010), each small square represents a transform subblock in the transform block, and the numbers in the small squares represent the scanning order of the transform subblocks in a diagonal (vertical raster) scan that is scanned diagonally (or in a vertical raster order) from the upper left corner of the transform block to the lower right corner of the transform block, for example, from 0 to 3. The reverse diagonal (vertical) scanning pattern scans the transform subblocks in the transform block in the reverse order of the diagonal (vertical) scanning, for example, from 3 to 0 as indicated by the arrows.

[0125] exist Fig.10 In the scanning pattern (1020), the scanning pattern (1020) may be referred to as a reverse horizontal scanning pattern of four transform sub-blocks in an 8×8 transform block. In the scanning pattern (1020), each small square represents a transform sub-block in the transform block, and the numbers in the small squares represent the scanning order of the transform sub-blocks in a horizontal raster scan that scans the transform sub-blocks from left to right and from top to bottom, for example, from 0 to 3. The reverse horizontal scanning pattern scans the transform sub-blocks in the transform block in the reverse order of the horizontal raster scan, for example, from 3 to 0 as indicated by the arrow.

[0126] In some embodiments, for inter-coded blocks, reverse diagonal scanning is used for all block sizes. For all block sizes, the scanning order of transform subblocks in a transform block follows the same pattern as the scanning of coefficients within a TSB. For example, the scanning pattern (910) for transform coefficients in a TSB and the scanning pattern (1010) for a TSB can be used for transform blocks of an 8×8 inter-coded block.

[0127] For intra-coded blocks of size 4×4 or 8×8, three scanning modes (910), (920) and (930) are available. In some embodiments, one of the three scanning modes (910), (920) and (930) can be selected based on the direction of the intra-prediction mode. Note that the ordering of the transform subblocks in the transform block follows the same pattern as the coefficient scanning within the TSB. In an example, the scanning mode (910) of the transform coefficients in the TSB and the scanning mode (1010) of the TSB can be used for the transform block of an 8×8 intra-coded block. In another example, the scanning mode (920) of the transform coefficients in the TSB and the scanning mode (1020) of the TSB can be used for the transform block of an 8×8 intra-coded block. In another example, the scanning mode (930) of the transform coefficients in the TSB and the scanning mode (1010) of the TSB can be used for the transform block of an 8×8 intra-coded block.

[0128] In some embodiments, for intra-coded blocks of size 16×16 and 32×32, reverse diagonal scanning is used for all block sizes. In some examples, the scanning order of transform subblocks in a transform block follows the same pattern as the scanning of coefficients within a TSB. For example, the scanning pattern (910) for transform coefficients in a TSB and a scanning pattern similar to (910) for scanning a TSB (e.g., a 4×4 TSB) can be used for transform blocks of 16×16 intra-coded blocks.

[0129] Fig.11 A diagram of a residual encoding module (1100) in some examples is shown. For example, the residual encoding module (1100) can be used to encode transform coefficients from (one or more) intra-frame prediction residuals in HEVC. For the size of the intra-frame coding block, the transform (e.g., the transform matrix) is fixed based on the size of the intra-frame coding block, and the scanning mode can vary according to the intra-frame prediction mode.

[0130] Specifically, in Fig.11 In this example, the residual coding module (1100) includes Fig.11 Shown are a transform module (1110), an optional scan module (1170), a quantizer module (1180), and an entropy coding module (1190) coupled together.

[0131] The transform module (1110) receives the intra prediction residual (e.g., generated by subtracting the intra prediction from the original sample) and divides the intra prediction residual into TBs in the spatial domain. The transform module (1110) can perform a transform (e.g., fixed transform matrix, DCT-2 in the horizontal and vertical directions) to convert the TBs in the spatial domain into TBs in the transform domain.

[0132] The selectable scanning module (1170) receives the TB in the transform domain and a select scanning signal indicating a selected scanning mode. The TB in the transform domain includes transform coefficients in 2D form. The scanning module (1170) can use the selected scanning mode to convert the transform coefficients in 2D form into transform coefficients in 1D form.

[0133] The quantizer module (1180) may perform quantization and scaling to compress transform coefficients to generate transform coefficient levels.

[0134] The entropy coding module (1190) can perform binarization of the transform coefficient level and generate coded bits representing the transform coefficient level.

[0135] In some examples, Fig.11 The modules in the embodiment can be implemented using integrated circuits (ICs) to perform the functions of the modules. Fig.11 The modules in the description may be respectively implemented as software instructions that can be executed by one or more processors to perform the functions of the modules.

[0136] Although in the above examples (e.g., HEVC), the transforms used to encode intra-coded blocks and inter-coded blocks are fixed, in some other examples, a technique called multiple transform selection (MTS) can be used for residual encoding of both inter-coded blocks and intra-coded blocks, where the transform is selected from multiple transform candidates (e.g., transform matrices of the same size, transform basis functions). In some examples, the multiple transform candidates may include DCT-2, type 7 discrete sine transform (DST-7), and type 8 discrete cosine transform (DCT-8). In some examples, MTS is applied only to residual encoding of luma blocks.

[0137] Fig.12 Transform base functions for DCT-2, DST-7, and DCT-8 are shown in some examples. In some examples, a separable transform refers to a transform that can be applied separately as a horizontal transform and a vertical transform. In some examples, the horizontal transform can be selected from DCT-2, DST-7, and DCT-8 transform base functions, and the vertical transform can be selected from DCT-2, DST-7, and DCT-8 transform base functions. In some examples, a non-separable transform refers to a transform that cannot be separated into a vertical transform and a horizontal transform.

[0138] According to one aspect of the present disclosure, the selection of a transform from the transform candidates can be implicitly or explicitly signaled from the encoder side to the decoder side. In the example of implicit MTS, a combination of DCT-2 and DST-7 can be derived based on the width and height of the current TB. In the example, when the width value of the current TB is greater than the height value, DCT-2 is used to perform the transform in the horizontal direction, and DST-7 is used to perform the transform in the vertical direction; when the width value of the current TB is less than the height value, DCT-2 is used to perform the transform in the vertical direction, and DST-7 is used to perform the transform in the horizontal direction. In another example, when the width value of the current TB is greater than the height value, DST-7 is used to perform the transform in the horizontal direction, and DCT-2 is used to perform the transform in the vertical direction; when the width value of the current TB is less than the height value, DST-7 is used to perform the transform in the vertical direction, and DCT-2 is used to perform the transform in the horizontal direction.

[0139] In some examples, to control the MTS scheme, separate enable flags are specified at the SPS level for intra and inter, respectively. In some embodiments, when MTS is enabled at the SPS, a CU-level flag represented by mts_idx is signaled to indicate how MTS is applied.

[0140] Fig.13 A table (1300) showing the application of MTS based on the CU level flag mts_idx in some examples is shown. Each value of mts_idx is associated with a horizontal transform and a vertical transform. For example, when the value of mts_idx is 0, MTS is not applied, and DCT-2 is used to perform the transforms in the horizontal and vertical directions; when the value of mts_idx is 1, DST-7 is used to perform the transform in the horizontal direction, and DST-7 is used to perform the transform in the vertical direction; when the value of mts_idx is 2, DCT-8 is used to perform the transform in the horizontal direction, and DST-7 is used to perform the transform in the vertical direction; when the value of mts_idx is 3, DST-7 is used to perform the transform in the horizontal direction, and DCT-8 is used to perform the transform in the vertical direction; and when the value of mts_idx is 4, DCT-8 is used to perform the transform in the horizontal direction, and DCT-8 is used to perform the transform in the vertical direction.

[0141] Note that in some embodiments, MTS is only applied to residual coding of luma blocks. For example, only DCT-2 is used for residual coding of chroma blocks.

[0142] According to one aspect of the present disclosure, MTS signaling may be skipped in some examples. In an example, MTS signaling is skipped when the position of the last significant coefficient of the luma TB is less than 1 (e.g., has only a DC component). In another example, MTS signaling is skipped when the last significant coefficient of the luma TB is located within an MTS return-to-zero region (e.g., a region where coefficients in the luma TB are set to zero by a video encoder).

[0143] According to some aspects of the present disclosure, the encoding of the residual signal may include a primary transform and a secondary transform. In some examples (e.g., VVC), DCT-2 is applied to intra-coded blocks (in the horizontal and vertical directions) as a primary transform to generate frequency components, and a low-frequency non-separable transform (LFNST (low-frequency non-separable transform, LFNST)) is applied as a secondary transform to the low-frequency components output from the primary transform to better utilize the directional characteristics of the intra-coded blocks.

[0144] Note that LFNST is an inseparable transform that cannot be separated into a vertical transform and a horizontal transform. In general, using a separable transform performed as a horizontal transform and a vertical transform can capture the correlation in the horizontal direction and the vertical direction; and using an inseparable transform can additionally capture the correlation in other directions such as the diagonal direction, etc.

[0145] According to one aspect of the present disclosure, LFNST may be mode-dependent, and a transform (e.g., matrix) to which LFNST is applied may be selected based on an intra-picture prediction mode of an intra-coded block. LFNST may be applied between a primary transform (also referred to as a forward primary transform on the encoder side) and quantization on the encoder side, and LFNST may be applied between inverse quantization and inverse primary transform on the decoder side. In some examples, when LFNST is applied, the CU level flag mts_idx is not signaled, and mts_idx is inferred to be 0 to indicate that DCT2 is used for both horizontal and vertical transforms (in the primary transform).

[0146] Fig.14 A diagram of a residual coding module (1400) in some examples is shown. For example, the residual coding module (1400) can be used to encode transform coefficients from intra-frame prediction residuals in VVC. The residual coding module (1400) can apply an optional primary transform to the intra-frame prediction residual (the residual after intra-frame prediction), or can apply a non-selectable primary transform and an optional secondary transform to the intra-frame prediction residual. The residual coding module (1400) can apply a fixed scan, such as an inverse diagonal scan, to convert 2D transform coefficients into 1D transform coefficients.

[0147] Specifically, in Fig.14 In this example, the residual coding module (1400) includes Fig.14 Shown are an optional primary transform module (1410), a fixed primary transform module (1420), an optional secondary transform module (1430), a fixed scanning module (1470), a quantizer module (1480) and an entropy coding module (1490) coupled together.

[0148] The optional primary transform module (1410) can receive an intra prediction residual (e.g., generated by subtracting the intra prediction from the original sample) and divide the intra prediction residual into TBs in the spatial domain. The optional primary transform module (1410) also receives a selected transform signal indicating a selected transform, such as a transform function to be applied in the horizontal and vertical directions. The optional primary transform module (1410) can perform a transform according to the selected transform signal to convert the TB in the spatial domain into a TB in the transform domain.

[0149] The fixed primary transform module (1420) may receive the intra prediction residual (e.g., generated by subtracting the intra prediction from the original samples) and divide the intra prediction residual into TBs in the spatial domain. The fixed primary transform module (1420) may perform a fixed transform, such as DCT-2 in the horizontal and vertical directions, to convert the TBs in the spatial domain into intermediate TBs in the frequency domain.

[0150] In some examples, the selectable secondary transform module (1430) is configured to perform LFNST. The selectable secondary transform module (1430) may receive the intermediate TBs and a selected secondary transform signal indicating a selected secondary transform, such as a selected LFNST. The selectable secondary transform module (1430) may perform the selected LFNST on the intermediate TBs and generate a TB in the transform domain.

[0151] The fixed scan module (1470) may receive TBs in the transform domain from the selectable primary transform module (1410) or from the selectable secondary transform module (1430). The fixed scan module (1470) applies a fixed scan pattern, such as the reverse diagonal scan pattern shown by (910) and (1010), to convert the transform coefficients in 2D form into transform coefficients in 1D form.

[0152] The quantizer module (1480) may perform quantization and scaling to compress transform coefficients to generate transform coefficient levels.

[0153] The entropy coding module (1490) can perform binarization of the transform coefficient level and generate coded bits representing the transform coefficient level.

[0154] In some examples, integrated circuits (ICs) may be used to implement Fig.14 In some examples, Fig.14 The modules in the description may be respectively implemented as software instructions that can be executed by one or more processors to perform the functions of the modules.

[0155] exist Fig.14 In the example, the transform coefficients of the coding block are encoded by classifying them into transform sub-blocks (TSBs). The TSB sizes can be 4×4, 1×16, 16×1, 2×8, 8×2, 2×4, and 4×2. In some examples (e.g., VVC), intra sub-partitioning (ISP) can be applied, which may result in narrow luma TBs and small chroma TBs. The transform coefficients within the TSB are encoded in a single reverse diagonal scan order, such as shown by (910), and the TSBs within the TB are encoded in a single reverse diagonal scan order, such as shown by (1010).

[0156] Note that in some examples (e.g., residual coding of intra-predicted blocks by HEVC), the transform is fixed for a given block size and the scanning mode can vary depending on the intra-prediction mode. Since the transform is fixed, the scanning mode does not adapt to the transform. In some other examples (e.g., residual coding of intra-predicted blocks by VVC), the scanning mode is fixed (not adaptive), but multiple transform selection can be applied to improve coding efficiency for both intra-coded blocks and inter-coded blocks.

[0157] According to one aspect of the present disclosure, under certain conditions (e.g., common test conditions for VVC and VTM-11), certain values ​​of mts_idx are used with high probability, such as mts_idx equal to 0 or mts_idx equal to 1, and other values ​​of mts_idx are rarely used. In some embodiments, calculations can be saved at the encoder to evaluate the rate-distortion performance of certain values ​​of mts_idx but not all values ​​of mts_idx. Some aspects of the present disclosure can improve the coding efficiency of a video codec.

[0158] Fig.15A diagram of a residual encoding module (1500) according to some embodiments of the present disclosure is shown. For example, the residual encoding module (1500) can be used to encode transform coefficients from intra-frame prediction residuals at an encoder. The residual encoding module (1500) can include multiple transform and scan processing paths for applying appropriate transform and scan techniques to intra-frame prediction residuals (residuals after intra-frame prediction) to form 1D transform coefficients. For example, the residual encoding module (1500) includes: a first transform and scan processing path (1501), which can apply a selectable primary transform and a selectable scan to the intra-frame prediction residual to form a 1D transform coefficient; and a second transform and scan processing path (1502), which can apply a fixed primary transform, a selectable secondary transform, and a fixed scan to the intra-frame prediction residual to form a 1D transform coefficient.

[0159] Specifically, in Fig.15 In this example, the residual coding module (1500) includes Fig.15 Shown are an optional primary transform module (1510), an optional scanning module (1560), a fixed primary transform module (1520), an optional secondary transform module (1530), a fixed scanning module (1570), a quantizer module (1580) and an entropy coding module (1590) coupled together.

[0160] The optional primary transform module (1510) can receive an intra prediction residual (e.g., generated by subtracting the intra prediction from the original sample) and divide the intra prediction residual into TBs in the spatial domain. The optional primary transform module (1510) also receives a selected transform signal indicating a selected transform, such as a transform function to be applied in the horizontal and vertical directions. The optional primary transform module (1510) can perform a transform according to the selected transform signal to convert the TB in the spatial domain into a TB in the transform domain.

[0161] The optional scanning module (1560) receives the TB in the transform domain output from the optional primary transform module (1510) and a selected scanning signal indicating a selected scanning mode. The TB in the transform domain includes transform coefficients in 2D form. The optional scanning module (1560) can use the selected scanning mode to convert the transform coefficients in 2D form into transform coefficients in 1D form.

[0162] The fixed primary transform module (1520) may receive the intra prediction residual (e.g., generated by subtracting the intra prediction from the original samples) and divide the intra prediction residual into TBs in the spatial domain. The fixed primary transform module (1520) may perform a fixed transform, such as DCT-2 in the horizontal and vertical directions, to convert the TBs in the spatial domain into intermediate TBs in the frequency domain.

[0163] In some examples, the selectable secondary transform module (1530) is configured to perform LFNST. The selectable secondary transform module (1530) may receive the intermediate TB and a selected secondary transform signal indicating a selected secondary transform, such as a selected LFNST. The selectable secondary transform module (1530) may perform the selected LFNST on the intermediate TB and generate a TB in the transform domain.

[0164] The fixed scan module (1570) may receive the TB in the transform domain output from the optional secondary transform module (1530). The fixed scan module (1570) applies a fixed scan pattern, such as the reverse diagonal scan pattern shown by (910) and (1010), to convert the transform coefficients in 2D form into transform coefficients in 1D form.

[0165] The quantizer module (1580) may receive the transform coefficients in a 1D form output from the selectable scanning module (1560) or the fixed scanning module (1570). The quantizer module (1580) may perform quantization and scaling to compress the transform coefficients to generate a transform coefficient hierarchy.

[0166] The entropy coding module (1590) can perform binarization of the transform coefficient level and generate coded bits representing the transform coefficient level.

[0167] In some examples, Fig.15 The modules in the embodiment can be implemented using integrated circuits (ICs) to perform the functions of the modules. Fig.15 The modules in the description may be respectively implemented as software instructions that can be executed by one or more processors to perform the functions of the modules.

[0168] like Fig.15 As shown, the first transform and scan processing path (1501) can apply adaptive scanning (selectable scanning) to the encoding of the intra-frame prediction residual block when the primary transform is selectable and the secondary transform is not enabled. When the primary separable transform is selectable and the secondary transform is not enabled, a scanning mode is selected from a plurality of scanning modes to be applied to the output of at least one of the selected primary transforms, thereby reordering the two-dimensional transform coefficients into one dimension before sending the transform coefficients for quantization.

[0169] According to one aspect of the present disclosure, the scan selection may be signaled implicitly or explicitly. When the scan selection is signaled explicitly, the scan selection for each TB is encoded in the bitstream for the TB. When the scan selection is signaled implicitly, in some examples, the scan selection may be derived from the direction of the intra prediction mode and / or the selection of the primary transform.

[0170] In some embodiments, the primary transform selection and the scan selection are signaled separately.

[0171] In some embodiments, the primary transform selection and the scan selection are jointly signaled.

[0172] In an embodiment, transform selection and scan selection are jointly signaled, and the scan selection is applicable to a subset of selectable primary transforms. Fig. 9 and Fig.10 The inverse diagonal scan mode, the inverse horizontal scan mode, and the inverse vertical scan mode shown may be used for scanning.The transform selection and the scan selection may be jointly encoded, such as by encoding mts_idx based on a transform and scan signaling mapping table.

[0173] Fig.16 A table (1600) is shown for illustrating transform and scan signaling mapping tables according to some examples. In some examples, statistics for intra-coding with explicit MTS (e.g., in VVC) show that most blocks are coded with DCT-2 for both horizontal and vertical transforms or with DST-7 for both horizontal and vertical transforms. The example in table (1600) utilizes statistics and includes the following two transform selections: a first selection: DCT-2 for both horizontal and vertical transforms, and a second selection: DST-7 for both horizontal and vertical transforms.

[0174] exist Fig.16 In Table (1600), corresponding to the first choice of using DCT-2 for both horizontal transform and vertical transform, fixed scan is used. In addition, corresponding to the second choice of using DST7 for both horizontal transform and vertical transform, three scan mode candidates can be applied. Fig.16 In the example, the selectable scan is only applicable to a subset of the selectable transforms.

[0175] In another embodiment, the primary transform selection and the scan selection are jointly signaled, and the scan selection applies to each transform selection in the selectable primary transforms.

[0176] Fig.17 A table (1700) is shown for illustrating transform and scan signaling mapping tables according to some examples. In some examples, statistics for intra-coding with explicit MTS (e.g., in VVC) show that most blocks are coded with DCT-2 for both horizontal and vertical transforms or with DST-7 for both horizontal and vertical transforms. The example in table (1700) utilizes statistics and includes the following two transform selections: a first selection: DCT-2 for both horizontal and vertical transforms, and a second selection: DST-7 for both horizontal and vertical transforms.

[0177] exist Fig.17 In Table (1700), corresponding to the first selection of using DCT-2 for both horizontal transform and vertical transform, the scan mode can be selected from three scan mode candidates (e.g., inverse diagonal scan mode, inverse vertical scan mode, and inverse horizontal scan mode). Similarly, corresponding to the second selection of using DST7 for both horizontal transform and vertical transform, the scan mode can be selected from three scan mode candidates (e.g., inverse diagonal scan mode, inverse vertical scan mode, and inverse horizontal scan mode).

[0178] In some embodiments, the transform selection is explicitly signaled and the scan selection is implicitly signaled based on the intra prediction mode.

[0179] Fig.18 A table (1800) is shown for illustrating transform and scan signaling mapping tables according to some examples. In some examples, statistics for intra-frame coding with explicit MTS (e.g., in VVC) show that most blocks are coded with DCT-2 for both horizontal and vertical transforms or with DST-7 for both horizontal and vertical transforms. The example in table (1800) utilizes statistics and includes the following two transform selections: a first selection: DCT-2 for both horizontal and vertical transforms, and a second selection: DST-7 for both horizontal and vertical transforms.

[0180] exist Fig.18 In table (1800), 1 bit is used to signal mts_idx to indicate one of a first choice of using DCT-2 for both horizontal and vertical transforms and a second choice of using DST7 for both horizontal and vertical transforms. The scan mode is selected based on the intra prediction mode. In some examples, on the encoder side, the encoder can perform rate-distortion optimization on two choices of mts_idx instead of the 5 choices shown in table (1300), and therefore the encoder complexity can be lower.

[0181] In some embodiments, the scanning mode is determined based on an intra prediction mode, such as the intra prediction mode represented by predModeIntra in VVC.

[0182] In the example, when |preModeIntra-–50|<=10 or the value of preModeIntra<=-5, reverse horizontal scanning can be selected; when |preModeIntra-–18|<=10 or the value of preModeIntra>=70, reverse vertical scanning is used; for other values ​​of predModeIntra, reverse diagonal scanning is used.

[0183] Fig.19 A flow chart outlining a process (1900) according to an embodiment of the present disclosure is shown. The process (1900) may be used for a video encoder. In various embodiments, the process (1900) is performed by a processing circuit system, such as a processing circuit system in a terminal device (310), (320), (330), and (340), a processing circuit system that performs the functions of a video encoder (403), a processing circuit system that performs the functions of a video encoder (603), a processing circuit system that performs the functions of a video encoder (703), and the like. In some embodiments, the process (1900) is implemented as software instructions, so that when the processing circuit system executes the software instructions, the processing circuit system performs the process (1900). The process starts at (S1901) and proceeds to (S1910).

[0184] At (S1910), intra prediction of a coding block is performed based on an intra prediction mode.

[0185] At (S1920), it is determined to select a separable primary transform from a plurality of separable primary transform candidates and a scan mode from a plurality of scan mode candidates for encoding a residual from an intra-predicted coding block. In some examples, the video encoder utilizes statistics of most intra-coded blocks that are encoded using a transform function DCT-2 or DST-7. In an example, the video encoder performs a rate-distortion performance evaluation based on the transform function DCT-2 or DST-7 and an applicable scan mode to determine the selection of a separable primary transform from a plurality of separable primary transform candidates and the selection of a scan mode from a plurality of scan mode candidates.

[0186] In some examples, a determination is made to exclude application of the selectable secondary inseparable transform, and thus the selectable secondary inseparable transform is not applied.

[0187] In some examples, the scan patterns are applied to subsets of multiple separable primary transform candidates respectively, such as in table (1600).

[0188] In some examples, the scan mode is applied separately to each of a plurality of separable primary transform candidates, such as in table (1700).

[0189] At (S1930), a residual of the coding block is encoded based on the selection of the separable primary transform and the selection of the scanning mode.

[0190] In some examples, a separable primary transform is applied to the residual to generate transform coefficients in the transform domain. The transform coefficients in the transform domain can then be converted from 2D to 1D based on a scan pattern. The transform coefficients in 1D can be quantized and entropy encoded into bits in a bitstream.

[0191] At (S1940), the selection of the separable primary transform and the selection of the scanning mode are signaled.Then, the process proceeds to (S1999) and terminates.

[0192] In some examples, the selection of the separable primary transform and the selection of the scan mode are jointly and explicitly signaled. In an example, the value of the syntax (e.g., mts_idx) is selected to indicate a combination of the selection of the separable primary transform from a plurality of separable primary transform candidates and the selection of the scan mode from a plurality of scan mode candidates, e.g., using table (1600), table (1700), etc. The value of the syntax is encoded into the bitstream.

[0193] In some examples, selection of a separable primary transform and selection of a scan mode are signaled separately and explicitly. In an example, a value of a first syntax is determined to indicate selection of a separable primary transform, and a value of a second syntax is determined to indicate selection of a scan mode. The first syntax and the second syntax are encoded into a bitstream.

[0194] In some examples, the selection of the separable primary transform is explicitly signaled, and the selection of the scan mode is implicitly signaled. For example, the selection of the scan mode is based on the intra prediction mode.

[0195] In some examples, the selection of the separable primary transform is implicitly signaled, and the selection of the scan mode is implicitly signaled. For example, the selection of the scan mode is based on the intra prediction mode.

[0196] The process (1900) may be adjusted appropriately. The step(s) in the process (1900) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of implementation may be used.

[0197] Fig. 20 A flow chart outlining a process (2000) according to an embodiment of the present disclosure is shown. The process (2000) may be used for a video decoder. In various embodiments, the process (2000) is performed by a processing circuit system, such as a processing circuit system in a terminal device (310), (320), (330), and (340), a processing circuit system that performs the functions of a video decoder (410), a processing circuit system that performs the functions of a video decoder (510), etc. In some embodiments, the process (1800) is implemented as software instructions, so when the processing circuit system executes the software instructions, the processing circuit system performs the process (2000). The process starts at (S2001) and proceeds to (S2010).

[0198] At (S2010), information of a coding block is decoded from a bitstream, the information indicating an intra prediction mode of the coding block, a selection of a separable primary transform from a plurality of separable primary transform candidates, and a selection of a scan mode from a plurality of scan mode candidates.

[0199] In an example, the scanning mode is respectively applied to subsets of the plurality of separable primary transform candidates. In another example, the scanning mode is respectively applied to the plurality of separable primary transform candidates.

[0200] In some examples, the selection of the separable primary transform and the selection of the scan mode are jointly and explicitly signaled, for example, by decoding a syntax value from the bitstream that indicates a combination of the selection of the separable primary transform from a plurality of separable primary transform candidates and the selection of the scan mode from a plurality of scan mode candidates.

[0201] In some examples, the selection of the separable primary transform and the selection of the scan mode are signaled separately and explicitly, for example, a value of a first syntax is decoded from the bitstream, the value of the first syntax indicating the selection of the separable primary transform; and a value of a second syntax is decoded from the bitstream, the value of the second syntax indicating the selection of the scan mode.

[0202] In some examples, the selection of the separable primary transform is explicitly signaled and the selection of the scan mode is implicitly signaled, so the selection of the scan mode can be inferred and can be determined based on syntax decoded from the bitstream. In some examples, both the selection of the separable primary transform and the selection of the scan mode are implicitly signaled, so both the selection of the separable primary transform and the selection of the scan mode can be inferred. In an example, the selection of the scan mode can be inferred based on the direction of the intra prediction mode.

[0203] At (S2020), a residual of a coding block from the bitstream is determined based on the selection of the scan mode and the selection of the separable primary transform.

[0204] In some examples, based on the selection of the scan mode, transform coefficients in the transform domain are decoded from the bitstream.An inverse transform associated with the separable primary transform is then applied to the transform coefficients in the transform domain to determine a residual.

[0205] In some examples, the decoding information of the coding block indicates that the selectable secondary non-separable transform is excluded. Therefore, the application of the inverse transform associated with the selectable secondary non-separable transform is excluded from the process for determining the residual of the coding block.

[0206] At (S2030), the coding block is reconstructed based on a combination of a residual of the coding block and an intra prediction, the intra prediction of the coding block being based on an intra prediction mode.

[0207] Then, the process proceeds to (S2099) and terminates.

[0208] The process (2000) may be adjusted appropriately. The step(s) in the process (2000) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of implementation may be used.

[0209] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.21 A computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0210] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to mechanisms such as assembly, compilation, linking, etc. to create code comprising instructions that may be executed directly by one or more computer central processing units (CPU), graphics processing units (GPU), etc., or through interpretation, microcode execution, etc.

[0211] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0212] Fig.21 The components for the computer system (2100) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement related to any one or combination of components shown in the exemplary embodiment of the computer system (2100).

[0213] The computer system (2100) may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input by one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, tapping), visual input (such as gestures), olfactory input (not depicted). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (such as voice, music, ambient sound), images (such as scanned images, photographic images obtained from a still image camera), and videos (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0214] The input human-machine interface device may include one or more of the following items (only one of each item is drawn): keyboard (2101), mouse (2102), touch pad (2103), touch screen (2110), data gloves (not shown), joystick (2105), microphone (2106), scanner (2107), camera (2108).

[0215] The computer system (2100) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback through a touch screen (2110), a data glove (not shown), or a joystick (2105), but there may also be tactile feedback devices that are not used as input devices); audio output devices (such as: speakers (2109), headphones (not depicted)); visual output devices (such as screens (2110), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be able to output two-dimensional visual output or more than three-dimensional output through methods such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and cigarette cans (not depicted)); and printers (not depicted).

[0216] The computer system (2100) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2120) with CD / DVD etc. media (2121), thumb drives (2122), removable hard drives or solid-state drives (2123), legacy magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), etc.

[0217] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0218] The computer system (2100) may also include an interface (2154) to one or more communication networks (2155). The network may be, for example, wireless, wired, optical. The network may also be local, wide, urban, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks include: local area networks such as Ethernet; wireless LANs; cellular networks including GSM (Global System for Mobile Communication), 3G (Third Generation), 4G (Fourth Generation), 5G (Fifth Generation), LTE (Long Term Evolution), etc.; television wired connections or wireless wide-area digital networks including cable television, satellite television, and terrestrial broadcast television; vehicle networks and industrial networks including CAN buses, etc. Some networks typically require an external network interface adapter that is attached to some universal data port or peripheral bus (2149) (such as, for example, a USB (Universal Serial Bus, USB) port of the computer system (2100)); other networks are typically integrated into the core of the computer system (2100) by attaching to a system bus as described below (for example, an Ethernet interface to a PC computer system or a cellular network interface to a smart phone computer system). Using any of these networks, the computer system (2100) can communicate with other entities. Such communications can be one-way receive only (for example, broadcast television), one-way send only (for example, a CAN bus to some CAN bus devices), or bidirectional, for example, using a local area digital network or a wide area digital network to other computer systems. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0219] The above-mentioned human-machine interface device, human-accessible storage device, and network interface may be attached to the core ( 2140 ) of the computer system ( 2100 ).

[0220] The core (2140) may include one or more central processing units (CPUs) (2141), graphics processing units (GPUs) (2142), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (2143), hardware accelerators for certain tasks (2144), graphics adapters (2150), etc. These devices, along with read-only memory (ROM) (2145), random access memory (2146), internal mass storage devices such as internal non-user accessible hard drives, SSDs (Solid State Drives, SSDs), etc. (2147) may be connected via a system bus (2148). In some computer systems, the system bus (2148) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (2148) directly or via a peripheral bus (2149). In an example, a screen (2110) may be connected to a graphics adapter (2150). The architecture of the peripheral bus includes PCI (Peripheral Component Interconnect, PCI), USB, etc.

[0221] The CPU (2141), GPU (2142), FPGA (2143) and accelerator (2144) can execute certain instructions, which can be combined to form the computer code mentioned above. The computer code can be stored in ROM (2145) or RAM (2146). Transition data can also be stored in RAM (2146), while permanent data can be stored in, for example, an internal mass storage device (2147). Fast storage and retrieval of any memory device in the memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage devices (2147), ROMs (2145), RAMs (2146), etc.

[0222] The computer readable medium may have thereon computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be those of a type well known and available to those skilled in the art of computer software.

[0223] As an example and not limitation, a computer system (2100) having an architecture, and in particular a core (2140), can provide functionality due to the execution of software embodied in one or more tangible computer-readable media by (one or more) processors (including CPUs, GPUs, FPGAs, accelerators, etc.). Such computer-readable media can be media associated with a user-accessible mass storage device as described above and certain storage devices of the core (2140) having non-transitory properties, such as a core internal mass storage device (2147) or ROM (2145). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (2140). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (2140) and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific processing or specific parts of specific processing described herein, including defining data structures stored in RAM (2146) and modifying such data structures according to processing defined by the software. Additionally or alternatively, a computer system may provide functionality due to logic hardwired or otherwise embodied in circuits (e.g., accelerators (2144)) that may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuits (such as integrated circuits (ICs)) storing software for execution, circuits implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0224] Appendix A: Acronyms

[0225] JEM: Joint Development Model

[0226] VVC: Versatile Video Coding

[0227] BMS: Benchmark Set

[0228] MV: Motion Vector

[0229] HEVC: High Efficiency Video Coding

[0230] SEI: Supplemental Enhancement Information

[0231] VUI: Video Availability Information

[0232] GOP: Group of Pictures

[0233] TU: Transform Unit

[0234] PU: prediction unit

[0235] CTU: Coding Tree Unit

[0236] CTB: Coding Tree Block

[0237] PB: prediction block

[0238] HRD: Hypothetical Reference Decoder

[0239] SNR: Signal to Noise Ratio

[0240] CPU: Central Processing Unit

[0241] GPU: Graphics Processing Unit

[0242] CRT: cathode ray tube

[0243] LCD: Liquid Crystal Display

[0244] OLED: Organic Light Emitting Diode

[0245] CD: Compact Disc

[0246] DVD: Digital Video Disc

[0247] ROM: Read Only Memory

[0248] RAM: Random Access Memory

[0249] ASIC: Application-Specific Integrated Circuit

[0250] PLD: Programmable Logic Device

[0251] LAN: Local Area Network

[0252] GSM: Global System for Mobile Communications

[0253] LTE: Long Term Evolution CANBus: Controller Area Network Bus

[0254] USB: Universal Serial Bus

[0255] PCI: Peripheral Component Interconnect

[0256] FPGA: Field Programmable Gate Array

[0257] SSD: Solid State Drive

[0258] IC: Integrated Circuit

[0259] CU: Coding Unit

[0260] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. A video processing method, comprising: decoding information of a coding block from a bitstream, the information indicating an intra prediction mode, a selection of a separable primary transform, and a selection of a scanning mode for the coding block; When it is determined that a secondary inseparable transform is not included in the bitstream and a separable primary transform is included in the bitstream, determining a residual of the coding block from the bitstream based on the selection of the scanning mode; as well as The coding block is reconstructed based on a combination of a residual of the coding block and an intra prediction based on the intra prediction mode.

2. The method according to claim 1, wherein: Determining the residual of the coding block also includes: Based on the selection of the scan mode, decoding the bitstream to determine transform coefficients in a transform domain; and An inverse transform associated with the separable primary transform is applied to the transform coefficients in the transform domain to determine the residual.

3. The method according to claim 1, wherein: The information indicates exclusion of a selectable secondary inseparable transform, and determining a residual of the coding block excludes application of an inverse transform associated with the selectable secondary inseparable transform.

4. The method according to claim 1, wherein: The scanning modes are respectively applicable to subsets of the plurality of separable primary transform candidates.

5. The method according to claim 1, wherein: The scanning mode is respectively applicable to each of the plurality of separable primary transform candidates.

6. The method according to claim 1, further comprising: A syntax value is decoded from the bitstream, the syntax value indicating a combination of a selection of the separable primary transform from the plurality of separable primary transform candidates and a selection of the scan mode from the plurality of scan mode candidates.

7. The method according to claim 1, further comprising: decoding a value of a first syntax from the bitstream, the value of the first syntax indicating selection of the separable primary transform; as well as A value of a second syntax is decoded from the bitstream, the value of the second syntax indicating the selection of the scanning mode.

8. The method according to claim 1, further comprising: The selection of the scanning mode is inferred.

9. The method according to claim 8, further comprising: The selection of the scanning mode is inferred based on the intra prediction mode.

10. An apparatus for video decoding, comprising processing circuitry configured to perform the method of any one of claims 1 to 9.

11. A video processing method, comprising: performing intra prediction of a block of video data based on an intra prediction mode; determining to use a separable primary transform without using a secondary non-separable transform and selecting a scan mode to encode a residual of the block; The block is encoded based on a combination of a residual of the block and intra prediction to generate a bitstream of the encoded block, the bitstream including information indicating an intra prediction mode of the block, a selection of a separable primary transform, and a selection of a scanning mode.

12. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the video processing method according to any one of claims 1 to 9, or the video processing method according to claim 11.

13. A video processing device, comprising: one or more computer-readable media configured to store instructions; as well as One or more processors are configured to execute the instructions to perform the video processing method according to any one of claims 1-9 and 11.

14. A method for processing a bit stream, characterized in that: The bit stream is generated according to the encoding method according to claim 11, or is decoded based on the decoding method according to any one of claims 1 to 9.

Images