Video decoding method, device, computer equipment and storage medium

Abstract

Embodiments of the present application disclose a video decoding method, apparatus, computer device, and storage medium. The method includes: determining whether a current block in a video stream is predicted in an inter-frame and intra-frame joint prediction (CIIP) mode; generating a set of secondary transform coefficients for the current block; applying a combined inter-frame and intra-frame secondary transform by: performing an inverse separable secondary transform or an inverse non-separable secondary transform on the set of secondary transform coefficients to obtain a set of primary transform coefficients for the current block; performing an inverse primary transform on the set of primary transform coefficients to obtain a residual block for the current block; and, in the CIIP mode, decoding the current block based on the residual block.

Description

[0001] Incorporation by reference

[0002] This application claims priority to U.S. non-provisional application No. 17 / 949,429, filed on September 21, 2022, entitled “Secondary Transform for Inter- and Intra-Frame Joint Prediction Modes,” and U.S. provisional application No. 63 / 251,473, filed on October 1, 2021, entitled “Secondary Transform for Inter- and Intra-Frame Joint Prediction Modes,” the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to video encoding / decoding technology, and in particular to video decoding methods, devices, computer equipment and storage media. Background Art

[0004] The background description provided herein is intended to present the background of the present application as a whole. To the extent that the work of the presently named inventors is described in the background section and in various aspects of this specification, it is not intended that it be prior art at the time of filing this application, and it is neither expressly nor impliedly admitted that it is prior art to the present application.

[0005] Video encoding and decoding are made possible by inter-picture prediction techniques with motion compensation. An uncompressed digital video may comprise a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures has a fixed or variable picture rate (also informally referred to as a frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has very large bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution, 60 Hz frame rate) with 8 bits per sample requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 GB of storage space.

[0006] One goal of video encoding and decoding is to reduce redundant information in the input video signal through compression. Video compression can help reduce the bandwidth or storage space requirements mentioned above, in some cases by two or more orders of magnitude. Both lossless and lossy compression, as well as combinations of the two, can be used. Lossless compression refers to techniques that reconstruct an exact replica of the original signal from a compressed original. When lossy compression is used, the reconstructed signal may not be exactly the same as the original, but the distortion between the original and the reconstructed signal is small enough that the reconstructed signal is usable for the intended application. Lossy compression is widely used in video. The amount of distortion allowed depends on the application. For example, users of some consumer streaming applications may be able to tolerate higher distortion than users of television applications. The achievable compression ratio reflects the fact that higher allowed / tolerable distortion results in higher compression ratios.

[0007] Video encoders and decoders may utilize several broad categories of techniques including, for example: motion compensation, transforms, quantization, and entropy coding.

[0008] Video codec techniques may include known intra-frame coding techniques. In intra-frame coding, sample values ​​are represented without reference to samples or other data of previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all sample blocks are encoded in intra-frame mode, the picture may be an intra-frame picture. Intra-frame pictures and their derivatives (such as independent decoder refresh pictures) can be used to reset the decoder state and can therefore be used as the first picture in an encoded video stream and video session, or as a still image. The samples of the intra-frame block can be used for transformation, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0009] Traditional intra-frame coding, as known from coding techniques such as MPEG-2, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to derive data blocks from, for example, surrounding sample data and / or metadata obtained during spatially adjacent encoding / decoding sequences and prior to the decoding order. Such techniques have subsequently become known as "intra-frame prediction" techniques. It should be noted that, at least in some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, not reference data from reference pictures.

[0010] There are many different forms of intra-frame prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be coded as intra-frame prediction modes. In some cases, a mode can have sub-modes and / or parameters, and these modes can be coded separately or included in a mode codeword. Which codeword is used for a given mode / sub-mode / parameter combination affects the codec efficiency gain through intra-frame prediction, and therefore the entropy coding technique used to convert the codewords into the bitstream.

[0011] H.264 introduced an intra-frame prediction mode, which was improved in H.265 and further improved in newer coding technologies such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). The values ​​of neighboring samples belonging to an existing sample can be used to form a prediction block. Depending on the direction, the sample values ​​of the neighboring samples are copied to the prediction block. The reference for the direction used can be encoded in the bitstream or can be predicted by itself.

[0012] In the prior art, when performing a primary transform on a block encoded in the joint inter-frame and intra-frame prediction (CIIP) mode, the primary transform (or primary transform kernel) must be selected from the available triangular transform and identity transform. However, these available choices cannot effectively decorrelate the correlated residual signals when performing intra prediction. Therefore, an optimized configuration for a specific CIIP mode is required. Summary of the Invention

[0013] The embodiments of the present application provide a method, apparatus, computer device, and storage medium for video decoding, which can select a suitable secondary transform kernel for a specific CIIP mode.

[0014] In one aspect, an embodiment of the present application provides a video decoding method, comprising:

[0015] Determine whether the current block in the video stream is predicted in the CIIP mode;

[0016] For the current block, generating a set of secondary transform coefficients;

[0017] Apply a combined inter-intra secondary transform in the following way:

[0018] performing an inverse separable secondary transform or an inverse non-separable secondary transform on the secondary transform coefficient set to obtain a primary transform coefficient set of the current block;

[0019] performing an inverse main transform on the main transform coefficient set to obtain a residual block of the current block; and

[0020] In the CIIP mode, the current block is decoded based on the residual block.

[0021] On the other hand, an embodiment of the present application further provides a video decoding device, comprising:

[0022] A first determination module is used to determine whether a current block in a video stream is predicted in an inter-frame and intra-frame joint prediction CIIP mode;

[0023] A generating module, configured to generate a set of secondary transform coefficients for the current block;

[0024] A transform module is configured to apply a combined inter-frame and intra-frame secondary transform by:

[0025] performing an inverse separable secondary transform or an inverse non-separable secondary transform on the secondary transform coefficient set to obtain a primary transform coefficient set of the current block;

[0026] performing an inverse main transform on the main transform coefficient set to obtain a residual block of the current block; and

[0027] A decoding module is configured to decode the current block based on the residual block in the CIIP mode.

[0028] On the other hand, an embodiment of the present application further provides a computer device, including a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the video decoding method as described above.

[0029] On the other hand, an embodiment of the present application further provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor, the processor implements the video decoding method as described above.

[0030] In another aspect, embodiments of the present application further provide a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described video decoding method.

[0031] It can be seen from the above technical solution that the method provided in the embodiment of the present application, when the current block is predicted using the CIIP mode, obtains the residual block of the current block by applying a combined inter-frame and intra-frame secondary transform, effectively decorrelates the residual signal, and can select a secondary transform kernel suitable for a specific CIIP configuration, thereby improving the decoding processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, in which:

[0033] Figure 1A is a schematic diagram of an exemplary subset of intra prediction modes;

[0034] Figure 1Bis a schematic diagram of an exemplary intra prediction direction;

[0035] Figure 2 A schematic diagram showing a current block and its surrounding spatial merging candidates for motion vector prediction according to an embodiment;

[0036] Figure 3 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment;

[0037] Figure 4 is a schematic diagram of a simplified block diagram of a communication system according to another embodiment;

[0038] Figure 5 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment;

[0039] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment;

[0040] Figure 7 shows a block diagram of an encoder according to another embodiment;

[0041] Figure 8 shows a block diagram of a decoder according to another embodiment;

[0042] Figure 9 An example of coding block partitioning according to an embodiment of the present application is shown;

[0043] Figure 10 An example of coding block partitioning according to another embodiment of the present application is shown;

[0044] Figure 11 An example of coding block partitioning according to another embodiment of the present application is shown;

[0045] Figure 12 An example of coding block partitioning according to another embodiment of the present application is shown;

[0046] Figure 13 An example of partitioning a coding block into multiple transform blocks and an encoding order of the transform blocks according to an embodiment of the present application is shown;

[0047] Figure 14 An example of partitioning a coding block into multiple transform blocks and an encoding order of the transform blocks according to another embodiment of the present application is shown;

[0048] Figure 15 An example of partitioning a coding block into multiple transform blocks according to another embodiment of the present application is shown;

[0049] Figure 16 An example of an intra-frame prediction scheme based on various reference lines according to an embodiment of the present application is shown;

[0050] Figure 17 shows the top, left, and top-left positions of the PAETH mode for a block according to an embodiment of the present application;

[0051] Figure 18 An example of a recursive intra-frame filtering mode according to an embodiment of the present application is shown;

[0052] Figure 19 An example of plane rotation transformation according to an embodiment of the present application is shown;

[0053] Figure 20 Various DCT-2 and DCT-4 partial butterfly lookup tables according to embodiments of the present application are shown;

[0054] Figure 21 shows a DST-7 partial butterfly lookup table according to an embodiment of the present application;

[0055] Figure 22 The low-frequency non-separation transformation process according to an embodiment of the present application is shown;

[0056] Figure 23 A flow chart of a method according to an embodiment of the present application is shown;

[0057] Figure 24 is a schematic diagram of a computer device according to an embodiment. DETAILED DESCRIPTION

[0058] Reference Figure 1A , a subset of nine known prediction directions from the 33 possible prediction directions of H.265 (corresponding to the 33 angular modes of the 35 intra-frame prediction modes) is depicted in the lower right corner. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted based on at least one sample in the upper right corner that is at a 45-degree angle to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted based on at least one sample in the lower left corner that is at a 22.5-degree angle to the horizontal.

[0059] Still refer to Figure 1A, a square block (104) consisting of 4×4 samples is shown in the upper left (indicated by the thick dashed line). The square block (104) consists of 16 samples, each of which is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second (from the top) sample in the Y dimension and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the X and Y dimensions. Since the block is a 4×4 size sample, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. Reference samples are labeled with "R" and their Y position (e.g., row index) and X position (column index) relative to the block (104). In H.264 and H.265, the prediction samples and blocks are adjacent during reconstruction, so there is no need to use negative values.

[0060] Intra-picture prediction of block 104 is performed by copying reference sample values ​​from neighboring samples occupied by a signaled prediction direction. For example, assume that the encoded video stream includes signaling that, for this block, the prediction direction is consistent with arrow (102), i.e., the sample is predicted based on at least one prediction sample to the upper right at a 45-degree angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0061] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to calculate the reference sample, particularly when the direction is not divisible by 45 degrees.

[0062] As video coding technology has developed, the number of directions has gradually increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013) and JEM / VVC / BMS, and at the time of this application, up to 65 directions can be supported. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding can be used to identify these most likely directions with a small number of bits, receiving the loss of some less likely directions. Furthermore, these directions themselves can sometimes be predicted from adjacent directions used by adjacent, decoded blocks.

[0063] Figure 1B A schematic diagram (180) depicting 65 intra prediction directions according to the JEM is shown to illustrate the increasing number of prediction directions over time.

[0064] The mapping of intra-prediction direction bits in the coded video stream to represent directions can vary depending on the video coding technique; and can range, for example, from simple direct mappings of prediction directions to intra-prediction modes, to codewords, to complex adaptive schemes involving most probable modes, and similar techniques. However, in all cases, there are some directions that are statistically less likely to occur in the video content than others. Since the goal of video compression is to reduce redundancy, in a well-functioning video codec, these less likely directions are represented using more bits than more likely directions.

[0065] Motion compensation can be a lossy compression technique and may involve using a block of sample data from a previously reconstructed picture or part of a reconstructed picture (reference picture) for prediction of a newly reconstructed picture or part of a picture, after being spatially shifted in the direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, where the third dimension indicates the reference picture being used (the latter may indirectly be the temporal dimension).

[0066] In some video compression techniques, the MV applied to a region of sample data can be predicted based on other MVs, such as those associated with another region of sample data that is spatially adjacent to the region being reconstructed and that precedes the MV in decoding order. This can significantly reduce the amount of data required to encode the MV, thereby eliminating redundant information and increasing compression. MV prediction can be effective, for example, when encoding an input video signal derived from a camera (referred to as natural video). There is a statistical probability that regions larger than the region to which a single MV applies will move in a similar direction. Therefore, in some cases, similar motion vectors derived from MVs in neighboring regions can be used for prediction. This results in the MV discovered for a given region being similar or identical to the MV predicted from surrounding MVs and, after entropy coding, can be represented using fewer bits than would be used if the MV were encoded directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, MV prediction can itself be lossy, for example due to rounding errors when calculating the predicted value based on several surrounding MVs.

[0067] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding," December 2016). Among the various MV prediction mechanisms provided by H.265, this application describes a technique referred to below as "spatial merging."

[0068] Please refer to Figure 2 , the current block (201) includes samples that have been discovered by the encoder during the motion search process, and the samples can be predicted based on the previous block of the same size that has been spatially offset. In addition, the MV can be derived from metadata associated with one or at least two reference pictures instead of encoding the MV directly. For example, the MV is derived from the metadata of the nearest reference picture (in decoding order) using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206 respectively). In H.265, MV prediction can use the prediction value of the same reference picture that is also being used by the neighboring blocks.

[0069] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present application is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3 In the example, a first pair of terminal devices (310) and (320) performs unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., a video picture stream captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data can be transmitted in the form of at least one encoded video code stream. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video picture, and display the video picture based on the restored video data. In media service applications, etc., unidirectional data transmission may be common.

[0070] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, such as may occur during a video conference. For the bidirectional transmission of data, in the example, each of the terminal devices (330) and (340) can encode video data (e.g., a video picture stream captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), and can decode the encoded video data to restore the video picture, and can display the video picture on an accessible display device based on the restored video data.

[0071] exist Figure 3In the example of , terminal devices (310), (320), (330) and (340) can be shown as servers, personal computers and smart phones, but the principles of the present application may not be limited to this. Embodiments of the present application can be applied to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired and / or wireless communication networks. Communication network (350) can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunications networks, local area networks, wide area networks and / or the Internet. For the purposes of this discussion, the architecture and topology of network (350) may be irrelevant to the operation of the present application, except as explained below.

[0072] As an example, Figure 4 The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0073] The streaming system may include an acquisition subsystem (413), which may include a video source (401), such as a digital camera, for creating an uncompressed video picture stream (402). In an embodiment, the video picture stream (402) includes samples captured by the digital camera of the video source (401). The video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video data (404) (or encoded video bitstream), and the video picture stream (402) can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream (402), the encoded video data (404) (or the encoded video code stream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video code stream (404)), which can be stored on the streaming server (405) for future use. At least one streaming client subsystem, such as Figure 3The client subsystem (406) and the client subsystem (408) in the streaming server (405) can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in the electronic device (330). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0074] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0075] Figure 5 1 is a block diagram of a video decoder (510) according to an embodiment disclosed in the present application. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) of an embodiment.

[0076] A receiver (531) may receive at least one encoded video sequence to be decoded by a video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, wherein each encoded video sequence is decoded independently of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective consuming entities (not shown). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be provided external to the video decoder (510) (not shown). In other cases, a buffer memory (not shown) is provided external to the video decoder (510) to, for example, mitigate network jitter, and another buffer memory (515) may be provided internally to the video decoder (510) to, for example, handle broadcast timing. Furthermore, when the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory may be smaller. Of course, for use on a traffic packet network such as the Internet, the buffer memory (515) may also be required. The buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or similar component (not shown) external to the video decoder (510).

[0077] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. The types of symbols include information for managing the operation of the video decoder (510) and potential information for controlling a display device such as a display device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5As shown in . The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be performed according to a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0078] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0079] Depending on the type of coded video picture or portion of a coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). For the sake of brevity, the flow of such subgroup control information between the parser (520) and the multiple units below is not described.

[0080] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually broken down into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and may be integrated with each other. However, for the purposes of describing the disclosed subject matter, the conceptual breakdown into the following functional units is appropriate.

[0081] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantized transform coefficients as symbols (521) from the parser (520) along with control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) may output a block comprising sample values, which may be input to an aggregator (555).

[0082] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses reconstructed information extracted from the current picture buffer (558) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0083] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coded and potentially motion compensated blocks. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signal) by the aggregator (555) to generate output sample information. The retrieval of the prediction samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), which may include, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0084] The output samples of the aggregator (555) may be used by various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filtering techniques that are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of the coded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0085] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0086] Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

[0087] The video decoder (510) may perform decoding operations according to a predetermined video compression technique, such as that in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata about the HRD buffer management signaled in the encoded video sequence.

[0088] In an embodiment, a receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

[0089] Figure 6 1 is a block diagram of a video encoder (603) according to an embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (403) in an embodiment.

[0090] The video encoder (603) can be used to generate a video from a video source (601) (not Figure 6 In another embodiment, the video source (601) is a part of the electronic device (620) to receive video samples, and the video source can capture video images to be encoded by the video encoder (603). In another embodiment, the video source (601) is a part of the electronic device (620).

[0091] The video source (601) may provide a source video sequence in the form of a stream of digital video samples to be encoded by the video encoder (603), wherein the stream of digital video samples may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that are given motion when viewed sequentially. The pictures themselves may be constructed as a spatial array of pixels, where each pixel may include at least one sample, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples can be readily understood by those skilled in the art. The following description focuses on samples.

[0092] According to an embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, the coupling is not shown in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0093] In some embodiments, the video encoder (603) operates in a coding loop. As a simplified description, in some embodiments, the coding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input picture to be encoded and the reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results regardless of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0094] The operation of the "local" decoder (633) can be combined with the operation of Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0095] At this point, it can be observed that any decoder technology other than parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in essentially the same functional form. For this reason, this application focuses on the decoder operation. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only required in certain areas and is provided below.

[0096] During operation, in some embodiments, the source encoder (630) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes an input picture with reference to at least one previously encoded picture from a video sequence designated as a "reference picture." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

[0097] The local video decoder (633) can decode the coded video data that can be designated as a reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can be a lossy process. When the coded video data is available at the video decoder ( Figure 6 When decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that the video decoder may perform on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture that will be obtained by the remote video decoder.

[0098] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may perform operations on a sample block-by-pixel block basis to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (634).

[0099] The controller (650) can manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0100] The outputs of all the above functional units may be entropy coded in an entropy encoder (645). The entropy encoder (645) losslessly compresses the symbols generated by the various functional units using techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0101] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0102] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned to any of the following picture types:

[0103] An intra picture (I picture) can be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0104] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0105] Bidirectionally predictive pictures (B pictures) can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0106] A source picture is typically spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples) and coded block-wise. These blocks can be predictively coded with reference to other (already coded) blocks, determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture can be non-predictively coded, or they can be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Pixel blocks of a P picture can be predictively coded using spatial prediction or temporal prediction with reference to one previously coded reference picture. Blocks of a B picture can be predictively coded using spatial prediction or temporal prediction with reference to one or two previously coded reference pictures.

[0107] The video encoder (603) may perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video coding technique or standard used.

[0108] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, redundant pictures and slices, and other forms of redundant data, SEI messages, VUI parameter set fragments, and the like.

[0109] The captured video may be presented as a temporal sequence of multiple source pictures (video pictures). Intra-picture prediction (often shortened to intra prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In an embodiment, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a previously encoded and buffered reference picture in the video, the block in the current picture can be encoded using a vector called a motion vector. The motion vector points to the reference block in a reference picture, and when multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0110] In some embodiments, bidirectional prediction techniques can be used for inter-picture prediction. According to bidirectional prediction techniques, two reference pictures are used, for example, a first reference picture and a second reference picture, both preceding the current picture in the video in decoding order (but potentially in the past and future, respectively, in display order). A block in the current picture can be encoded using a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0111] In addition, merge mode technology can be used in inter-frame picture prediction to improve encoding and decoding efficiency.

[0112] According to some embodiments disclosed in the present application, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally speaking, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into at least one coding unit (CU) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type used for the CU, such as an inter prediction type or an intra prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into at least one prediction unit (PU). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Taking the luma prediction block as an example, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0113] Figure 7 is a diagram of a video encoder (703) according to another embodiment disclosed herein. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures and to encode the processed block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (703) is configured to replace Figure 4A video encoder (403) in an embodiment.

[0114] In an HEVC embodiment, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to use intra mode, inter mode, or bi-prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when encoding the processing block in inter mode or bi-prediction mode, the video encoder (703) may use inter prediction or bi-prediction techniques, respectively, to encode the processing block into an encoded picture. In some video coding techniques, merge mode may be an inter-picture prediction submode, in which a motion vector is derived from at least one motion vector predictor without the aid of an encoded motion vector component external to the predictor. In some other video coding techniques, there may be a motion vector component applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0115] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes Figure 7 Shown are an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0116] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the current block with at least one reference block in a reference picture (e.g., a block in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0117] The intra-frame encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously encoded blocks in the same picture in some cases, generate quantization coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information based on at least one intra-frame coding technique). In an embodiment, the intra-frame encoder (722) further calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same picture.

[0118] The general controller (721) is used to determine general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and add the inter prediction information to the bitstream.

[0119] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to operate based on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (724) is used to convert the residual data from the time domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

[0120] The entropy encoder (725) is used to format the codestream to produce encoded blocks. The entropy encoder (725) generates various information according to a suitable standard, such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information from the codestream. It should be noted that according to the disclosed subject matter, when encoding a block in inter-frame mode or the merge sub-mode of bidirectional prediction mode, there is no residual information.

[0121] Figure 8FIG is a diagram of a video decoder (810) according to another embodiment disclosed herein. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is configured to replace Figure 3 A video decoder (410) of an embodiment.

[0122] exist Figure 8 In one embodiment, the video decoder (810) includes Figure 8 , an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together are shown.

[0123] The entropy decoder (871) can be used to reconstruct certain symbols from the encoded picture, which represent syntax elements that constitute the encoded picture. Such symbols may include, for example, the mode used to encode the block (e.g., intra mode, inter mode, bidirectional prediction mode, a combined submode of the latter two, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata for prediction by the intra decoder (872) or the inter decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an embodiment, when the prediction mode is inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0124] The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0125] The intra-frame decoder (872) is configured to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0126] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to obtain the quantizer parameter QP), and this information can be provided by the entropy decoder (871) (the data path is not shown because this is only low-level control information).

[0127] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block. The reconstructed block can be part of a reconstructed picture, which in turn can be part of a reconstructed video. It should be noted that other suitable operations such as deblocking can be performed to improve visual quality.

[0128] It should be noted that the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using any suitable technology. In one embodiment, the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using at least one integrated circuit. In another embodiment, the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using at least one processor that executes software instructions.

[0129] Turning to encoding the block partitions, and in some example implementations, a predetermined pattern may be applied. Figure 9 As shown, an example 4-way partition tree may be used, starting from a first predefined level (e.g., 64×64 block level) and descending to a second predefined level (e.g., 4×4 level). For example, a basic block may have four partitioning options, as shown in 902, 904, 906, and 908, where the partition designated as R is allowed for recursive partitioning because Figure 9 The same partition tree indicated may be repeated at a lower scale until the lowest level (e.g., 4×4 level). Figure 9 Additional restrictions apply to the partitioning scheme. Figure 9 In the implementation of , rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but they may not be allowed to be recursive, while square partitions may be allowed to be recursive. If necessary, follow Figure 9 The recursive partitioning of generates the final set of coded blocks. Such a scheme can be applied to one or more of the color channels.

[0130] Figure 10 Another example predefined partitioning scheme that allows recursive partitioning to form a partition tree is shown. Figure 10 As shown, an example 10-way partition structure or pattern may be predefined. A root block may start from a predefined level (eg, from a 128x128 level, or a 64x64 level). Figure 10Example partition structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. Figure 10 The partition types 1002, 1004, 1006, and 1008 in the second row of the table with three subpartitions may be referred to as "T-type" partitions. The "T-type" partitions 1002, 1004, 1006, and 1008 may be referred to as left T-type, top T-type, right T-type, and bottom T-type. In some implementations, further subdivision is not allowed. Figure 10 The coding tree depth can be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root black such as a 128×128 block can be set to 0, and Figure 10 After further splitting the root block, the coding tree depth increases by 1. In some implementations, Figure 10 After the pattern of 1002, only all square partitions in 1010 may be allowed for recursive partitioning to the next level of the partition tree. In other words, recursive partitioning may not be allowed for square partitions with patterns 1002, 1004, 1006, and 1008. If necessary, Figure 10 The recursive partitioning of generates the final set of coded blocks. Such a scheme can be applied to one or more of the color channels.

[0131] After dividing or partitioning the basic blocks according to any of the above-mentioned partitioning procedures or other procedures, a final set of partitions or coding blocks can also be obtained. Each of these partitions can be at one of the various partition levels. Each of these partitions can be called a coding block (CB). For the various example partitioning implementations described above, each resulting CB can have any allowed size and be at any allowed partition level. They are called coding blocks because they can form units for which some basic encoding / decoding decisions can be made and the encoding / decoding parameters can be optimized, determined and signaled in the encoded video stream. The highest level in the final partition represents the depth of the coding block partition tree. The coding block can be a luminance coding block or a chrominance coding block.

[0132] In some other example implementations, a quadtree structure may be used to recursively partition base luma and chroma blocks into coding units. Such a partitioning structure may be referred to as a Coding Tree Unit (CTU), which is partitioned into Coding Units (CUs) using a quadtree structure to adapt the partitioning to the various local characteristics of the base CTU. In such an implementation, an implicit quadtree partitioning may be performed at picture boundaries so that the block will remain quadtree partitioned until the size fits within the picture boundary. The term CU is used to collectively refer to units of luma coding blocks (CBs) and chroma CBs.

[0133] In some implementations, the CB can be further partitioned. For example, to perform intra-frame or inter-frame prediction during the encoding and decoding process, the CB can be further partitioned into multiple prediction blocks (PBs). In other words, the CB can be further divided into different sub-partitions, in which separate prediction decisions / configurations can be made. In parallel, to interpret the level at which the transform or inverse transform of the video data is performed, the CB can be further partitioned into multiple transform blocks (TBs). The partitioning schemes from the CB to the PBs and TBs can be the same or different. For example, each partitioning scheme can be implemented using its own procedure based on various characteristics of the video data. In some example implementations, the PB partitioning scheme and the TB partitioning scheme can be independent. In some other example implementations, the PB partitioning scheme and the TB partitioning scheme and boundaries can be related. In some implementations, for example, the TB can be partitioned after the PB partitioning, and specifically, after each PB is determined after partitioning the coding block, each PB can then be further partitioned into one or more TBs. For example, in some implementations, the PB can be split into one, two, four, or another number of TBs.

[0134] In some implementations, in order to partition a basic block into coding blocks and further into prediction blocks and / or transform blocks, the luma channel and the chroma channels may be treated differently. For example, in some implementations, partitioning of a coding block into prediction blocks and / or transform blocks may be allowed for the luma channel, while such partitioning of a coding block into prediction blocks and / or transform blocks may not be allowed for one or more chroma channels. In such implementations, the transform and / or prediction of the luma block may therefore only be performed at the coding block level. For another example, the minimum transform block size for the luma channel and one or more chroma channels may be different, e.g., the coding block for the luma channel may be allowed to be partitioned into smaller transform blocks and / or prediction blocks than the chroma channels. For yet another example, the maximum depth for partitioning a coding block into transform blocks and / or prediction blocks may be different between the luma channel and the chroma channels, e.g., the coding block for the luma channel may be allowed to be partitioned into deeper transform blocks and / or prediction blocks than the one or more chroma channels. For a specific example, a luma coding block can be partitioned into transform blocks of multiple sizes, which can be represented as recursive partitioning up to level 2, and can allow transform block shapes of square, 2:1 / 1:2, and 4:1 / 1:4, with transform block sizes from 4×4 to 64×64. However, for chroma blocks, only the largest possible transform block specified for the luma block is allowed.

[0135] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partitions may depend on whether the PB is intra-coded or inter-coded.

[0136] Partitioning a coding block (or prediction block) into transform blocks can be implemented in various example schemes, including but not limited to recursive or non-recursive quadtree partitioning and predefined pattern partitioning, and additionally considering transform blocks at the boundaries of the coding block or prediction block. In general, the resulting transform blocks can be at different partitioning levels, may not have the same size, and may not need to be square in shape (e.g., they can be rectangular with some allowed size and aspect ratio).

[0137] In some implementations, a coding partition tree scheme or structure may be used. The coding partition tree schemes for the luma channel and the chroma channel may not need to be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures. In addition, whether the luma channel and the chroma channel use the same or different coding partition tree structures and the actual coding partition tree structure to be used may depend on whether the stripe being coded is a P stripe, a B stripe, or an I stripe. For example, for an I stripe, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure patterns, while for a P stripe or a B stripe, the luma channel and the chroma channel may share the same coding partition tree scheme. When separate coding partition tree structures or patterns are applied, the luma channel may be partitioned into CBs through one coding partition tree structure, and the chroma channel may be partitioned into chroma CBs through another coding partition tree structure.

[0138] A specific example implementation of coding block and transform block partitioning is described below. In such an example implementation, a basic coding block can be partitioned into coding blocks using the recursive quadtree partitioning described above. At each level, whether further quadtree partitioning of a particular partition should be continued can be determined by local video data characteristics. The resulting CBs can be at various quadtree partitioning levels of various sizes. The decision as to whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode a picture region can be made at the CB level (or CU level, for all three color channels). Depending on the PB partitioning type, each CB can be further partitioned into one, two, four, or another number of PBs. Within a PB, the same prediction process can be applied, and relevant information is transmitted to the decoder based on the PB. After obtaining the residual block by applying the prediction process based on the PB partitioning type, the CB can be partitioned into TBs according to another quadtree structure similar to the coding tree used for the CB. In this specific implementation, the CB or TB can be, but need not be, limited to square shapes. Furthermore, in this specific example, for inter-frame prediction, the PB can be either square or rectangular, while for intra-frame prediction, the PB can only be square. The coding block can be further partitioned into, for example, four square TBs. Each TB can be further recursively partitioned (using quadtree partitioning) into smaller TBs, which are called residual quadtrees (RQTs).

[0139] Another specific example for partitioning a basic coding block into CB and other PBs and / or TBs is described below. Figure 10 Instead of using the multiple partition unit types shown in FIG, a quadtree with nested multi-type trees is used, which uses a binary and ternary partitioning segment structure. The separation concept of CB, PB and TB (i.e., partitioning CB into PB and / or TB, and partitioning PB into TB) can be abandoned, unless it is required when the size of some CBs is too large for the maximum transform length, where such CBs may require further partitioning. This example partitioning scheme can be designed to support more flexible CB partition shapes, so that both prediction and transformation can be performed at the CB level without further partitioning. In such a coding tree structure, the CB can have a square or rectangular shape. Specifically, through the quadtree structure, the coding tree block (CTB) can be partitioned first. Then, through the multi-type tree structure, the quadtree leaf nodes can be further partitioned. Figure 11 An example of a multi-type tree structure is shown. Specifically, Figure 11 The example multi-type tree structure includes four partition types, which are called vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106) and horizontal ternary split (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaves of the multi-type tree. In this example implementation, unless the CB is too large for the maximum transform length, the split is used for both prediction and transform processing without any further partitioning. This means that in most cases, the CB, PB and TB have the same block size in the quadtree with the nested multi-type tree coding block structure. An exception occurs when the maximum supportable transform length is less than the width or height of the color components of the CB.

[0140] Figure 12 An example of a quadtree with a nested multi-type tree coding block structure for block partitioning of one CTB is shown. In more detail, Figure 12 The CTB 1200 is shown to be partitioned by a quadtree into four square partitions 1202, 1204, 1206 and 1208. For each of the quadtree partitions, further use can be made of Figure 11 The multi-type tree structure is used to split. Figure 12 In the example, partition 1204 is not further partitioned. Partitions 1202 and 1208 each use another quadtree partition. For partition 1202, the upper left, upper right, lower left and lower right partitions of the second level quadtree partitioning use the third level of quadtree partitioning, Figure 11 1104, non-dividable and Figure 11 1108. Partition 1208 uses another quadtree partition, and the upper left, upper right, lower left and lower right partitions of the second level quadtree partition are respectively Figure 11 The third level of 1106 is split, non-split, non-split and Figure 11 1104. According to 1104 and 1108, two of the sub-partitions of the third-level upper left partition of 1208 are further divided. Partition 1206 is divided according to Figure 11 The second level partitioning mode of 1102 is divided into two partitions according to Figure 11 1108 and 1102, further perform the third level segmentation on these two partitions. Figure 11 At 1104 , the fourth level segmentation is further applied to one of them.

[0141] For the specific example above, the maximum luma transform size may be 64x64, and the maximum supported chroma transform size may be different, for example, 32x32 for luma. When the width or height of a luma coding block or a chroma coding block is larger than the maximum transform width or height, the luma coding block or the chroma coding block may be automatically split in the horizontal and / or vertical direction to meet the transform size constraints in that direction.

[0142] In the above specific example for partitioning basic coding blocks into CBs, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same coding tree structure. For example, for I slices, luma and chroma can have separate coding block tree structures. When the separated block tree mode is applied, the luma CTB can be partitioned into luma CBs through one coding tree structure, and the chroma CTB is partitioned into chroma CBs through another coding tree structure. This means that a CU in an I slice can consist of coding blocks of the luma component or coding blocks of two chroma components, and a CU in a P slice or B slice always consists of coding blocks of all three color components unless the video is monochrome.

[0143] Example implementations of partitioning coding blocks or prediction blocks into transform blocks and the encoding order of transform blocks are further described in detail below. In some example implementations, transform partitioning can support transform blocks of multiple shapes, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from, for example, 4×4 to 64×64. In some implementations, if the coding block is less than or equal to 64×64, the transform block can only be applied to the luma component, so that for chroma blocks, the transform block size is equal to the coding block size. Otherwise, if the coding block width or height is greater than 64, both the luma coding block and the chroma coding block can be implicitly split into multiple transform blocks: minimum (W, 64) × minimum (H, 64) and minimum (W, 32) × minimum (H, 32), respectively.

[0144] In some example implementations, for both intra-coded blocks and inter-coded blocks, the coding blocks can be further partitioned into multiple transform blocks with a partition depth of up to a predefined number of levels (e.g., 2 levels). The transform block partition depth and size can be related. An example mapping from the transform size of the current depth to the transform size of the next depth is shown in Table 1.

[0145] Table 1: Change partition size settings

[0146]

[0147] Based on the example mapping in Table 1, for a 1:1 square block, the next level of transform partitioning can create four 1:1 square sub-transform blocks. Transform partitioning can stop at, for example, 4×4. In this way, the transform size of 4×4 at the current depth corresponds to the same size of 4×4 at the next depth. In the example in Table 1, for a 1:2 / 2:1 non-square block, the next level of transform partitioning will create two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next level of transform partitioning will create two 1:2 / 2:1 sub-transform blocks.

[0148] In some example implementations, additional restrictions may be applied to the luma component of intra-coded blocks. For example, for each level of transform partitioning, all sub-transform blocks may be constrained to be of equal size. For example, for a 32×16 coding block, a level 1 transform partition creates two 16×16 sub-transform blocks, and a level 2 transform partition creates eight 8×8 sub-transform blocks. In other words, the second level partitioning must be applied to all first level sub-blocks to keep the transform unit sizes equal. Figure 13An example of transform block partitioning for intra-coded square blocks based on Table 1 is shown, along with the coding order illustrated by arrows. Specifically, 1302 shows a square coding block. 1304 shows a first-level partitioning into four equal-sized transform blocks according to Table 1, with the coding order indicated by arrows. 1306 shows a second-level partitioning of all first-level equal-sized blocks into 16 equal-sized transform blocks according to Table 1, with the coding order indicated by arrows.

[0149] In some example implementations, the above-described restrictions on intra-coding may not apply to the luma component of an inter-coded block. For example, after the first level of transform splitting, any of the sub-transform blocks can be further independently split one level further. Thus, the resulting transform blocks may or may not be of the same size. Figure 14 An example of partitioning an inter-coded block into transform blocks with their coding order is shown. Figure 14 In the example of FIG1 , an inter-coded block 1402 is partitioned into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is partitioned into four transform blocks of equal size. Then, only one of the four transform blocks (not all) is further partitioned into four sub-transform blocks, resulting in a total of 7 transform blocks with two different sizes, as shown in 1404. The example coding order of these 7 transform blocks is represented by Figure 14 The arrow in 1404 is shown.

[0150] In some example implementations, for one or more chroma components, some additional restrictions may be applied to the transform block. For example, for one or more chroma components, the transform block size may be as large as the coding block size, but not smaller than a predefined size, such as 8×8.

[0151] In some other example implementations, for coding blocks with a width (W) or height (H) greater than 64, both the luma coding block and the chroma coding block may be implicitly split into multiple transform units: minimum (W, 64) × minimum (H, 64) and minimum (W, 32) × minimum (H, 32), respectively.

[0152] Figure 15 Another optional example of partitioning a coding block or prediction block into transform blocks is further shown. Figure 15 As shown, instead of using recursive transform partitioning, a predefined set of partition types can be applied to a coding block according to its transform type. Figure 15 In the specific example shown, one of six example partition types can be applied to split a coding block into various numbers of transform blocks. Such a scheme can be applied to either a coding block or a prediction block.

[0153] In more detail, Figure 15 The partitioning scheme is as follows Figure 15 Any given transform type shown provides up to 6 partition types. In this scheme, a transform type can be assigned to each coding block or prediction block based on (for example) rate-distortion cost. In an example, the partition type assigned to a coding block or prediction block can be determined based on the transform partition type of the coding block or prediction block. A specific partition type can correspond to a transform block partition size and mode (or partition type), such as Figure 15 As shown in the four partition types shown in the figure. The correspondence between various transform types and various partition types can be predefined. An example correspondence is shown below, where the capital mark indicates the transform type that can be assigned to the coding block or prediction block based on the rate-distortion cost:

[0154] PARTITION_NONE: Allocate the transform size equal to the block size.

[0155] PARTITION_SPLIT: Allocate the transform size so that the width is 1 / 2 of the block size width and the height is 1 / 2 of the block size height.

[0156] PARTITION_HORZ: Allocate a transform size with a width equal to the width of the block size and a height that is 1 / 2 the height of the block size.

[0157] PARTITION_VERT: Allocate a transform size with a width that is 1 / 2 the width of the block size and a height that is the same as the height of the block size.

[0158] PARTITION_HORZ4: Allocates a transform size whose width is the same as the width of the block size and whose height is 1 / 4 of the block size height.

[0159] PARTITION_VERT4: Allocates a transform size with a width that is 1 / 4 the width of the block size and a height that is the same as the height of the block size.

[0160] In the above example, if Figure 15 The partition types shown all include a uniform transform size for the partitioned transform blocks. This is merely an example and not a limitation. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks in a particular partition type (or mode).

[0161] The PB (or CB, also referred to as PB when not further partitioned into prediction blocks) obtained from any of the above partitioning schemes can become a separate block for intra-frame prediction coding or inter-frame prediction coding. For inter-frame prediction of the current PB, the residual between the current block and the prediction block can be generated, encoded, and included in the encoded bitstream.

[0162] Returning to the intra prediction process, samples in a block (e.g., a luma prediction block or a chroma prediction block, or a coding block if it has not been further partitioned into prediction blocks) are predicted by samples from an adjacent line, the next adjacent line, or other one or more lines, or a combination thereof, to generate a prediction block. The residual between the actual block being coded and the prediction block can then be processed by a transform followed by quantization. Various intra prediction modes can be used, and parameters related to intra mode selection and other parameters can be signaled in the codestream. For example, various intra prediction modes can involve one or more line positions for prediction samples, directions along which prediction samples are selected from one or more prediction lines, and other special intra prediction modes.

[0163] For example, a set of intra-prediction modes (interchangeably referred to as "intra modes") may include a predefined number of directional intra-prediction modes. As described above with respect to the example implementation of FIG. 1 , these intra-prediction modes may correspond to a predefined number of directions along which out-of-block samples are selected for prediction of samples being predicted in a particular block. In another specific example implementation, eight (8) primary directional modes may be supported and predefined, corresponding to angles from 45 to 207 degrees from the horizontal axis.

[0164] In some other implementations of intra prediction, in order to further exploit more types of spatial redundancy in directional texture, the directional intra mode can be further extended to an angle set with finer granularity. For example, the above 8-angle implementation can be configured to provide eight nominal angles, such as Figure 16 As shown in the figure, they are called V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED and D67_PRED, and for each nominal angle, a predetermined number (e.g., 7) of finer angles can be added. With this extension, a larger total number (e.g., 56 in this example) of directional angles are available for intra prediction, corresponding to the same number of predefined directional intra modes. The predicted angle can be represented by the nominal intra angle plus the angle δ. For the 7 finer angular directions for each nominal angle in the specific example above, the angle δ can be from -3 to 3 times the 3 degree step size.

[0165] In some implementations, the above directional intra modes may be selected, or in addition to the above directional intra modes, a predefined number of non-directional intra prediction modes may be predefined and made available. For example, five non-directional intra modes referred to as smooth intra prediction modes may be specified. These non-directional intra prediction modes may be specifically referred to as DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. Figure 17The prediction of samples for a particular block under these example non-directional modes is illustrated in FIG. As an example, Figure 17 A 4×4 block 2002 is shown, predicted by samples from the upper and / or left neighboring lines. A particular sample 1710 in block 1702 may correspond to sample 1704 directly above sample 1710 in the upper neighboring line of block 1702, sample 1706 to the upper left of sample 1710, which is the intersection of the upper and left neighboring lines, and sample 1708 to the immediate left of sample 1710 in the left neighboring line of block 1702. For the example DC intra prediction mode, the average of the left and upper neighboring samples 1708 and 1704 may be used as the prediction value for sample 2010. For the example PAETH intra prediction mode, upper, left, and upper left reference samples 1704, 1708, and 1706 may be extracted, and whichever of these three reference samples is closest to (top+left–topleft) may be set as the prediction value for sample 1710. For the example SMOOTH_V intra prediction mode, sample 1710 can be predicted by quadratic interpolation in the vertical direction of the top left neighboring sample 1706 and the left neighboring sample 1708. For the example SMOOTH_H intra prediction mode, sample 1710 can be predicted by quadratic interpolation in the horizontal direction of the top left neighboring sample 1706 and the top neighboring sample 1704. For the example SMOOTH intra prediction mode, sample 1710 can be predicted by the average of the quadratic interpolation in the vertical and horizontal directions. The above non-directional intra mode implementations are shown as non-limiting examples only. Other non-directional selections of other neighboring lines and samples, as well as combinations of prediction samples used to predict a particular sample in a prediction block, are also contemplated.

[0166] The encoder's selection of a particular intra-prediction mode from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) can be signaled in the codestream. In some example implementations, an exemplary eight nominal directional modes and five non-angular smoothing modes (a total of 13 options) can be first signaled. Then, if the signaled mode is one of the eight nominal angular intra-frame modes, an index is further signaled to indicate the selected angle δ for the corresponding signaled nominal angle. In some other example implementations, all intra-frame prediction modes can be indexed together (e.g., 56 directional modes plus five non-directional modes to produce 61 intra-frame prediction modes) for signaling.

[0167] In some example implementations, the example 56 or other number of directional intra prediction modes may be implemented with a unified directional predictor that projects each sample of the block to a reference subsample location and interpolates the reference samples through a 2-tap bilinear filter.

[0168] In some implementations, in order to capture the attenuation space relative to the reference on the edge, additional filter modes can be designed, which are called filter intra modes. For these modes, the intra-block prediction samples, in addition to the samples outside the block, can be used as intra-frame prediction reference samples. For some patches within the block. For example, these modes can be predefined and made valid for intra-frame prediction of at least luminance blocks (or only luminance blocks). A predefined number (for example, 5) of filter intra modes can be predesigned, each filter intra mode being represented by a set of n-tap filters (for example, 7-tap filters) that reflect the correlation between, for example, a sample in a 4×2 patch and n neighboring samples adjacent to the sample. In other words, the weight factors for the n-tap filters can be position-dependent. Taking 8×8 blocks, 4×2 patches and 7-tap filtering as an example, as Figure 18 As shown, the 8×8 block 1802 can be split into eight 4×2 patches. Figure 18 In , these patches are represented by B0, B1, B2, B3, B4, B5, B6 and B7. For each patch, its 7 neighboring patches (in Figure 18 (indicated by R0 to R7 in the figure) can be used to predict the samples in the current patch. For patch B0, all neighboring patches may have been reconstructed. But for other patches, some neighboring patches are in the current block and therefore may not have been reconstructed. Then the predicted values ​​of the immediately neighboring patches are used as reference. For example, Figure 18 All neighboring patches of patch B7 shown in are not reconstructed, so the predicted samples of the neighboring patches are used instead.

[0169] In some implementations of intra prediction, a color component can be predicted using one or more other color components. The color component can be any of the components in the color space YCrCb, RGB, XYZ, etc. For example, prediction of a chroma component (e.g., a chroma block) from a luma component (e.g., a luma reference sample) (referred to as chroma from luma, or CfL) can be implemented. In some example implementations, cross color prediction can only be allowed from luma to chroma. For example, the chroma samples in the chroma block can be modeled as a linear function that conforms to the reconstructed luma samples. CfL prediction can be implemented as follows:

[0170] CfL(α)=α×L AC +DC (1)

[0171] Among them L ACrepresents the AC contribution of the luma component, α represents the parameters of the linear model, and DC represents the DC contribution of the chroma component. For example, the AC component is obtained for each sample of the block, while the DC component is obtained for the entire block. Specifically, the reconstructed luma samples can be subsampled to the chroma resolution, and then the average luma value (DC of luma) can be subtracted from each luma value to form the AC contribution in luma. The AC contribution of luma is then used in the linear model of equation (1) to predict the AC value of the chroma component. In order to approximate or predict the chroma AC component from the luma AC contribution, instead of requiring the decoder to calculate the scaling parameters, the example CfL implementation can determine the parameter α based on the original chroma samples and signal them in the codestream. This reduces decoder complexity and produces more accurate predictions. As for the DC contribution of the chroma component, in some example implementations, the DC contribution can be calculated by using the intra-frame DC mode within the chroma component.

[0172] As an alternative to intra-frame prediction, inter-frame prediction can be performed on the PB in single-reference inter-frame prediction mode or joint-reference inter-frame prediction mode. Specifically, in inter-frame prediction mode, a video block can be predicted from one or more other reference blocks or inter-frame prediction blocks from one or more other frames using single-reference inter-frame prediction or joint-reference inter-frame prediction. To implement inter-frame prediction, a reference block can be specified by its frame identifier (the temporal location of the reference block) and a motion vector indicating the spatial offset between the current block being encoded or decoded and the reference block (the spatial location of the reference block). The reference frame identifier and motion vector can be signaled in the codestream. The motion vector, which serves as the spatial block offset, can be directly signaled or automatically predicted from another reference motion vector or a predictor motion vector. For example, the current motion vector can be predicted directly from a reference motion vector (e.g., of a candidate neighboring block) or from a combination of the reference motion vector and the merged motion vector difference (MVD) between the current motion vector and the reference motion vector. The latter is referred to as merge mode with motion vector difference (MMVD). The reference motion vector can be identified in the code stream as a pointer, for example, pointing to a spatially adjacent block or a temporally adjacent but spatially co-located block of the current block.

[0173] In some implementations, an inter-intra joint prediction (CIIP) mode may be implemented. In CIIP mode, a prediction block is derived as a combination of an intra prediction (or intra predictor) block and an inter prediction (intra predictor) block. An inter prediction block for CIIP may be derived using a single reference inter prediction with translational motion corresponding to a motion vector, while an intra prediction block for CIIP may be determined from adjacent samples based on a subset of the intra prediction modes described above. In some example implementations, spatial samples in an intra prediction block may be derived from intra reference row samples for a current block predicted in CIIP mode according to a mode in a subset of intra prediction modes including the DC_PRED, V_PRED, H_PRED, and SMOOTH modes described above. As shown in Table 2, the intra prediction block derived using each mode in the subset of intra prediction modes may correspond to a CIIP submode index.

[0174] Table 2: Inter-frame and intra-frame sub-modes

[0175] Inter-frame and intra-frame submode indexes Inter-frame and intra-frame submode names 0 II_DC_PRED 1 II_V_PRED 2 II_H_PRED 3 II_SMOOTH_PRED

[0176] The sample-level intra prediction block and inter prediction block derived from the above description are weighted and summed to generate a joint inter-frame and intra-frame prediction (or predictor) block for the current block. The relative weights between the intra prediction block and the inter prediction block can then be represented by a weight matrix. Various exemplary methods for determining the weight matrix can be implemented corresponding to various inter-frame and intra-frame weight modes of the CIIP.

[0177] In one example implementation, the weight pattern of CIIP, referred to as conventional CIIP, the elements of the inter- and intra-weight matrices corresponding to samples in the current block will follow a deterministic relationship with the position of the sample. Such a deterministic relationship may depend on the intra-prediction mode being used. In one specific example, the weight applied to an intra-prediction sample P0(x, y) (where x and y represent the sample position in the block) may be derived as follows:

[0178]

[0179] Among them, sizeScale is derived by using block width (W) and block height (H) as shown below

[0180] sizeScale=128 / max(W,H), (3)

[0181] And WeightLUT represents a one-dimensional lookup table. The value of WeightLUT[index] is determined by the entry at the position indicated by index in the lookup table. An example one-dimensional weight lookup table is shown in Table 3 below.

[0182] Table 3: Lookup table for conventional inter-frame and intra-frame joint prediction mode

[0183] 60 58 56 54 52 50 48 47 45 44 42 41 39 38 37 35 34 33 32 31 30 29 28 27 26 25 24 23 22 22 21 20 19 19 18 18 17 16 16 15 15 14 14 13 13 12 12 12 11 11 10 10 10 9 9 9 8 8 8 8 7 7 7 7 6 6 6 6 6 5 5 5 5 5 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

[0184] In the conventional CIIP example above, the weight for intra prediction generally decreases as the sample moves away from the top left corner of the block (or away from the intra prediction reference sample), except for the DC_PRED mode, in which the inter prediction weight is independent of the sample position. In other words, the implementation illustrated in Table 3 reflects a scheme in which the inter prediction weight increases as the sample position moves away from the intra reference sample.

[0185] In some other exemplary inter- and intra-weighted implementations of CIIP, referred to as wedge CIIP, a set of weight patterns can be defined, and one of the patterns in the set of weight patterns can be selected by the encoder for the current block. The index of the selected pattern in the set of weight patterns can be signaled in the bitstream. Such a pattern can be applied to the current block to determine a specific weight matrix for combining / summing the intra- and inter-prediction blocks for each sample of the block. For example, 16 different patterns (optionally referred to as wedge patterns) can be predefined and represented by indices 0 to 15. In wedge CIIP, once the index of the pattern is specified, the entire weight matrix for the block can be derived, rather than being signaled sample by sample in the conventional CIIP method using the predefined lookup table of Table 3.

[0186] In any CIIP, once the above-mentioned weight matrix is ​​obtained and applied to the combined intra-frame prediction block and inter-frame prediction block, the block thus combined can be used as the actual prediction block of the current block to obtain a residual block. Then, from the perspective of the encoder, the residual block can undergo a primary transform and an optional secondary transform, as well as the rest of the quantization and entropy coding process. For the decoder, the code stream will be parsed / decoded and inversely transformed to obtain the residual block. If the decoder determines from the code stream that the CIIP is for the current block, the weight matrix for the CIIP can be further obtained based on information extracted from the code stream (the search index signaled as described above or the wedge pattern index signaled as described above). The prediction block can then be derived from the weight matrix and the corresponding intra-frame prediction block, and the inter-frame prediction can be derived from the reconstructed samples for the current frame or reference frame. The original block can then be restored from the residual block and the prediction block.

[0187] Returning to the main transform, an example 2-D transform process, in addition to involving the use of the same transform kernel for both dimensions, may also involve the use of a hybrid transform kernel (e.g., which may consist of a different 1-D transform for each dimension of the coded residual block). Example main 1-D transform kernels may include, but are not limited to: a) 4-point (4p), 8-point (8p), 16-point (16p), 32-point (32p), and 64-point (64p) DCT-2; b) 4-point, 8-point, 16-point asymmetric DST and their flipped versions; c) 4-point, 8-point, 16-point, or 32-point identity transform (DST stands for discrete sine transform). Thus, the 2-D transform process may involve the use of a hybrid transform or transform kernel (a different transform for each dimension of the coded residual block), where the selection of the transform or transform kernel to be used for each dimension may be based on a rate-distortion (RD) criterion. The term transform kernel may be referred to as a transform basis function. For example, the basis functions for 1-D DCT-2, DST-4, and DST-7 (for 2-D transforms, a hybrid can be implemented) are listed in Table 4 (where DCT stands for discrete cosine transform).

[0188] Table 4 Example main transform basis functions (DCT-2, DST-4 and DST-7 for N-point input)

[0189]

[0190] For example, DCT-2 (4p-64p), DST-4 (8p, 16p), and DST-7 (4p) transforms exhibit symmetric / asymmetric properties, and therefore may support "partial butterfly" implementations in some example implementations to reduce the number of operation counts (multiplications, additions / subtractions, shifts). Partial butterfly may involve plane rotations using trigonometric cosine and sine functions, such as Figure 19 As shown, at all angles. Figure 20 and Figure 21 An example 12-bit lookup table is shown in and can be used to generate values ​​for trigonometric functions.

[0191] In some implementations, a secondary transform may be performed on the primary transform coefficients. Figure 22As shown, LFNST (low-frequency non-separable transform), which is called a reduced secondary transform, can be applied between the forward main transform and quantization (at the encoder) and between dequantization and the inverse main transform (at the decoding end) to further decorrelate the main transform coefficients. Essentially, LFNST can remove a portion of the main transform coefficients, such as the low-frequency portion (thus "reduced" from the complete set of main transform coefficients of the transform block) to proceed to the secondary transform. In the example LFNST, a 4×4 non-separable transform or an 8×8 non-separable transform can be applied depending on the transform block size. For example, 4×4 LFNST can be applied to small transform blocks (e.g., minimum (width, height) <8), while 8×8 LFNST can be applied to larger transform blocks (e.g., minimum (width, height) >8). For example, if an 8×8 transform block undergoes 4×4 LFNST, only the low-frequency 4×4 portion of the 8×8 main transform coefficients will be further subjected to the secondary transform.

[0192] like Figure 22 As specifically shown, the transform block can be 8×8 (or 16×16). Therefore, the forward main transform 2202 of the transform block produces an 8×8 (or 16×16) main transform coefficient matrix 2204, where each square cell represents a 2×2 (or 4×4) portion. The input to the forward LFNST may not be the entire 8×8 (or 16×16) main transform coefficient matrix. For example, a 4×4 (or 8×8) LFNST may be used for the secondary transform. Thus, as shown by the shaded portion (upper left) 2206, only the 4×4 (or 8×8) low-frequency main transform coefficients of the main transform coefficient matrix 2204 may be used as input for the LFNST. The remaining portion of the main transform coefficient matrix may not undergo the secondary transform. Thus, after the secondary transform, the portion of the main transform coefficients that undergo LFNST becomes the secondary transform coefficients, while the remaining portion that does not undergo LFNST (e.g., the unshaded portion of matrix 2204) retains the corresponding main transform coefficients. In some example implementations, the remaining portion that does not undergo the secondary transform may be set entirely to zero coefficients.

[0193] The following describes an example of the application of the non-separable transform used in LFNST. To apply the example 4×4 LFNST, a 4×4 input block X (denoted as Figure 22 The 4×4 low-frequency portion of the main transform coefficient block (the shaded portion 2206 of the main transform matrix 2204) can be expressed as:

[0194]

[0195] This 2-D input matrix can first be linearized, or scanned into a vector in the order of examples

[0196]

[0197] Then the non-separable transform of the 4×4 LFNST can be calculated as where denotes the output transform coefficient vector, and T is a 16×16 transform matrix. Subsequently, the obtained 16×1 coefficient vector is scanned backward into 4×4 blocks in the scan order of the blocks (e.g., horizontal, vertical, or diagonal). Coefficients with smaller indices can be placed in the 4×4 coefficient blocks together with smaller scan indices. In this way, the redundancy in the primary transform coefficients X can be further exploited via the secondary transform T, thereby providing additional compression enhancement.

[0198] The above example LFNST is based on a direct matrix multiplication method that applies a non-separable transform, enabling it to be implemented in a single pass without multiple iterations. In some further example implementations, the dimension of the non-separable transform matrix (T) of the example 4×4 LFNST can be further reduced to minimize the computational complexity and the storage space requirement for storing the transform coefficients. Such an implementation can be referred to as a reduced non-separable transform (RST). More specifically, the main idea of RST is to map an N-dimensional vector (where N is 4×4 = 16 in the above example, but can be equal to 64 for an 8×8 block) to an R-dimensional vector in a different space, where N / R (R < N) represents the dimension reduction factor. Thus, instead of an N×N transform matrix, the RST matrix becomes an R×N matrix as follows:

[0199]

[0200] where the R rows of the transform matrix are the reduced R basis of the N-dimensional space. Thus, this transform converts the input vector or N-dimension into an output vector of reduced R-dimension. In this way, as Figure 22 shown, the secondary transform coefficients 2208 transformed from the primary coefficients 2206 are reduced by a factor or N / R in the dimension. Figure 22 The three squares around 2208 in can be zero-padded.

[0201] ​The inverse transform matrix of RTS can be the transpose of its forward transform. For the example 8×8 LFNST (described more differently here compared to the above 4×4 LFNST), an example reduction factor of 4 can be applied, so the 64×64 direct non-separable transform matrix is ​​correspondingly reduced to a 16×64 direct matrix. Further, in some implementations, part of the input main coefficients, rather than all, can be linearized into the input vector of LFNST. For example, only a part of the example 8×8 input main transform coefficients can be linearized into the above X vector. For a specific example, in the four 4×4 quadrants of the 8×8 main transform coefficient matrix, the lower right (high frequency coefficient) can be omitted, and only the other three quadrants can be linearized into 48×1 vectors, rather than 64×1 vectors, using a predefined scan order. In such an implementation, the non-separable transform matrix can be further reduced from 16×64 to 16×48.

[0202] Thus, as an example, a reduced 48×16 inverse RST matrix can be used at the decoding end to generate the upper left, upper right, and lower left 4×4 quadrants of the 8×8 core (main) transform coefficients. Specifically, when a further reduced 16×48 RST matrix is ​​applied instead of a 16×64 RST with the same transform set configuration, the inseparable secondary transform takes as input the vectorized 48 matrix elements from the three 4×4 quadrant blocks of the 8×8 main coefficient block except for the lower right 4×4 block. In such an implementation, the omitted lower right 4×4 main transform coefficients will be ignored in the secondary transform. The further reduced transform converts the 48×1 vector into a 16×1 output vector, which is reverse scanned into a 4×4 matrix to fill the Figure 22 2208. The three squares of secondary transform coefficients around 2208 can be zero-filled.

[0203] In this way, by reducing the dimensionality in RST, the memory usage for storing all LFNST matrices is reduced. In the above example, the memory usage can be reduced from 10KB to 8KB, for example, while the performance degradation is quite insignificant compared to the implementation without dimensionality reduction.

[0204] In some implementations, to reduce complexity, LFNST can be further restricted to only be applied outside the portion of the main transform coefficients to be LFNSTed (e.g., in Figure 22 Therefore, when LFNST is applied, all primary-only transform coefficients (e.g., Figure 22The unshaded portion of the main coefficient matrix 2204 can be close to zero. Such a restriction allows the LFNST index signaling to be adjusted at the last significant position, thereby avoiding some additional coefficient scans that might be required to check for significant coefficients at specific positions when the restriction is not applied. In some implementations, the worst-case processing of LFNST (in terms of per-pixel multiplications) can limit the non-separable transforms for 4×4 and 8×8 blocks to 8×16 and 8×48 transforms, respectively. In these cases, for other sizes less than 16, the last significant scan position must be less than 8 when LFNST is applied. For blocks of shape 4×N and N×4, and N>8, the above restriction means that LFNST is now only applied once to the top-left 4×4 region. Since all the main-only coefficients are zero when LFNST is applied, the number of operations required for the main transform is reduced in this case. From the encoder's perspective, the quantization of the coefficients can be simplified when testing the LFNST transform. For the first 16 coefficients (in scan order), rate-distortion optimized quantization (RDO) must be performed at the maximum value, and the remaining coefficients can be forced to zero.

[0205] In some example implementations, the available RST kernels can be specified as multiple transform sets, each of which includes multiple non-separable transform matrices. For example, each transform set used in LFNST can have a total of four transform sets and two non-separable transform matrices (kernels). These kernels can be pre-trained offline, making them data-driven. The offline-trained transform kernels can be stored in memory or hard-coded in the encoding or decoding device for use during the encoding / decoding process. The selection of the transform set during the encoding or decoding process can be determined by the intra prediction mode. The mapping from intra prediction mode to transform set can be pre-defined. An example of such a pre-defined mapping is shown in Table 4. For example, as shown in Table 4, if one of the three cross-component linear model (CCLM) modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (i.e., 81 <= predModeIntra <= 83), then transform set 0 can be selected for the current chroma block. For each transform set, the selected non-separable secondary transform candidate can be further specified by an explicitly signaled LFNST index. For example, for each intra CU after the transform coefficients, the index may be signaled once in the codestream.

[0206] Table 4: Transformation selection table

[0207]

[0208]

[0209] Because LFNST in the example implementation described above is restricted to being applicable only when all coefficients outside the first coefficient subgroup or portion are non-significant, the LFNST index encoding depends on the position of the last significant coefficient. In addition, the LFNST index can be context-coded, but does not depend on the intra prediction mode, and only the first binary number can be context-coded. In addition, LFNST can be applied to intra CUs in intra and inter slices, as well as luma and chroma. If dual-tree is enabled, the LFNST indexes for luma and chroma can be signaled separately. For inter slices (dual-tree disabled), a single LFNST index can be signaled, and a single LFNST index can be used for luma and chroma.

[0210] In some example implementations, when Intra Subpartitioning (ISP) mode is selected, LFNST may be disabled and the RST index may not be signaled. This is because even if RST is applied to every feasible partition block, the performance improvement may be negligible. In addition, disabling RST for the residual of ISP prediction can reduce coding complexity. In some further implementations, when Multi-Linear Regression Intra Prediction (MIP) mode is selected, LFNST may also be disabled and the RST index may not be signaled.

[0211] Considering that large CUs larger than 64×64 (or any other predefined size representing the maximum transform block size) are implicitly split (e.g., TU tiling) due to existing maximum transform size limits (e.g., 64×64), LFNST index searches can quadruple the data buffering for a certain number of decoding pipeline stages. Therefore, in some implementations, the maximum size allowed for LFNST can be limited to, for example, 64×64. In some implementations, LFNST can be enabled only with DCT2 as the primary transform.

[0212] In CIIP mode, it is necessary to perform inter-frame prediction and intra-frame prediction, and involve a main transform and a secondary transform. When a main transform is performed on a block encoded in CIIP mode, the main transform (or main transform kernel) must be selected from the available triangular transform and identity transform, for example, the transforms listed in Table 4. Intra-frame prediction in CIIP mode can be performed in one of the DC_PRED, V_PRED, H_PRED and SMOOTH modes, and when using CIIP mode to perform intra-frame prediction, if the correlated residual signal is effectively decorrelated, these available choices may not be ideal. Since there are various sub-modes in CIIP, and CIIP mode can include conventional CIIP and wedge-shaped CIIP, it is necessary to determine a mechanism to select a secondary transform kernel that is suitable for a specific CIIP configuration (for example, CIIP sub-modes, CIIP weight modes such as conventional CIIP and wedge-shaped CIIP). In the present application, various implementations are described that are intended to at least achieve this task.

[0213] In this application, a secondary transform set refers to a set of transform kernel (or candidate) options. A transform set may include one or more transform kernel (or candidate) options.

[0214] In this application, the main transform may refer to a combination of DCT, ADST, flip ADST (FLIPADST), LGT and row-column transform (RCT). For example, DCT may be applied horizontally and ADST may be applied vertically on a block.

[0215] In some implementations, in CIIP mode, the secondary transform may be non-separable. In addition, the selection of the non-separable secondary transform may depend on the intra prediction mode of the intra prediction part of CIIP.

[0216] In some implementations, the CIIP mode may be related to the intra prediction portion of the CIIP. Thus, the selection of the non-separable secondary transform may depend directly or indirectly on the CIIP mode.

[0217] In some implementations, when CIIP mode is used to predict a block, a separable / non-separable primary transform can be used to encode the residual (or residual block, residual matrix). Then, a non-separable secondary transform, i.e., a combined inter-intra secondary transform (CIIST), can be further applied.

[0218] In some implementations, the secondary transform may simply be non-separable.

[0219] In some implementations, the secondary transform may only be separable.

[0220] In some implementations, the secondary transform may be separable or non-separable.

[0221] In some implementations, each intra prediction mode can be mapped to or correspond to a transform core set comprising multiple transform cores. When the CIIP mode is used to perform prediction, the intra prediction mode (of the intra prediction portion of the CIIP) can be determined based on the CIIP sub-mode. Example CIIP sub-modes can be found in Table 2 of the previous section. For example, each CIIP sub-mode can be mapped to an intra prediction mode. Once the intra prediction mode is determined, a secondary transform core set can be derived based on the intra prediction mode. The transform core for the secondary transform can be determined by signaling, for example, by a core index that is signaled. The transform core can also be derived based on the block itself, for example, the mode of the block.

[0222] In some implementations, the CIIP sub-mode may be mapped to any of the DC_PRED, V_PRED, H_PRED, or SMOOTH_PRED modes.

[0223] In some implementations, the II_DC_PRED CIIP sub-mode may be mapped to the DC_PRED intra prediction mode.A secondary transform kernel set corresponding to (or pre-associated with) the DC_PRED intra prediction mode may be selected.

[0224] In some implementations, when the CIIP sub-mode is II_DC_PRED, then the secondary transform kernel may be pre-associated with the DC_PRED intra prediction mode.

[0225] In some implementations, the II_SMOOTH_PRED CIIP submode may be mapped to the SMOOTH_PRED intra prediction mode.A secondary transform kernel set corresponding to (or pre-associated with) the SMOOTH_PRED intra prediction mode may be selected.

[0226] In some implementations, when the CIIP sub-mode is II_SMOOTH_PRED, then the secondary transform kernel may be pre-associated with the SMOOTH_PRED intra prediction mode.

[0227] In some implementations, the II_V_PRED CIIP sub-mode may be mapped to the V_PRED intra prediction mode.A secondary transform kernel set corresponding to (or pre-associated with) the V_PRED intra prediction mode may be selected.

[0228] In some implementations, when the CIIP sub-mode is II_V_PRED, then the secondary transform kernel may be pre-associated with the V_PRED intra prediction mode.

[0229] In some implementations, the II_H_PREDCIIP submode may be mapped to the H_PRED intra prediction mode.A secondary transform kernel set corresponding to (or pre-associated with) the H_PRED intra prediction mode may be selected.

[0230] In some implementations, when the CIIP sub-mode is II_H_PRED, then the secondary transform kernel may be pre-associated with the H_PRED intra prediction mode.

[0231] In some implementations, further restrictions or preconditions may be imposed when mapping CIIP submodes to intra-prediction modes. For example, a specific weighting pattern for the CIIP may be required (e.g., conventional CIIP, wedge-shaped CIIP). In one implementation, the scheme for mapping CIIP submodes to intra-prediction modes may be applied only when the CIIP is conventional CIIP. As an example, conventional CIIP, the intra-prediction weights associated with a video block depend on the intra-prediction mode and decrease along the prediction direction of the intra-prediction.

[0232] In some implementations, the scheme for mapping CIIP sub-modes to intra prediction modes may be applied when the CIIP is a wedge-shaped CIIP.

[0233] In some implementations, when the CIIP is a wedge-shaped CIIP, a kernel for the secondary transform may be selected from the secondary transform kernel set based on a wedge mode (see the previous section for details on the wedge mode). For example, the intra prediction weights associated with a video block in the CIIP mode may be characterized by a spatial weight mode (which is a wedge mode).

[0234] In some implementations, when the CIIP is a wedge-shaped CIIP, the secondary transform kernel can be directly determined based on the wedge pattern.

[0235] Figure 23 A flowchart 2300 of an example video decoding method in accordance with the principles constituting the above implementation is shown. Method 2300 may include some or all of the following steps: step 2310, determining whether a current block in a video stream is predicted in CIIP mode; step 2320, generating a set of secondary transform coefficients for the current block; step 2330, applying a combined inter-frame and intra-frame secondary transform by performing an inverse separable secondary transform or an inverse non-separable secondary transform on the set of secondary transform coefficients to obtain a set of primary transform coefficients for the current block; performing an inverse primary transform on the set of primary transform coefficients to obtain a residual block for the current block; and step 2340, decoding the current block in the CIIP mode based on the residual block.

[0236] In the embodiments and implementations of the present application, any steps and / or operations can be combined or arranged in any number or order as needed. Two or more steps and / or operations can be performed in parallel. The embodiments and implementations of the present application can be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder and decoder can be implemented by a processing circuit system (for example, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium. The embodiments in the present application can be applied to luminance blocks or chrominance blocks. The term "block" can be interpreted as a prediction block, a coding block or a coding unit, i.e., a CU. The term "block" can also be used herein to refer to a transform block. In the following items, when a block size is mentioned, it can refer to block width or height, or the maximum value of width and height, or the minimum value of width and height, or the area size (width*height), or the aspect ratio of the block (width:height or height:width).

[0237] The above technology can be implemented as computer software through computer-readable instructions and physically stored in at least one computer-readable storage medium. For example, Figure 24 A computer device (2500) is shown that is suitable for implementing certain embodiments of the disclosed subject matter.

[0238] The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or other mechanisms to create a code comprising instructions, which may be directly executed by at least one computer central processing unit (CPU), graphics processing unit (GPU), or the like, or executed through decoding, microcode, or the like.

[0239] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, and the like.

[0240] Figure 24 The components shown for the computer device (2500) are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing embodiments of the present application. Nor should the configuration of components be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of the computer device (2500).

[0241] The computer device (2500) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from at least one human user via tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), or olfactory input (not shown). The human-computer interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0242] The human-computer interface input device may include at least one of the following (only one of which is drawn): keyboard (2501), mouse (2502), touchpad (2503), touch screen (2510), data gloves (not shown), joystick (2505), microphone (2506), scanner (2507), camera (2508).

[0243] The computer device (2500) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate at least one sense of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (2510), a data glove (not shown), or a joystick (2505), but tactile feedback devices that are not used as input devices may also be present), audio output devices (e.g., speakers (2509), headphones (not shown)), visual output devices (e.g., screens (2510) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities—some of which may output two-dimensional visual outputs or outputs of more than three dimensions through means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and cigarette boxes (not shown)), and printers (not shown).

[0244] The computer device (2500) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) (2520) with CD / DVD or similar media (2521), thumb drives (2522), removable hard drives or solid state drives (2523), traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security software dongles (not shown), and the like.

[0245] Those skilled in the art will also understand that the term "computer-readable storage medium" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0246] The computer device (2500) may also include an interface (2554) to at least one communication network (2555). For example, the network may be wireless, wired, or optical. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle network, an industrial network, a real-time network, a delay-tolerant network, and the like. Networks also include local area networks such as Ethernet, wireless local area networks, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), in-vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter for connecting to some common data port or peripheral bus (2549) (for example, a USB port of the computer device (2500)); other systems are typically integrated into the core of the computer device (2500) by connecting to a system bus as described below (for example, an Ethernet interface integrated into a PC computer device or a cellular network interface integrated into a smartphone computer device). By using any of these networks, the computer device (2500) can communicate with other entities. The communication can be one-way, for receiving only (e.g., wireless television), one-way, for sending only (e.g., a CAN bus to certain CAN bus devices), or two-way, such as to other computer devices via a local or wide area digital network. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0247] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the core (2540) of the computer device (2500).

[0248] The core (2540) may include at least one central processing unit (CPU) (2541), a graphics processing unit (GPU) (2542), a dedicated programmable processing unit in the form of a field programmable gate array (FPGA) (2543), a hardware accelerator for specific tasks (2544), a graphics adapter (2550), and the like. These devices, as well as read-only memory (ROM) (2545), random access memory (2546), internal mass storage (e.g., an internal non-user accessible hard drive, solid-state drive, etc.) (2547), and the like, may be connected via a system bus (2548). In some computer devices, the system bus (2548) may be accessed in the form of at least one physical plug so that it can be expanded with additional central processing units, graphics processing units, and the like. Peripheral devices may be attached directly to the core's system bus (2548) or connected via a peripheral bus (2549). In one example, a screen (2510) may be connected to a graphics adapter (2550). Peripheral bus architectures include PCI (Peripheral Controller Interface), USB (Universal Serial Bus), and the like.

[0249] The CPU (2541), GPU (2542), FPGA (2543), and accelerator (2544) can execute certain instructions, which, when combined, can constitute the aforementioned computer code. The computer code can be stored in ROM (2545) or RAM (2546). Transient data can also be stored in RAM (2546), while permanent data can be stored, for example, in internal mass storage (2547). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with at least one of the CPU (2541), GPU (2542), mass storage (2547), ROM (2545), RAM (2546), etc.

[0250] The computer-readable storage medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of this application, or may be medium and code well known and available to those skilled in the art of computer software.

[0251] As an example and not a limitation, a computer device having the architecture (2500), in particular the core (2540), can provide the functionality of executing software contained in at least one tangible computer-readable storage medium as a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable storage medium can be a medium associated with the above-mentioned user-accessible mass storage, as well as a specific memory of the core (2540) having non-volatile properties, such as a core internal mass storage (2547) or ROM (2545). Software for implementing various embodiments of the present application can be stored in such a device and executed by the core (2540). Depending on specific needs, the computer-readable storage medium may include one or more storage devices or chips. The software can enable the core (2540), in particular the processor therein (including a CPU, GPU, FPGA, etc.), to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM (2546) and modifying such a data structure according to a software-defined process. Additionally or alternatively, a computer device may provide functionality hardwired in logic or otherwise contained in circuitry (e.g., an accelerator (2544)) that may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to a computer-readable storage medium may include circuitry (e.g., an integrated circuit (IC)) storing the executing software, circuitry containing the executing logic, or both. The present application includes any suitable combination of hardware and software.

[0252] Although this application has described a number of exemplary embodiments, various modifications, permutations, and equivalent substitutions of the embodiments are within the scope of this application. Therefore, it should be understood that those skilled in the art will be able to design a variety of systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are therefore within the spirit and scope of this application.

[0253] Appendix A: Acronyms

[0254] JEM: Joint Development Model

[0255] VVC: Versatile Video Coding

[0256] BMS: Benchmark Collection

[0257] MV: Motion Vector

[0258] HEVC: High Efficiency Video Coding

[0259] SEI: Supplemental Enhancement Information

[0260] VUI: Video Availability Information

[0261] GOP: Group of Pictures

[0262] TU: Transform Unit

[0263] PU: prediction unit

[0264] CTU: Coding Tree Unit

[0265] CTB: Coding Tree Block

[0266] PB: prediction block

[0267] HRD: Hypothesized Reference Decoder

[0268] SNR: Signal-to-Noise Ratio

[0269] CPU: Central Processing Unit

[0270] GPU: Graphics Processing Unit

[0271] CRT: cathode ray tube

[0272] LCD: Liquid Crystal Display

[0273] OLED: Organic Light-Emitting Diode

[0274] CD: compact disc

[0275] DVD: Digital Video Disc

[0276] ROM: Read-Only Memory

[0277] RAM: Random Access Memory

[0278] ASIC: Application-Specific Integrated Circuit

[0279] PLD: Programmable Logic Device

[0280] LAN: Local Area Network

[0281] GSM: Global System for Mobile Communications

[0282] LTE: Long Term Evolution

[0283] CANBus: Controller Area Network Bus

[0284] USB: Universal Serial Bus

[0285] PCI: Peripheral Component Interconnect

[0286] FPGA: Field Programmable Gate Array

[0287] SSD: Solid State Drive

[0288] IC: integrated circuit

[0289] HDR: High Dynamic Range

[0290] SDR: Standard Dynamic Range

[0291] JVET: Joint Video Development Team

[0292] MPM: Most Probable Mode

[0293] WAIP: Wide Angle Intra Prediction

[0294] CU: Coding Unit

[0295] PU: prediction unit

[0296] TU: Transform Unit

[0297] CTU: Coding Tree Unit

[0298] PDPC: Position Dependent Prediction Combination

[0299] ISP: Intra-frame sub-partitioning

[0300] SPS: Sequence Parameter Set

[0301] PPS: Picture Parameter Set

[0302] APS: Adaptive Parameter Set

[0303] VPS: Video Parameter Set

[0304] DPS: Decoding Parameter Set

[0305] ALF: Adaptive Loop Filter

[0306] SAO: Sample Adaptive Abnormality

[0307] CC-ALF: Cross-Component Adaptive Loop Filter

[0308] CDEF: Constrained Directional Enhancement Filter

[0309] CCSO: Cross-Component Sample Bias

[0310] LSO: Local Sample Bias

[0311] LR: Loop Restoration Filter

[0312] AV1:AOM Video 1

[0313] AV2:AOM Video 2

Claims

1. A video decoding method, characterized in that: The method comprises: Determine whether the current block in the video stream is predicted in the CIIP mode; For the current block, generating a set of secondary transform coefficients; Determine, from a CIIP submode candidate set, a CIIP submode for the CIIP mode of the current block, the CIIP submode indicating an intra prediction mode used in the CIIP mode of the current block in an intra prediction mode set; wherein the CIIP submode candidate set includes at least one of the following: II_DC_PRED mode; II_V_PRED mode; II_H_PRED mode; or II_SMOOTH_PRED mode; Determining a transform kernel set from a plurality of transform kernel sets based on the intra prediction mode; wherein determining the transform kernel set from the plurality of transform kernel sets based on the intra prediction mode is only responsive to: determining that an intra prediction weight in the CIIP mode for a sample P0(x, y) of the current block is derived by formulating based on a position of the sample, wherein x and y represent a position of the sample in the current block; Determining a transform kernel for an inverse separable secondary transform or an inverse non-separable secondary transform based on the transform kernel set; Apply a combined inter-intra secondary transform in the following way: Using the determined transform kernel, perform the inverse separable secondary transform or the inverse non-separable secondary transform on the set of secondary transform coefficients to obtain a set of main transform coefficients of the current block; performing an inverse main transform on the set of main transform coefficients to obtain a residual block of the current block; and, in the CIIP mode, decoding the current block based on the residual block; The intra prediction weight of the sample P0(x, y) of the current block in the CIIP mode is determined using the following formula: Among them, WeightLUT represents a preset one-dimensional weight lookup table, sizeScale=128 / max(W,H), W is the width of the current block, and H is the height of the current block.

2. The method according to claim 1, characterized in that The determining, based on the transform kernel set, a transform kernel for the inverse separable secondary transform or the inverse non-separable secondary transform comprises: Extracting a kernel selection indicator associated with the current block from the video stream; The transformation kernel is selected from the transformation kernel set based on the kernel selection index.

3. The method according to claim 1, characterized in that The intra prediction mode set includes at least one of the following: DC_PRED mode; V_PRED mode; H_PRED mode; or SMOOTH_PRED mode.

4. The method according to claim 1, wherein include: In response to the CIIP sub-mode being the II_DC_PRED mode, determining that the transform kernel is pre-associated with the DC_PRED intra prediction mode; In response to the CIIP sub-mode being the II_V_PRED mode, determining that the transform kernel is pre-associated with a V_PRED intra prediction mode; In response to the CIIP sub-mode being the II_H_PRED mode, determining that the transform kernel is pre-associated with the H_PRED intra prediction mode; In response to the CIIP sub-mode being the II_SMOOTH_PRED mode, determining that the transform kernel is pre-associated with the SMOOTH_PRED intra prediction mode.

5. The method according to claim 2, characterized in that The intra prediction weight depends on the intra prediction mode and decreases along the prediction direction of the intra prediction.

6. The method according to claim 1, characterized in that The intra prediction weight associated with the current block in the CIIP mode includes a spatial weight mode in a predetermined spatial weight mode set.

7. The method according to claim 6, characterized in that include: Based on the spatial weight pattern, a transform kernel used in the inverse non-separable quadratic transform is determined.

8. The method according to claim 6, characterized in that include: Extracting a spatial weight mode indicator from the video stream, the spatial weight mode indicator indicating the spatial weight mode in the predetermined spatial weight mode set; The spatial weight pattern is determined according to the spatial weight pattern indicator.

9. A video encoding method, characterized in that: The method comprises: Predict the current block in the video stream in the inter-frame and intra-frame joint prediction CIIP mode; For the current block, generating a set of secondary transform coefficients; Determine, from a CIIP submode candidate set, a CIIP submode for the CIIP mode of the current block, the CIIP submode indicating an intra prediction mode used in the CIIP mode of the current block in an intra prediction mode set; wherein the CIIP submode candidate set includes at least one of the following: II_DC_PRED mode; II_V_PRED mode; II_H_PRED mode; or II_SMOOTH_PRED mode; Determining a transform kernel set from a plurality of transform kernel sets based on the intra prediction mode; wherein determining the transform kernel set from the plurality of transform kernel sets based on the intra prediction mode is only responsive to: determining that an intra prediction weight in the CIIP mode for a sample P0(x, y) of the current block is derived by formulating based on a position of the sample, wherein x and y represent a position of the sample in the current block; determining a transform kernel for a separable secondary transform or a non-separable secondary transform based on the transform kernel set; Apply a combined inter-intra secondary transform in the following way: Using the determined transform kernel, perform a separable secondary transform or a non-separable secondary transform on the secondary transform coefficient set to obtain a main transform coefficient set of the current block; performing a main transform on the main transform coefficient set to obtain a residual block of the current block; and In the CIIP mode, encoding the current block based on the residual block; The intra prediction weight of the sample P0(x, y) of the current block in the CIIP mode is determined using the following formula: Among them, WeightLUT represents a preset one-dimensional weight lookup table, sizeScale=128 / max(W,H), W is the width of the current block, and H is the height of the current block.

10. A video decoding device, characterized in that: comprising a memory for storing computer instructions and a processor in communication with the memory, wherein when the processor executes the computer instructions, the processor is configured to: Determine whether the current block in the video stream is predicted in the CIIP mode; For the current block, generating a set of secondary transform coefficients; Determine, from a CIIP submode candidate set, a CIIP submode for the CIIP mode of the current block, the CIIP submode indicating an intra prediction mode used in the CIIP mode of the current block in an intra prediction mode set; wherein the CIIP submode candidate set includes at least one of the following: II_DC_PRED mode; II_V_PRED mode; II_H_PRED mode; or II_SMOOTH_PRED mode; Determining a transform kernel set from a plurality of transform kernel sets based on the intra prediction mode; wherein determining the transform kernel set from the plurality of transform kernel sets based on the intra prediction mode is only responsive to: determining that an intra prediction weight in the CIIP mode for a sample P0(x, y) of the current block is derived by formulating based on a position of the sample, wherein x and y represent a position of the sample in the current block; determining a transform kernel for a separable secondary transform or a non-separable secondary transform based on the transform kernel set; Apply a combined inter-intra secondary transform in the following way: Using the determined transform kernel, perform an inverse separable secondary transform or an inverse non-separable secondary transform on the secondary transform coefficient set to obtain a primary transform coefficient set of the current block; performing an inverse main transform on the set of main transform coefficients to obtain a residual block of the current block; and, in the CIIP mode, decoding the current block based on the residual block; The intra prediction weight of the sample P0(x, y) of the current block in the CIIP mode is determined by the following formula, including: Among them, WeightLUT represents a preset one-dimensional weight lookup table, sizeScale=128 / max(W,H), W is the width of the current block, and H is the height of the current block.

11. The device according to claim 10, characterized in that The processor is configured to: A kernel selection index associated with the current block is extracted from the video code stream; and the transform kernel is selected from the transform kernel set based on the kernel selection index.

12. A computer device, characterized in that: The system comprises a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

13. A non-transitory computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor, the processor is caused to implement the method according to any one of claims 1 to 9.

14. A method for storing a video bitstream, characterized in that: A video bit stream is stored on a non-volatile computer-readable medium, where the video bit stream is decoded according to the video decoding method according to any one of claims 1 to 8, or is generated based on the video encoding method according to claim 9.

Images