Method, apparatus, and program for video decoding, and method for video encoding

Abstract

The processing circuit decides to use at least two prediction models for different samples in the current block. The processing circuit generates a predicted value for the first sample in the current block according to the first prediction model and generates a predicted value for the second sample in the current block according to a second prediction model different from the first prediction model. The processing circuit decides whether to apply a filter to the current sample in the current block based on whether one or more adjacent neighboring samples use a different prediction model than the current sample. Then, in response to the decision to apply a filter, the processing circuit reconstructs the current sample based on the filtered output from the filter, with the predicted values ​​of the current sample and one or more adjacent neighboring samples as inputs to the filter.

Description

[Technical Field]

[0001] This disclosure describes embodiments that generally relate to video coding. [Background technology]

[0002] The background information provided herein is for general purposes only. The research of the currently designated inventors is not considered prior art to this disclosure, either explicitly or implicitly, to the extent described in the background section, and any aspects of the description that would otherwise not qualify as prior art at the time of filing are not considered prior art to this disclosure.

[0003] Image / video compression can help transmit image / video data between different devices, storage, and networks with minimal quality degradation. In some cases, video codec technology can compress video based on spatial and temporal redundancy. In one example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the currently reconstructed picture for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can predict samples in the current picture from a previously reconstructed picture using motion compensation. Motion compensation can be indicated by motion vectors (MV). [Overview of the Initiative]

[0004] Aspects of this disclosure include methods and apparatus for video coding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit receives a coded video bitstream containing coded information for the current block in the current picture. The coded information instructs the use of at least two predictive models for different samples in the current block. The processing circuit generates a predicted value for a first sample in the current block according to a first predictive model and generates a predicted value for a second sample in the current block according to a second predictive model different from the first predictive model. The processing circuit determines whether a filter should be applied to the current sample in the current block, based on whether one or more adjacent neighboring samples of the current sample use a different predictive model than the current sample. Then, in response to the decision to apply a filter, the processing circuit reconstructs the current sample based on the filtered output from a filter, the predicted values ​​of the current sample and one or more adjacent neighboring samples becoming the inputs to the filter.

[0005] In some examples, the coded information instructs the use of multi-model cross-component prediction, which predicts the second color component based on the first color component. The processing circuit generates a first predicted value of the second color component based on a first reconfigured value of the first color component according to a first prediction model for a first sample within the first sample, and generates a second predicted value of the second color component based on a second reconfigured value of the first color component according to a second prediction model for a second sample within the second sample.

[0006] In some examples, the coded information instructs the use of multiple Local Illumination Compensation (LIC) models, and the processing circuit applies the first LIC model to the first class of samples to produce the predicted value for the first sample, and applies the second LIC model to the second class of samples to produce the predicted value for the second sample.

[0007] In some examples, each of at least two prediction models is a linear model that defines a linear combination of a plurality of terms.

[0008] In some examples, the filter includes at least a first term of the current sample and at least a second term of an adjacent neighboring sample.

[0009] In some examples, the processing circuit determines to apply the filter when an adjacent neighboring sample of the current sample is using a prediction model different from the current sample.

[0010] In some examples, the processing circuit determines to apply the filter when each adjacent neighboring sample of the current sample is using a prediction model different from the current sample.

[0011] In some examples, one or more adjacent neighboring samples of the current sample are within the current block.

[0012] In some examples, the processing circuit decodes a flag indicating whether to apply a filter in the current block in response to the coded information indicating the use of at least two prediction models in the current block.

[0013] In some examples, the processing circuit determines whether to apply a filter in the current block based on a ratio of the number of samples of different prediction models. For example, the processing circuit calculates the ratio as the number of first samples to the number of second samples of the second sample. In other examples, the processing circuit calculates the ratio as the number of samples of one of the first prediction model and the second prediction model to the total number of samples in the current block.

[0014] In some examples, the processing circuit determines whether a ratio satisfies a requirement based on a comparison of the ratio with a threshold, and in response to the ratio satisfying the requirement, decides to apply a filter in the current block. The ratio is defined as at least one of a constant value or the reciprocal of a constant value. For example, a constant value is predefined. In other examples, the constant value is determined from high-level syntax.

[0015] Aspects of this disclosure also provide non-temporary computer-readable media that, when executed by a computer, stores instructions causing a computer to perform a video decoding / encoding method.

[0016] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the following detailed description and accompanying drawings. [Brief explanation of the drawing]

[0017] [Figure 1] This is a schematic diagram of an exemplary block diagram of a communication system (100). [Figure 2] This is a schematic diagram of an exemplary block diagram of a decoder. [Figure 3] This is a schematic diagram of an example block diagram of an encoder. [Figure 4] In some examples, diagrams of neighboring sample locations for parameter calculation are shown. [Figure 5] In several examples, diagrams illustrating the effect of the slope adjustment parameters are shown. [Figure 6] As an example, the position of the luma sample location, which is the input to the spatial 5-tap component, is shown. [Figure 7] The diagram shows the reference area, which includes the rows of chroma samples above and to the left of the prediction unit. [Figure 8] In several examples, diagrams of four Sobel-based gradient filter patterns are shown. [Figure 9] In several examples, the diagrams show the application of a second filter after multi-model cross-component prediction. [Figure 10] A flowchart illustrating another process according to some embodiments of this disclosure is shown. [Figure 11] A flowchart illustrating the process according to some embodiments of this disclosure is shown. [Figure 12] This is a schematic diagram of a computer system according to an embodiment. [Modes for carrying out the invention]

[0018] Figure 1 shows a block diagram of a video processing system (100) in several examples. The video processing system (100) is an example of the application of the disclosed subject, namely a video encoder and video decoder in a streaming environment. The disclosed subject can similarly be applied to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0019] The video processing system (100) includes a capture subsystem (113) which may include a video source (101), such as a digital camera, that generates a stream (102) of uncompressed video pictures. For example, the stream (102) of video pictures includes a sample captured by the digital camera. The stream (102) of video pictures is represented by a thick line to highlight its higher data volume compared to encoded video data (104) (or encoded video bitstream) and may be processed by an electronic device (120) which includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement the aspects of the subject disclosed, as will be described in more detail below. The encoded video data (104) (or encoded video bitstream) is represented by a thin line to highlight its lower data volume compared to the stream (102) of video pictures and may be stored in a streaming server (105) for future use. One or more streaming client subsystems, such as client subsystems (106) and (108) in Figure 1, can access a streaming server (105) to read copies (107) and (109) of the encoded video data (104). Client subsystem (106) may include a video decoder (110), for example, in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and generates an output stream (111) of a video picture that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstreams) may be encoded according to a specific video coding / compression standard. An example of such a standard is ITU-T Recommendation H.265.For example, a video coding standard under development is commonly known as Versatile Video Coding (VVC). The disclosed material may be used in connection with VVC.

[0020] Electronic devices (120) and (130) may include other components (not shown). For example, electronic device (120) may include a video decoder (not shown), and electronic device (130) may similarly include a video encoder (not shown).

[0021] Figure 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) may be contained within an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). The video decoder (210) may be used in place of the video decoder (110) in the example of Figure 1.

[0022] The receiver (231) may receive one or more coded video sequences, for example, contained in a bitstream, which are to be decoded by the video decoder (210). In embodiments, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device storing coded video data. The receiver (231) may receive coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective usage entities (not shown). The receiver (231) may isolate the coded video sequences from other data. To counteract network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). Otherwise, it can be outside the video decoder (210) (not shown). Further variations may include a buffer memory outside the video decoder (210) (not shown), for example, to counter network jitter, and another buffer memory (215) within the video decoder (210), for example, to manipulate playback timing. When the receiver (231) is receiving data from a sufficiently bandwidth and controllable storage / transfer device or from an isosynchronous network, the buffer memory (215) may not be necessary or may be small. For use in best-effort packet networks such as the Internet, the buffer memory (215) may be necessary, may be relatively large, and advantageously, may be adaptively sized, and may be implemented at least partially outside the video decoder (210) in an operating system or similar element (not shown).

[0023] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from the coded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (210) and information for controlling rendering devices, such as a render device (212) (e.g., a display screen), which are not essential parts of the electronic device (230) but can be coupled to the electronic device (230) as shown in Figure 2. The control information for rendering devices may take the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-dependent or independent arithmetic coding, etc. The parser(220) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to that group. Subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), predictive units (PU), and so on. The parser(220) may also extract information from the coded video sequence such as transform coefficients, quantization parameter values, and motion vectors.

[0024] The parser (220) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (215) in order to generate symbols (221).

[0025] The reconstruction of symbol (221) can have a number of different units depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. How each unit is included may be controlled by subgroup control information parsed by parser (220) from the coded video sequence. The flow of such subgroup control information between parser (220) and the following multiple units is not shown for clarity.

[0026] Beyond the functional blocks already described, the video decoder (210) can be conceptually subdivided into numerous functional units, as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated with one another. However, for the purpose of illustrating the subject matter disclosed, the conceptual subdivision into functional units described below is appropriate.

[0027] The first unit is the scaler / inverse unit (251). The scaler / inverse unit (251) receives control information from the parser (220) as symbols (221), along with the quantized transformation coefficients, including which transformation to use, block size, quantization coefficients, and quantization scaling matrix. The scaler / inverse unit (251) can output a block containing sample values ​​that can be input to the aggregator (255).

[0028] In some cases, the output samples of the scaler / inverse unit (251) may relate to intracoded blocks. Intracoded blocks are blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from portions reconstructed before the current picture. Such predictive information may be supplied by an intrapicture predictive unit (252). In some cases, the intrapicture predictive unit (252) generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (255) adds the predictive information generated by the intra predictive unit (252) to the output sample information supplied by the scaler / inverse unit (251), sample by sample.

[0029] In other cases, the output samples of the scaler / inverse unit (251) may relate to an interconnected and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (253) may access the reference picture memory (257) to fetch samples to be used for prediction. After the fetched samples have been motion-compensated according to the symbols (221) relating to the block, those samples may be added by the aggregator (255) to the output of the scaler / inverse unit (251) (referred to in this case to residual samples or residual signal) to generate output sample information. The address in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches the predicted samples may be controlled by a motion vector that can be made available to the motion-compensated prediction unit (253) in the form of a symbol (221) which may have, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (257) when the precise motion vectors of subsamples are used, as well as motion vector prediction mechanisms.

[0030] The output samples of the aggregator (255) can undergo various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filtering techniques, which are included in the coded video sequence (also called the coded video bitstream) and controlled by parameters made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may also respond to metadata obtained during the decoding of previous parts (in the decoding order) of the coded picture or coded video sequence, and further, to previously reconstructed loop-filtered sample values.

[0031] The output of the loop filter unit (256) can be a sample stream that is output to the render device (212) and can also be stored in reference picture memory (257) for use in future interpicture prediction.

[0032] A particular coded picture, once fully reconfigured, may be used as a reference picture for future predictions. For example, once the coded picture corresponding to the current picture is fully reconfigured and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and any unused current picture buffer may be reallocated before the reconfiguration of subsequent coded pictures begins.

[0033] The video decoder (210) may perform decoding operations in accordance with a specified video compression technique or standard, such as the ITU-T recommended H.265. The coded video sequence may conform to the syntax defined by the video compression technique or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all the tools available in the video compression technique or standard as the only tool available for use under that profile. Furthermore, the complexity of the coded video sequence must be within the boundaries defined by the level of the video compression technique or standard in order to comply. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further limited through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management notified in the coded video sequence.

[0034] In an embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may also be included as parts of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a time, space, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, etc.

[0035] Figure 3 shows an exemplary block diagram of a video encoder (303). The video encoder (303) is contained within an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) can be used as a replacement for the video encoder (303) in the example of Figure 1.

[0036] The video encoder (303) may receive video samples from a video source (301) (not part of the electronic device (320) in the example in Figure 3) that can capture video images to be coded by the video encoder (303). In other examples, the video source (301) is part of the electronic device (320).

[0037] The video source (301) may supply a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (301) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be supplied as a series of individual pictures that give motion when viewed sequentially. The picture itself may be organized as a spatial array of pixels, each pixel may have one or more samples depending on the sampling structure, color space, etc., in use. This specification will focus on samples below.

[0038] According to the embodiment, the video encoder (303) can encode and compress pictures of a source video sequence into a coded video sequence (343) in real time or under any other time constraints required. Enforcing an appropriate coding speed is a function of the controller (350). In some embodiments, the controller (350) controls and is functionally coupled to other functional units, such as those described below. Couplings are not shown for clarity. Parameters set by the controller (350) may include parameters related to rate control (e.g., picture skip, quantizer, lambda value of rate distortion optimization technique), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured to have other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0039] In some embodiments, the video encoder (303) is configured to operate in a coding loop. In an overly simplified description, in the example, the coding loop may include a source coder (330) (which, for example, is involved in generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a similar manner to how a (remote) decoder would also generate. The reconstructed sample stream (sample data) is fed into a reference picture memory (334). Since the decoding of the symbol stream yields a bit-exact result independent of the decoder's location (local or remote), the contents within the reference picture memory (334) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the same sample values ​​as reference picture samples that the decoder would "see" when using predictions during decoding. This fundamental principle of reference picture synchronization (and the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is also used in several related technologies.

[0040] The operation of the “local” decoder (333) can be the same as that of the “remote” decoder, such as the video decoder (210), which has already been described in detail above with reference to Figure 2. Referring temporarily to Figure 2, however, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), does not have to be fully implemented in the local decoder (233), provided that symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder (345) and parser (220) can be reversible.

[0041] In embodiments, decoder techniques, excluding the parsing / entropy decoding present in the decoder, exist in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. Descriptions of encoder techniques can be omitted, as they are the reverse of the decoder techniques, which are comprehensively described. In certain scopes, more detailed descriptions are given below.

[0042] During operation, in some examples, the source coder (330) may perform motion-compensated predictive coding. This predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this way, the coding engine (332) codes the difference between the pixel blocks of the reference picture that may be selected as predictive references for the input picture and the pixel blocks of the input picture.

[0043] A local video decoder (333) can decode coded video data of a picture that may be designated as a reference picture, based on symbols generated by the source coder (330). The operation of the coding engine (332) can, advantageously, be an irreversible process. When coded video data can be decoded by a video decoder (not shown in Figure 3), the reconstructed video sequence is typically a copy of the source video sequence with some errors. The local video decoder (333) may replicate the decoding process that the video decoder may perform on the reference picture, so that the reconstructed reference picture is stored in the reference picture memory (334). In this way, the video encoder (303) can locally store a copy of the reconstructed reference picture that has the same content as the reconstructed reference picture that will be acquired by the far-end video decoder (without transmission errors).

[0044] The predictor (335) may perform a predictive search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which may serve as appropriate predictive criteria for the new picture. The predictor (335) may operate on a sample block-by-pixel block basis to find appropriate predictive criteria. In some cases, the input picture may have predictive criteria drawn from multiple reference pictures stored in the reference picture memory (334), as determined by the search results obtained by the predictor (335).

[0045] The controller (350) may manage the coding operation of the source coder (330), including, for example, setting parameters and subgroup parameters used to encode video data.

[0046] The outputs of all the above functional units can undergo entropic coding in the entropicorder (345). The entropicorder (345) converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0047] The transmitter (340) may buffer the coded video sequence generated by the entropicorder (345) to prepare it for transmission over the communication channel (360). The communication channel (360) may be a hardware / software link to a storage device that stores the coded video data. The transmitter (340) may merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0048] The controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign each coded picture to a specific coding picture type that may affect the coding techniques that may be applied to each picture. For example, a picture may often be assigned as one of the following picture types:

[0049] An intra-picture (I-picture) can be encoded and decoded without using any other pictures in the sequence as a source for prediction. Some video codecs allow various types of intra-pictures, including, for example, independent decoder refresh (IDR) pictures.

[0050] A predictive picture (P-picture) can be encoded and decoded by intra-prediction or inter-prediction, which uses motion vectors and reference indices to predict the sample values ​​of each block.

[0051] A bidirectionally predictive picture (B-picture) can be encoded and decoded using intra-prediction or inter-prediction, which uses two motion vectors and a reference index to predict the sample values ​​for each block. Similarly, multiple-predictive picture(s) can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0052] A source picture can generally be spatially subdivided into multiple sample blocks (e.g., 4x4, 8x8, 4x8, or 16x16 sample blocks, respectively), and each block can be coded. Blocks can be predictively coded by referencing other (already coded) blocks, determined by the coding assignment applied to each picture in the block. For example, blocks in picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks in picture P may be coded predictively by spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks in picture B may be coded predictively by spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0053] The video encoder (303) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Recommended H.265. During these operations, the video encoder (303) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax defined by the video coding technique or standard being used.

[0054] In an embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0055] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, is partitioned into blocks. If a block within the current picture is similar to a reference block in a previously coded, still-buffering reference picture within the video, that block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are used.

[0056] In some embodiments, a dual prediction technique may be used in interpicture prediction. According to the dual prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture, both of which precede the current picture in the video in the decoding order (but may be past and future, respectively, in the display order). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. That block is predictable by a combination of the first and second reference blocks.

[0057] Furthermore, merge mode techniques can be used in interpicture prediction to improve coding efficiency.

[0058] In some embodiments of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one lumen CTB and two chroma CTBs. Each CTU can be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. As an example, each CU is parsed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU contains one lumar prediction block (PB) and two chroma PBs. In embodiments, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a lumar prediction block as an example of a prediction block, the prediction block contains a matrix of pixel values ​​(e.g., lumar values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0059] The video encoders (103) and (303) and the video decoders (110) and (210) can be implemented by any suitable technology. In one embodiment, the video encoders (103) and (203) and the video decoders (110) and (210) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (203) and the video decoders (110) and (210) can be implemented using one or more processors that execute software instructions.

[0060] Some aspects of this disclosure provide techniques for smoothing predictions within a block, such as filtered cross-component prediction.

[0061] In some examples, cross-component prediction may include a first technique called the Cross-Component Linear Model (CCLM), a second technique called the Multi-Model Linear Model (MMLM), a third technique called the Convolutional Cross-Component Model (CCCM), and a fourth technique called the Gradient Linear Model (GLM).

[0062] In some examples (e.g., VVC), the first technique, CCLM, is used to reduce cross-component redundancy. In CCLM, the chromatic sample uses a linear model, e.g., equation (1): Nod C (i,j)=a·rec L '(i,j)+b Equation (1) By using this, predictions are made based on the same CU reconfigured luma sample. Here, pred C (i,j) represents the predicted chroma sample within the CU, and rec L'(i,j) represents a downsampled and reconstructed lumen sample of the same CU. The CCLM linear model includes parameters (a and b) that can be derived using, for example, at most four neighboring chromen samples and their corresponding downsampled lumen samples.

[0063] In some cases, based on the position of neighboring chroma samples, CCLM may include various modes called LM_T (LM top mode or upper mode LM_A), LM_L (LM left mode), and LM_LT (LM left top mode or upper left mode LM_LA or simply LM mode). For example, if the current chroma block size is W×H, then W' and H' can be set for the various modes in CCLM. When LM mode (also called LM_LT or LM_LA) is applied, W'=W and H'=H; when LM_A mode is applied, W'=W+H; and when LM_L is applied, H'=H+W.

[0064] The upper adjacent position is denoted as S[0,-1]···S[W'-1,-1], and the left adjacent position is denoted as S[-1,0]···S[-1,H'-1]. Then, four positions are selected. For example, if LM mode is applied and both the upper adjacent sample and the left adjacent sample are available, the four positions can include S[W' / 4,-1], S[3×W' / 4,-1], S[-1,H' / 4] and S[-1,3×H' / 4]; if LM_A mode is applied or only the upper adjacent sample is available, the four positions can include S[W' / 8,-1], S[3×W' / 8,-1], S[5×W' / 8,-1] and S[7×W' / 8,-1]; and if LM_L mode is applied or only the left adjacent sample is available, the four positions can include S[-1,H' / 8], S[-1,3×H' / 8], S[-1,5×H' / 8] and S[-1,7×H' / 8].

[0065] The four neighboring lumens samples at the selected location are x 0 A and x1 A and the two larger values denoted as, x 0 B and x 1 B are downsampled and compared to find the two smaller values denoted as, y 0 A , y 1 A , y 0 B and y 1 B Then, the intermediate parameters X a , X b , Y a and Y b are derived as in Equation (2): X a =(x 0 A +x 1 A +1)>>1; X b =(x 0 B +x 1 B +1)>>1; Y a =(y 0 A +y 1 A +1)>>1; Equation (2) Y b =(y 0 B +y 1 B +1)>>1

[0066] Finally, the linear model parameters a and b are obtained according to Equations (3) and (4): a=(Y a -Y b ) / (X a -X b ) Equation (3) b=Y b -a·X b Equation (4)

[0067] Figure 4 shows a diagram of neighboring sample locations for parameter calculation of CCLM in several examples. Referring to Figure 4, neighboring sample pairs (lumen samples and chroma samples) for deriving the parameters of CCLM prediction are indicated by shaded circles. In the example in Figure 4, the size of the current lumen block is 2N × 2N and the size of the current chroma block is N × N. The neighboring sample pairs can include 2N reference sample pairs, for example, 2N reference samples adjacent to the current chroma block and 2N reference samples adjacent to the current lumen block. In accordance with aspects of this disclosure, the neighboring sample locations in Figure 4 may be used when the LM mode (also called LM_LT or LM_LA) is applied.

[0068] In some examples, the division operation to calculate the parameter a is performed using a lookup table. To reduce the memory required to store the lookup table, the diff value (the difference between the maximum and minimum values) and the parameter a are expressed in exponential notation. For example, diff is approximated by a 4-bit effective part and an exponent. As a result, the lookup table for 1 / diff can be reduced to 16 elements for 16 values ​​in the mantissa, as shown in equation (5). Using the above technique has the advantage of reducing both the complexity of the calculation and the memory size required to store the necessary lookup table. DivTable[]= {0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} Formula (5)

[0069] In some examples, in LM_T (also called LM_A) mode, the above template is used to compute the parameters of a linear model. To obtain more samples, the above template is expanded to (W'=W+H) samples. In some examples, in LM_L mode, the left template is used to compute the parameters of a linear model. To obtain more samples, the left template is expanded to (H'=H+W) samples.

[0070] In some examples, in LM_LT mode, the left and upper templates are used to compute the parameters of a linear model. For example, two downsampling filters are applied to chroma samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions, matching the chroma sample positions of a 4:2:0 video sequence. The selection of downsampling filters can be specified by a flag at the SPS level. The two downsampling filters can be represented by equations (6) and (7), corresponding to "type-0" and "type-2" content, respectively:

number

[0071] In some cases, when the referenced row above is on a CTU boundary, only one lumern row (e.g., a typical row buffer for intra-prediction) is used to supply the downsampled lumern sample.

[0072] In some examples, parameter calculation is performed not only as an encoder search operation but also as part of the decoding process. As a result, no syntax is used to communicate the values ​​of the parameters (a and b) to the decoder.

[0073] In some examples, for chromatic intramode coding, a total of eight intramodes are allowed for chromatic intramode coding. These modes include five conventional intramodes and three cross-component linear model modes (e.g., CCLM (KM), LM_A, and LM_L).

[0074] Table 1 provides information for the process of signal notification and derivation of chroma modes. In some examples, chroma mode coding directly depends on the intra-prediction mode of the corresponding lumen block. Since separate block partitioning structures for lumen and chroma components are enabled in I-slice, one chroma block can correspond to multiple lumen blocks. Therefore, in the case of chroma DM mode, the intra-prediction mode of the corresponding lumen block covering the center position of the current chroma block is directly inherited. [Table 1]

[0075] In some cases, a single binarization table, as shown in Table 2, is used regardless of the value of sps_cclm_enabled_flag. [Table 2]

[0076] In Table 2, the first bin can indicate whether it is regular (0) or LM mode (1). If it is LM mode, the next bin indicates whether it is LM_CHOROMA (0). If it is not LM_CHOROMA, the next bin indicates whether it is LM_L (0) or LM_A (1). In some examples, if sps_cclm_enabled_flag is 0, the first bin of the corresponding intra_chroma_pred_mode binarization table may be discarded before entropic coding. In other words, the first bin is presumed to be 0 and is therefore not coded. This single binarization table is used for both cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two bins in Table 2 are context-coded in their own context models, while the remaining bins are bypass-coded.

[0077] In some cases, to reduce rumor-chroma latency in a dual tree, chroma CUs in 32x32 / 32x16 chroma coding tree nodes are allowed to use CCLM if the 64x64 rumor coding tree nodes are Not Split (and ISP is not used for 64x64 CUs) or partitioned with QT. For example, if a 32x32 chroma node is not split or is partitioned with QT, all chroma CUs within the 32x32 node can use CCLM, and if a 32x32 chroma node is partitioned with horizontal BT and the 32x16 child nodes are not split or use vertical BT partitioning, all chroma CUs within the 32x16 chroma node can use CCLM. For all other rumor and chroma coding tree partitioning conditions, CCLM is not allowed for chroma CUs.

[0078] In some examples, CCLM is extended by using a second technique, multi-model LM (MMLM). In MMLM mode, the threshold is calculated as the mean of the lumens reconstructed neighboring samples. The reconstructed neighboring samples are then classified into two classes by the threshold, e.g., a first class of reconstructed neighboring samples greater than the threshold and a second class of reconstructed neighboring samples less than the threshold. Linear models for each class are derived, for example, using the least squares method (LMS). In some examples, slope adjustment can be applied to both CCLM and MMLM. Slope adjustment can tilt the linear function that maps lumens to chromens with respect to a center point determined by the mean lumens of the reference samples within the neighboring samples.

[0079] In some examples, slope adjustment is applied to CCLM. For instance, CCLM uses a model such as the linear model in Equation 1 with two parameters to map lumen values ​​to chroma values. Parameter a is called the slope parameter, and parameter b is called the bias parameter.

[0080] For example, the slope adjustment parameter "u" for the slope parameter is signaled to update the model to equation (8): Nod C (i,j)=a'·rec L '(i,j)+b' Equation (8) Here, a'=a+u and b'=bu×y r Therefore, with this selection, the mapping function is the luminance value y r The model is tilted or rotated around a point that has y. To make meaningful changes to the model, the mean of the reference luma samples is used when the model is created. r It is used as such.

[0081] Figure 5 shows diagrams illustrating the effects of the slope adjustment parameter in several examples. Figure 5 includes the first model (510) and the second model (520). The first model (510) is generated according to CCLM without slope adjustment, and the second model (520) is generated according to CCLM with slope adjustment.

[0082] In some embodiments, the slope adjustment parameter u is supplied as an integer between -4 and 4 and is signaled in the bitstream. The unit of the slope adjustment parameter u is 1 / 8 of the chroma sample value per luma sample value (for 10-bit content).

[0083] In some examples, tilt adjustment is available for CCLM models that use reference samples both above and to the left of a block ("LM_CHOROMA_IDX" and "MMLM_CHROMA_IDX"), but not in "single-side" mode. This choice is based on a consideration of the trade-off between coding efficiency and complexity. For example, tilt adjustment is available in LM mode, but not in LM_A mode or LM_L mode.

[0084] In some examples, when slope adjustment is applied to a multimode CCLM model, both models may be adjusted, so up to two slope updates are signaled for a single chroma block. For example, in a multimode CCLM (e.g., MMLM), two linear models are derived for two classes. Two slope adjustment parameters are signaled and applied to the two linear models respectively to generate the updated models used in the MMLM.

[0085] In some cases, the encoder can perform a SATD (sum of absolute transform difference)-based search to find the first best value for the Cr tilt adjustment parameter, and a similar SATD-based search to find the second best value for the Cb tilt adjustment parameter. If either of these is a non-zero tilt adjustment parameter, the combined tilt adjustment pair (first best value, second best value) is included in the TU's RD check list.

[0086] In some cases, a third technique called the Convolutional Cross-Component Model (CCCM) can be used to predict the chroma sample from the reconstructed lumen sample, using a similar approach to the CCLM mode in ECM-6.0. In CCCM, as with CCLM, the reconstructed lumen sample is downsampled to fit a lower-resolution chroma grid when chroma subsampling is used. Similar to CCLM, a reference sample to the left or above and to the left can be used as a template for model derivation. For example, the above reference sample can be used as a template for model derivation. In other examples, the left reference sample can be used as a template for model derivation. In other examples, the left and above reference samples can be used as templates for model derivation.

[0087] Furthermore, similar to CCLM, there is the option to use single-model or multi-model variations of CCCM. For example, the multi-model variation uses two models, one derived for samples above the mean lumen reference value, and the other for the remaining samples (following the CCLM design concept). In some examples, the multi-model CCCM mode may be chosen for PUs where at least 128 reference samples are available.

[0088] In CCLM, a convolutional filter is used. In some examples, the convolutional filter is a 7-tap convolutional filter. A convolutional 7-tap filter may include a first term of a 5-tap plus sign shaped spatial component (also called a spatial 5-tap component, which can have a cross shape, also called a plus sign shape), a second term of a nonlinear term P, and a third term of a bias term.

[0089] Figure 6 shows, as an example, the location of the lumar sample position, which is the input to the spatial 5-tap component. The input to the spatial 5-tap component of the convolutional 7-tap filter includes the central (C) lumar sample that is collocated with the chroma sample to be predicted, and the neighborhoods of up / north (N), down / south (S), left / west (W), and right / east (E).

[0090] In some examples, the nonlinear term P is expressed as the central luma sample C raised to the power of 2 and scaled to the sample value range of the content, for example, according to equation (9): P=(C×C+midVal)>>bitDepth Formula (9) For example, in the case of 10-bit content, the nonlinear term P is calculated according to equation (10): P=(C×C+512)>>10 Formula (10)

[0091] In some examples, the bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM), and is set to the middle chroma value (512 for 10-bit content) as an example.

[0092] In some examples, the output of a convolutional 7-tap filter is the filter coefficient c. i It is calculated as a convolution between the input value and is clipped to the range of valid chroma samples, for example, according to equation (11): predChromaVal =c0C+c1N+c2S+c3E+c4W+c5P+c6B Equation (11)

[0093] In some examples, the filter coefficient c i This can be determined (calculated) by minimizing the mean squared error (MSE) between the predicted chroma sample and the reconstructed chroma sample within the reference area.

[0094] Figure 7 shows a diagram representing a reference area containing six chroma sample rows above and to the left of the PU. The reference area extends to the right of the PU boundary by 1 PU width and below by 1 PU height. In some examples, the reference area is adjusted to include only available samples. In some examples, the extension of the reference area is used to support “side samples” of a plus-shaped spatial filter, which are padded if they fall within an unavailable area.

[0095] In some examples, MSE minimization is performed by computing the autocorrelation matrix of the lumern input and the cross-correlation vector between the lumern input and the chroma output. The autocorrelation matrix is ​​LDL decomposed, and the final filter coefficients are computed using back substitution. For example, the MSE minimization process for filter coefficients is roughly similar to the computation of ALF filter coefficients in ECM, but LDL decomposition is used instead of Cholesky decomposition to avoid the use of square root operations.

[0096] In some examples, the autocorrelation matrix is ​​calculated using reconstructed values ​​of lumern and chroma samples. For example, these samples are within a sufficient range (e.g., between 0 and 1023 for 10-bit content) to yield relatively large values ​​in the autocorrelation matrix. This requires high-bit-depth arithmetic during model parameter calculation. In some examples, removing a fixed offset from the lumern and chroma samples within each PU may be used per model. This reduces the magnitude of the values ​​used in model creation and lowers the precision required for fixed-point arithmetic. As a result, 16-bit decimal precision may be used instead of the 22-bit precision of the original CCCM implementation.

[0097] In some examples, a reference sample value located just outside the upper-left corner of the PU is used as the offset (offsetLuma, offsetCb, and offsetCr) for simplicity. The sample values ​​used in both model building and final prediction (i.e., the lumens and chromas in the reference area, as well as the lumens currently in the PU) are subtracted by these fixed values. For example, the sample values ​​could include C'=C-offsetLuma, N'=N-offsetLuma, S'=S-offsetLuma, E'=E-offsetLuma, W'=W-offsetLuma, P'=nonLinear(C'), and B=midValue=1<<(bitDepth-1), and the chroma value is predicted using equation (12), where offsetChroma is equal to offsetCr and offsetCb for the Cr and Cb components, respectively: predChromaVal =c0C'+c1N'+c2S'+c3E'+c4W'+c5P'+c6B+offsetChroma Formula (12)

[0098] In some cases, the lumen offset is removed during lumen reference sample interpolation to avoid any additional sample-level calculations. This can be done, for example, by replacing the rounding term used in lumen reference sample interpolation with an updated term that includes both the rounding term and offsetLuma. The chromen offset can be removed by directly subtracting the chromen offset from the reference chromen sample. Alternatively, the effect of the chromen offset can be removed from the cross-component vector to obtain the same result. To return the chromen offset to the output of the convolution prediction operation, the chromen offset is added to the bias term of the convolution model.

[0099] In some cases, the CCCM model parameter calculation process requires division. Division is not always easy to implement. Division can be replaced by multiplication (with a scaling factor) and shift operations, where the scaling factor and the number of shifts are calculated based on the denominator, similar to how they are used in CCCM parameter calculations.

[0100] In some examples, a fourth technique, the Gradient Linear Model (GLM), is used. For example, in the case of the YUV 4:2:0 color format, the Gradient Linear Model (GLM) method can be used to predict the chroma sample from the lumana sample gradient. Two modes are supported: a two-parameter GLM mode and a three-parameter GLM mode.

[0101] Compared to CCLM, the two-parameter GLM utilizes the lumen sample gradient to derive a linear model, instead of downsampled lumen values. Specifically, when a two-parameter GLM is applied, the input to the GGLM process, i.e., the downsampled lumen sample L, is replaced by the lumen sample gradient G. Other parts of the CCLM (e.g., parameter derivation, linear transformation of predicted samples) remain unchanged. As an example, the predicted C is calculated according to equation (13): C=a·G+b Equation (13)

[0102] In some examples, for a 3-parameter GLM, the chroma sample can be predicted based on both the lumen sample gradient and the lumen value downsampled by different parameters. The model parameters for the 3-parameter GLM are derived from six row and column neighboring samples by an MSE minimization method based on LDL partitioning, as used in CCCM. As an example, the predicted CCCM is calculated according to equation (14): C=a0·G+a1·L+a2·b Formula (14)

[0103] In some examples, regarding signal notification, if the CCLM mode is currently enabled for the CU, one flag is signaled to indicate whether the GLM is enabled for both the Cb and Cr components; if the GLM is enabled, another flag is signaled to indicate which of the two GLM modes is selected; and furthermore, one syntax element is signaled to select one of four gradient filters for gradient calculation.

[0104] Figure 8 shows diagrams (800) of four Sobel-based gradient filter patterns (labeled 1, 2, 3, and 4) that can be used in GLM.

[0105] In some examples, the use of cross-component prediction techniques is signaled in bitstream signaling with CABAC-coded PU-level flags. A new CABAC context may be included to support signaling. In some examples, for signaling, CCCM is considered a submode of CCLM. For example, the CCCM flag is signaled only when the intra-prediction mode is LM_CHOROMA.

[0106] In accordance with several aspects of this disclosure, multiple models or modes may be used to generate samples within the same block. For example, in the case of cross-component prediction, the second technique, MMLM, may be used to generate samples within the same block using two linear models. Note that multi-model techniques may be combined with other techniques. For example, multi-model techniques can be applied to CCLM, and CCLM with multiple models may be called MM-CCLM. In another example, multi-model techniques can be applied to CCCM, and CCCM with multiple models may be called MM-CCCM. In the following description, the term Multi-Model (or Multi-Mode) Cross-Component Prediction (MMCCP) is used to describe techniques that use multiple models (or multiple modes) to generate samples within the same block (e.g., the same CU), and MMCCP may include MMLM, MM-CCLM, MM-CCCM, etc. Note that techniques that use multiple models or multiple modes to generate samples within the same block may be applied to other suitable prediction techniques such as Local Illuminance Compensation (LIC). Also note that the multiple models can be linear or nonlinear models. In some examples, a model can be a linear combination of multiple terms, such as a weighted sum of multiple terms. For example, the multiple terms can be linear terms. In other examples, the multiple terms may include nonlinear terms.

[0107] In some examples, local illuminance compensation (LIC) is used as an interprediction technique to model local illuminance changes between a current block and a predicted block (also called a reference block) of the current block by using a linear function. The predicted block resides in a reference picture and may be indicated by a motion vector (MV). The parameters of the linear function may include a scale α and an offset β, and the linear function may be expressed as α × p[x,y] + β to compensate for the change in illuminance, where p[x,y] represents a reference sample at position [x,y] in the reference block (also called the predicted block), and the reference block is indicated by the MV. In some examples, the scale α and offset β may be derived based on the template of the current block and the corresponding reference template of the reference block by using the least squares method, so no signaling overhead is required, except that an LIC flag may be signaled to indicate the use of LIC. In some embodiments, multiple models may be used for LIC (called multi-model LIC). In some examples, samples may be classified into different classes, and different classes may use different models derived from different classes of template samples.

[0108] In accordance with aspects of this disclosure, when different linear models are used to generate cross-component predictions for samples within the same block, the cross-component predictions for adjacent samples using different linear models may not be smooth. Several aspects of this disclosure provide techniques for generating smooth predictions for samples within a block when different models are used for different samples within a block. For example, an encoder / decoder may decide to use at least two prediction models for different samples in the current block. The encoder / decoder may generate a predicted value for a first sample in the current block according to a first prediction model and generate a predicted value for a second sample in the current block according to a second prediction model different from the first prediction model. The encoder / decoder decides whether to apply a filter to the current sample in the current block based on whether one or more adjacent neighboring samples of the current sample use a different prediction model than the current sample. Then, in response to the decision to apply a filter, the encoder / decoder reconstructs the current sample based on the filtered output from a filter, with the predicted values ​​of the current sample and one or more adjacent neighboring samples as inputs to the filter.

[0109] Some aspects of this disclosure provide techniques for generating smooth cross-component predictions of samples within a block when different models are used to generate cross-component predictions of samples within a block. In the following description, multi-model CCP with smoothing (MMCCP) is referred to as S-MMCCP.

[0110] In accordance with aspects of this disclosure, S-MMCCP may be an adaptive process comprising a first and a second stage, which in some examples may be referred to as S-MMCCP. In the first stage, MMCCP is used as a first filter for cross-component prediction to generate samples within a block, and in the second stage, a second filter may be adaptively applied to the samples within the block to smooth the block. Whether or not to apply the (second) filter to a sample is determined based on adjacent neighboring samples during the generation of the final prediction of MMCCP for the sample (e.g., the second stage). For example, in the first stage of adaptive S-MMCCP, MMCCP is initially applied to each sample. In the second stage of adaptive S-MMCCP, whether or not the second filter is applied to each sample is determined by checking the use of a linear model of the sample and its adjacent neighboring samples, which are inputs to the second filter after the MMCCP process. In this disclosure, a neighboring sample of the current sample refers to a sample that is near the current sample and not necessarily directly adjacent to the current sample, while an adjacent neighboring sample of the current sample refers to a sample that is directly adjacent to the current sample.

[0111] Figure 9 shows the application of a second filter after MMCCP in several examples. In Figure 9, the current block (900) contains multiple samples represented by small squares. As an example, in the first stage of adaptive S-MMCCP, MMCCP is first applied to each sample in the current block (900). Then, in the second stage of adaptive S-MMCCP, whether a second filter should be applied to a sample is determined by checking the use of a linear model of the sample and adjacent neighboring samples, which are inputs to the second filter after the MMCCP process.

[0112] In the example in Figure 9, for the current sample indicated by “C”, the input to the second filter includes the current sample “C” and the four neighboring samples shown in Figure 9 as “T”, “B”, “L”, and “R”. Whether the second filter should be applied to the current sample can be determined based on the use of a linear model of the current sample “C” and the four neighboring samples “T”, “B”, “L”, and “R”. For example, if it is determined that the current sample “C” satisfies the smoothing requirements with neighboring samples (described further) after the MMCCP process, the second filter may not be necessary. However, if it is determined that the current sample “C” does not satisfy the smoothing requirements, the second filter is applied to the current sample in the second stage to generate a final prediction for the current sample.

[0113] For example, the second filter can generate a weighted average as the final prediction for the current sample based on the current sample and adjacent neighboring samples. For instance, the weights of samples "C", "T", "B", "L", and "R" could be 1 / 2, 1 / 8, 1 / 8, 1 / 8, and 1 / 8, respectively, in one example.

[0114] Although the plus sign pattern is used as the filter pattern for the second filter in Figure 9, other suitable filter patterns can be used for the second filter.

[0115] The smoothing requirement can be defined by various techniques. In some embodiments, the smoothing requirement is defined based on whether different linear models are used for the current sample and adjacent neighboring samples during the MMCCP process. In some embodiments, the second filter should be applied to the current sample if the current sample and at least one of its adjacent neighboring samples use different linear models during the MMCCP process. In other embodiments, the second filter should be applied to the current sample if the current sample and all of its adjacent neighboring samples use different linear models during the MMCCP process. In some examples, only neighboring samples located within the current block together with the current sample are used to determine whether the second filter should be applied.

[0116] In some embodiments, a flag is used to indicate whether adaptive S-MMCCP should be used. In some embodiments, the flag is signaled to indicate whether adaptive S-MMCCP should be used when MMCCP is used. In other embodiments, the flag is signaled to indicate whether adaptive S-MMCCP should be used, regardless of whether MMCCP-related syntax is signaled. In other embodiments, adaptive S-MMCCP is always applied when MMCCP is used, and a flag indicating whether adaptive S-MMCCP should be used is not required. In other embodiments, the flag indicating whether adaptive S-MMCCP should be used is context-coded using a context model that depends on whether neighboring coded blocks are coded with adaptive S-MMCCP.

[0117] In accordance with other aspects of this disclosure, whether a filter should be applied to the MMCCP is determined based on the ratio of the number of samples for different linear models. If the ratio is less than 1 / T or greater than T, the filter is not applied to the MMCCP, where T is a constant threshold. In some examples, the value of T is a predefined value. In some examples, the value of T or the index representing T can be a signal in high-level syntax.

[0118] In some examples, two models, such as a first model and a second model, are used in the MMCCP process of a block, with the first model having many more samples than the second model, for example, the ratio of the first model's samples to the second model's samples being greater than T (e.g., T is greater than 1), or the ratio of the second model's samples to the first model's samples being less than 1 / T, and the second filter is not used in the MMCCP (e.g., S-MMCCP is not used).

[0119] In some examples, the ratio of the number of samples in the first model (or second model) to the total number of samples in the block is calculated. For example, if the ratio is less than the first threshold, the second filter is not applied to MMCCP (e.g., S-MMCCP is not used). In other examples, if the ratio is greater than the second threshold, the second filter is not applied to MMCCP (e.g., S-MMCCP is not used). The first and / or second thresholds can be appropriately defined. For example, the first threshold is T and the second threshold is 1 / T.

[0120] Figure 10 shows a flowchart illustrating a process (1000) according to an embodiment of the present disclosure. Process (1000) may be used in a video decoder. In various embodiments, process (1000) is performed by processing circuits such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), and so on. In some embodiments, process (1000) is implemented by software instructions, so that when a processing circuit executes its software instructions, the processing circuit executes process (1000). The process starts at (S1001) and proceeds to (S1010).

[0121] In (S1010), a coded video bitstream is received containing coded information for the current block in the current picture. The coded information instructs the use of at least two predictive models for different samples in the current block.

[0122] In (S1020), the predicted value for the first sample in the current block is generated according to the first prediction model.

[0123] In (S1030), the predicted value for the second sample in the current block is generated according to a second prediction model that differs from the first prediction model.

[0124] In (S1040), it is determined whether a filter should be applied to the current sample within the current block, based on whether one or more adjacent neighboring samples are using a different predictive model than the current sample.

[0125] In (S1050), in response to the decision to apply the filter, the current sample is reconstructed based on the filtered output from the filter, with the predicted values ​​of the current sample and one or more adjacent neighboring samples as inputs to the filter.

[0126] In some examples, the coded information directs the use of multi-model cross-component prediction, which predicts the second color component based on the first color component. In this case, for a certain first sample in the first sample, the first predicted value of the second color component is generated based on the first reconfigured value of the first color component according to the first prediction model. For a certain second sample in the second sample, the second predicted value of the second color component is generated based on the second reconfigured value of the first color component according to the second prediction model.

[0127] In some examples, the coded information instructs the use of multiple local illuminance compensation (LIC) models. In this case, the first LIC model is applied to the first class of samples to produce the predicted value for the first sample, and the second LIC model is applied to the second class of samples to produce the predicted value for the second sample.

[0128] In some examples, each of the two predictive models is a linear model that defines a linear combination of multiple terms. For example, the multiple terms are linear terms. In other examples, the multiple terms include one or more nonlinear terms.

[0129] In some examples, the filter includes at least a first term based on the current sample and at least a second term based on adjacent neighboring samples.

[0130] In some cases, applying a filter to the current sample is decided when the adjacent neighboring samples of the current sample are using a different predictive model than the current sample.

[0131] In some examples, applying a filter to the current sample is determined when each adjacent neighboring sample of the current sample uses a different predictive model than the current sample. One or more adjacent neighboring samples of the current sample are currently within the same block.

[0132] In some examples, a flag indicating whether a filter should be applied in the current block is decoded from the coded video bitstream. For example, the flag is decoded in response to coded information indicating the use of at least two predictive models in the current block. In other examples, the flag is decoded regardless of whether syntax related to at least two predictive models is signaled. In other examples, a filter is always applied if at least two predictive models are used.

[0133] In some examples, whether a filter should be applied to the current block is determined based on the ratio of the sample sizes of the different prediction models. For example, the ratio is calculated as the number of samples in the first sample relative to the number of samples in the second sample. In other examples, the ratio is calculated as the number of samples in either the first or second prediction model relative to the total number of samples in the current block.

[0134] In some examples, whether a ratio satisfies a requirement is determined based on a comparison of the ratio with a threshold, and applying a filter in the current block is determined in response to the ratio satisfying the requirement. The ratio is defined as at least one of a constant value or the reciprocal of a constant value. For example, the constant value is predefined. In other examples, the constant value is determined from high-level syntax.

[0135] Then the process proceeds to (S1099) and terminates.

[0136] Process (1000) may be appropriately adapted. The steps of Process (1000) may be changed and / or omitted. Additional steps may be added. Any appropriate order of implementation may be used.

[0137] Figure 11 shows a flowchart illustrating a process (1100) according to an embodiment of the present disclosure. Process (1100) may be used in a video encoder. In various embodiments, process (1100) is performed by processing circuits such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), and so on. In some embodiments, process (1100) is performed by software instructions, so that when a processing circuit executes its software instructions, the processing circuit executes process (1100). The process begins at (S1101) and proceeds to (S1110).

[0138] In (S1110), it is decided to use at least two predictive models for different samples within the current block.

[0139] In (S1120), the predicted value for the first sample in the current block is generated according to the first prediction model.

[0140] In (S1130), the predicted value for the second sample in the current block is generated according to a second prediction model that differs from the first prediction model.

[0141] In (S1140), whether a filter should be applied to the current sample within the current block is determined based on whether one or more adjacent neighboring samples of the current sample are using a different predictive model than the current sample.

[0142] In (S1150), in response to the decision to apply the filter, the current sample is encoded based on the filtered output from the filter, with the predicted values ​​of the current sample and one or more adjacent neighboring samples as inputs to the filter.

[0143] In some examples, the use of multi-model cross-component prediction, which predicts the second color component based on the first color component, is decided upon. In this case, for a certain first sample within the first sample, the first predicted value of the second color component is generated based on the first reconstructed value of the first color component according to the first prediction model. For a certain second sample within the second sample, the second predicted value of the second color component is generated based on the second reconstructed value of the first color component according to the second prediction model.

[0144] In some cases, the use of multiple local illuminance compensation (LIC) models for a given block is determined. In such cases, the first LIC model is applied to the first class of samples to produce the predicted value for the first sample, and the second LIC model is applied to the second class of samples to produce the predicted value for the second sample.

[0145] In some examples, each of the two predictive models is a linear model that defines a linear combination of multiple terms. For example, the multiple terms are linear terms. In other examples, the multiple terms include one or more nonlinear terms.

[0146] In some examples, the filter includes at least a first term based on the current sample and at least a second term based on adjacent neighboring samples.

[0147] In some cases, applying a filter to the current sample is decided when the adjacent neighboring samples of the current sample are using a different predictive model than the current sample.

[0148] In some examples, applying a filter to the current sample is determined when each adjacent neighboring sample of the current sample uses a different predictive model than the current sample. One or more adjacent neighboring samples of the current sample are currently within the same block.

[0149] In some examples, a flag indicating whether a filter should be applied to the current block is encoded (signaled) in the coded video bitstream. For example, the flag is signaled if at least two predictive models are used in the prediction of the current block. In other examples, the flag is signaled regardless of whether the syntax related to at least two predictive models is signaled. In other examples, a filter is always applied if at least two predictive models are used in the prediction of the current block.

[0150] In some examples, whether a filter should be applied to the current block is determined based on the ratio of the sample sizes of the different prediction models. For example, the ratio is calculated as the number of samples in the first sample relative to the number of samples in the second sample. In other examples, the ratio is calculated as the number of samples in either the first or second prediction model relative to the total number of samples in the current block.

[0151] In some examples, whether a ratio satisfies a requirement is determined based on a comparison of the ratio with a threshold, and applying a filter in the current block is determined in response to the ratio satisfying the requirement. The ratio is defined as at least one of a constant value or the reciprocal of a constant value. For example, the constant value is predefined. In other examples, the constant value is signaled using high-level syntax.

[0152] Then the process proceeds to (S1199) and terminates.

[0153] Process (1100) may be appropriately adapted. The steps of Process (1100) may be changed and / or omitted. Additional steps may be added. Any appropriate order of implementation may be used.

[0154] The above technology can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, Figure 12 shows a computer system (1200) suitable for implementing a particular embodiment of the subject of disclosure.

[0155] Computer software can be coded in any suitable machine code or computer language that follows mechanisms such as assembly, compilation, and linking, to generate code that includes instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more central processing units (CPUs), graphics processing units (GPUs), etc.

[0156] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and Internet of Things devices.

[0157] The components shown in Figure 12 with respect to the computer system (1200) are illustrative in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of the components should not be construed as having any dependence or requirement on any one or combination of components described in the exemplary embodiments of the computer system (1200).

[0158] The computer system (1200) may include certain human interface input devices. Such human interface input devices may respond to input from one or more users, for example, through haptic input (e.g., keyboard, swipe, dataglobe motion), voice input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images taken from a still camera), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0159] The input human interface device may include one or more of the following: keyboard (1201), mouse (1202), trackpad (1203), touchscreen (1210), data glove (not shown), joystick (1205), microphone (1206), scanner (1207), and camera (1208) (only one of each is shown).

[0160] The computer system (1200) may also include certain human interface output devices that can stimulate the senses of one or more users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1210), data glove (not shown), or joystick (1205), although haptic feedback devices that do not function as input devices may also exist), audio output devices (e.g., speakers (1209), headphones (not shown)), visual output devices (e.g., CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality and each with or without haptic feedback functionality, some of which are screens (1210) capable of outputting two-dimensional visual output or output in more than three dimensions by means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0161] The computer system (1200) may also include human-accessible storage devices and their associated media, such as CD / DVD ROM / RW (1220) including CD / DVD or similar media (1221), thumb drives (1222), removable hard disks or solid-state drives (1223), legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices, such as security dongles (not shown), and the like.

[0162] Those skilled in the art will understand that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include transmission media, carrier waves, or other transient signals.

[0163] The computer system (1200) may also include an interface (1254) to one or more communication networks (1255). The networks may be, for example, wireless, wireline, or optical. The networks may further be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet®, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV, and vehicle and factory networks including CAN bus. Certain networks generally require an external network interface adapter attached to a specific general-purpose digital port or peripheral bus (1249) (e.g., a USB port on the computer system (1200)). Others are generally integrated into the core of the computer system (1200) by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1200) can communicate with other entities. Such communication can be unidirectional and receivable only (e.g., broadcast TV) or unidirectional and receivable only (e.g., a CAN bus to a specific CAN bus device), or it can be bidirectional to other computer systems using, for example, a local or wide-area digital network. Specific protocols or protocol stacks are available for use with each of the networks and network interfaces described above.

[0164] The above-mentioned human interface devices, human-accessible memory devices, and network interfaces may be attached to the core (1240) of the computer system (1200).

[0165] The core (1240) may include one or more central processing units (CPUs) (1241), graphics processing units (GPUs) (1242), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (1243), hardware accelerators for specific tasks (1244), graphics adapters (1250), etc. These devices may be connected via a system bus (1248) along with read-only memory (ROM) (1245), random access memory (RAM) (1246), internal mass storage devices such as internal user-inaccessible hard drives, SSDs, etc. (1247). In some computer systems, the system bus (1248) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus (1248) or via a peripheral bus (1249). In the example, the display (1210) may be connected to the graphics adapter (1250). Architectures for peripheral buses include PCI and USB.

[0166] The CPU (1241), GPU (1242), FPGA (1243), and accelerator (1244) are capable of executing specific instructions that can be combined to constitute the computer code described above. This computer code can be stored in ROM (1245) or RAM (1246). Temporary data can also be stored in RAM (1246), while persistent data can be stored, for example, in a built-in mass storage device (1247). High-speed storage and retrieval to any of the memory devices can be made possible by the use of cache memory. Cache memory may be closely associated with one or more CPUs (1241), GPUs (1242), mass storage devices (1247), ROMs (1245), RAM (1246), etc.

[0167] Computer-readable media may contain computer code for performing various computer implementation operations. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the art in computer software technology.

[0168] For example, and not as an limitation, a computer system having architecture (1200), specifically a core (1240), can provide functionality as a result of a processor (including CPUs, GPUs, FPGAs, accelerators, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media can be media related to user-accessible mass storage devices described earlier, in addition to specific storage devices of the core (1240) that have a non-transient nature, such as core-integrated mass storage (1247) or ROM (1245). Software implementing various embodiments of the present disclosure is stored in such devices and is executable by the core (1240). The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software may cause the core (1240) and, specifically, the processor within it (including CPU, GPU, FPGA, etc.) to execute a particular process or a particular part of a particular process as described herein, including defining data structures stored in RAM (1246) and modifying such data structures according to a process defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of logic (e.g., an accelerator (1244)) that is hardwired or otherwise embodied in the circuitry and can operate in place of or with the software to execute a particular process or a particular part of a particular process as described herein. References to software may, as necessary, include logic, and vice versa. References to computer-readable media may, as necessary, include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry embodying logic for execution, or both. This disclosure also encompasses any suitable combination of hardware and software.

[0169] The use of “at least one of…” or “one of…” in this disclosure is intended to include any one or combination of the elements described. For example, references to at least one of A, B, or C, at least one of A, B, and C, at least one of A, B, and / or C, and at least one of A through C are intended to include A only, B only, C only, or any combination thereof. References to one of A or B, and one of A and B are intended to include A or B or (A and B). The use of “one of…” does not exclude any combination of the elements described where applicable, such as when the elements are not mutually exclusive.

[0170] While this disclosure has described several exemplary embodiments, alternatives, substitutions, and various substitute equivalents exist and are included within the scope of this disclosure. Therefore, it will be apparent to those skilled in the art that numerous systems and methods embodying the principles of this disclosure, and thus falling within its spirit and scope, can be conceived, even if not explicitly illustrated or described herein.

[0171] [References] This application claims priority to U.S. Patent Application No. 18 / 406323, filed on January 8, 2024, for the same invention

Claims

1. A method of video decoding performed by a decoder, The step of receiving a coded video bitstream containing coded information of the current block in the current picture, wherein the coded information instructs the use of at least two predictive models for different samples in the current block, A step of generating a predicted value for the first sample in the current block according to the first prediction model, A step of generating a predicted value for the second sample in the current block according to a second prediction model different from the first prediction model, A step of determining whether a filter should be applied to the current sample based on whether one or more adjacent neighboring samples use a different predictive model than the current sample in the current block, In response to the decision to apply the filter, the current sample is reconstructed based on the filtered output from the filter, where the predicted values ​​of the current sample and one or more adjacent neighboring samples become the inputs to the filter. A method of having.

2. The coded information instructs the use of multi-model cross-component prediction, which predicts the second color component based on the first color component, and the method For a given sample among the first samples, the steps include generating a first predicted value of the second color component based on a first reconfigured value of the first color component according to the first prediction model, For a given second sample among the second samples, the steps include generating a second predicted value of the second color component based on a second reconfigured value of the first color component according to the second prediction model, and The method according to claim 1, comprising:

3. The coded information instructs the use of multiple local illumination compensation (LIC) models, and the method The steps include applying a first LIC model to a first class of samples to generate the predicted values ​​of the first sample, The steps include applying a second LIC model to a second class of samples to generate the predicted values ​​for the second sample, and The method according to claim 1, comprising:

4. Each of the two prediction models is a linear model that defines a linear combination of multiple terms. The method according to claim 1.

5. The filter includes at least a first term of the current sample and at least a second term of adjacent neighboring samples. The method according to claim 1.

6. The step of determining whether the aforementioned filter should be applied is: The step further includes deciding to apply the filter if adjacent neighboring samples to the current sample are using a different predictive model than the current sample. The method according to claim 1.

7. The step of determining whether the aforementioned filter should be applied is: The step further includes deciding to apply the filter if each adjacent neighboring sample of the current sample uses a different predictive model than the current sample. The method according to claim 1.

8. The one or more adjacent neighboring samples of the current sample are located within the current block. The method according to claim 1.

9. The method according to claim 1, further comprising the step of decoding a flag indicating whether the filter should be applied in the current block in response that the coded information indicates the use of at least two predictive models in the current block.

10. The method according to claim 1, further comprising the step of determining whether the filter should be applied to the current block based on the ratio of the sample sizes of different prediction models.

11. The steps include: calculating the ratio of the first sample to the second sample number in the second sample, The steps include: calculating the ratio as the number of samples in one of the first prediction model and the second prediction model with respect to the total number of samples in the current block; The method according to claim 10, further comprising at least one of the following.

12. A step of determining whether the ratio satisfies the requirements based on a comparison of the ratio with a threshold, The steps include: determining to apply the filter in the current block in response that the ratio satisfies the requirements; The method according to claim 10, further comprising:

13. The ratio is defined as at least one of a constant value or the reciprocal of the constant value. The method according to claim 12.

14. The aforementioned constant value is predefined. The method according to claim 13.

15. The aforementioned constant value is determined from the high-level syntax. The method according to claim 13.

16. A video decoding device, The device comprises a memory for storing a program and a processing circuit, wherein the processing circuit is configured to execute the program stored in the memory and perform the method according to any one of claims 1 to 15. Device.

17. A program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 15.

18. A video encoding method performed by an encoder, The steps include deciding to use at least two predictive models for different samples within the current block currently contained in the picture, A step of generating a predicted value for the first sample in the current block according to the first prediction model, A step of generating a predicted value for the second sample in the current block according to a second prediction model different from the first prediction model, The step of determining whether a filter should be applied to the current sample based on whether one or more neighboring samples adjacent to the current sample in the current block use a different predictive model than the current sample, In response to the decision to apply the filter, the current sample is encoded based on the filtered output from the filter, where the predicted values ​​of the current sample and one or more adjacent neighboring samples are inputs to the filter. A method of having.

Images