Video decoding method, apparatus, and medium

Abstract

A method, an apparatus, and a medium for video decoding are provided. One apparatus includes processing circuitry that decodes prediction information for a current block in a current picture, wherein the current picture is part of a coded video sequence. The prediction information indicates a cross-component filtering (CCF) process for the current block. The processing circuitry generates filtered reconstructed samples for the current block by applying the CCF process on at least one of predicted samples, residual values, or reconstructed samples for the current block. The filtered reconstructed samples for the current block are used for reconstruction of a subsequent block. The processing circuitry reconstructs the current block and the subsequent block based on the filtered reconstructed samples for the current block.

Description

[0001] Incorporation by Reference

[0002] This application claims priority to U.S. Patent Application No. 17 / 315,989, “Method and Apparatus for Video Coding,” filed on May 10, 2021, which claims priority to U.S. Provisional Patent Application No. 63 / 079,322, “Cross-Component Filtering for Block-Level Reconstruction,” filed on September 16, 2020. The entire contents of the priority applications are incorporated herein by reference. TECHNICAL FIELD

[0003] This disclosure describes embodiments related to video coding. BACKGROUND

[0004] The background description provided herein is intended to present the context of the present application. As such, work of the inventors in this regard, to the extent it is currently known, is considered to be part of the state of the art. The background description is provided for the purpose of generally presenting the context of the application. The background description is not intended to be, and should not be construed as, admitting prior art nor should it be considered to be prior art to the present application.

[0005] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having a spatial dimension of, for example, 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (informally also known as frame rate), of, for example 60 pictures per second or 60 Hz. Uncompressed video has a very high data rate. For example, 1080p60 4:2:0 video at 8 bit per sample and 60 Hz frame rate requires close to 1.5 Gbit / s bandwidth for compression. One hour of such video requires more than 600 GBytes of storage.

[0006] One purpose of video coding and decoding is to compress the video so that the resulting bitstream uses a lower bitrate for a given perceptual quality, or uses the same bitrate to achieve a better perceptual quality. In either case, compression is helpful in reducing the cost of storing and transmitting video data. Compression can be lossy, where some data is lost, or lossless, where no data is lost. In general, video compression results in some loss of data.

[0007] Video encoders and decoders can utilize several major categories of techniques, including motion compensation, transform, quantization, and entropy coding.

[0008] Video codec techniques may include known intra-frame coding techniques. In intra-frame coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference image. In some video codecs, an image is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the image can be an intra-frame image. Intra-frame images and their derivatives (e.g., independent decoder refresh images) can be used to reset the decoder state and are therefore used as the first image in the encoded video bitstream and video session, or as still images. Samples from an intra-frame block can be used for transform, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block for a given quantization step size.

[0009] As is known from technologies such as MPEG-2, traditional intra-frame coding does not use intra-frame prediction. However, some newer video compression techniques include those that attempt to derive data blocks from, for example, surrounding sample data and / or metadata, which are obtained during spatially adjacent encoding and / or decoding, and prior to the decoding sequence. This technique is later referred to as "intra-frame prediction." It is important to note that, at least in some cases, intra-frame prediction uses only reference data from the current frame being reconstructed, and not reference data from a reference frame.

[0010] There can be many different forms of intra-prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be coded in intra-prediction modes. In some cases, a mode may have sub-modes and / or parameters, and these modes may be encoded individually or contained in mode codewords. Which codeword is used for a given mode, sub-mode, and / or parameter combination will affect the coding efficiency gain through intra-prediction, and this also applies to entropy coding techniques used to convert codewords into bitstreams.

[0011] H.264 introduced an intra-frame prediction mode, which was improved in H.265 and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Prediction blocks are formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of neighboring samples are copied into the prediction block in a specific direction. References to the direction used can be encoded in the bitstream or can be predicted themselves.

[0012] Referring to Figure 1A, the lower right corner depicts a subset of nine known prediction directions from the 33 possible prediction directions of H.265 (corresponding to 33 angular modes of 35 intra-frame modes). The point (101) where the arrows converge represents the sample being predicted. The arrow indicates the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples at a 45-degree angle to the horizontal direction in the upper right. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples at a 22.5-degree angle to the horizontal direction in the lower left.

[0013] Referring again to Figure 1, a square block (104) comprising 4×4 samples is shown in the upper left (represented by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from top to bottom) and the first sample in the X dimension (from left to right). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block is 4×4 in size, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. Reference samples are labeled with "R" and their Y position (e.g., row index) and X position (e.g., column index) relative to block (104). In H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, so negative values ​​are not required.

[0014] Intra-frame image prediction can be performed by copying reference sample values ​​from adjacent samples occupied by the prediction direction indicated by the signal. For example, suppose the encoded video bitstream includes signaling that, for this block, the signaling indicates a prediction direction consistent with arrow (102), i.e., predicting samples based on one or more prediction samples at a 45-degree angle to the horizontal direction from the upper right. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Then, sample S44 is predicted based on reference sample R08.

[0015] In some cases, such as through interpolation, the values ​​of multiple reference samples can be combined to compute a reference sample, especially when the direction is not divisible by 45 degrees.

[0016] With the development of video coding technology, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013) and JEM / VVC / BMS, and at the time of this application, up to 65 directions could be supported. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding have been used to represent those possible directions using a small number of bits, while accepting some cost for less likely directions. Furthermore, the direction itself can sometimes be predicted based on the adjacent directions used in adjacent, already decoded blocks.

[0017] Figure 1B shows a schematic diagram (105) depicting 65 intra-frame prediction directions according to JEM to illustrate the number of prediction directions increasing over time.

[0018] The mapping of intra-predicted direction bits in a coded video bitstream can vary depending on the video coding technique, and can range from a simple, direct mapping of intra-predicted modes to predicted directions in codewords to complex adaptive schemes that incorporate the most probable modes and similar techniques. However, in all cases, there may be certain directions in the video content that are statistically less likely to occur than others. Since the purpose of video compression is to reduce redundancy, in well-functioning video coding techniques, less probable directions will be represented using a greater number of bits compared to the more probable directions.

[0019] Motion compensation can be a lossy compression technique and may involve using sample data blocks from a previously reconstructed image or a portion of a reconstructed image (reference image) spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) for prediction of a newly reconstructed image or image portion. In some cases, the reference image may be the same as the image currently being reconstructed. MV may have two dimensions, X and Y, or three dimensions, where the third dimension represents the reference image in use (the latter may indirectly be a temporal dimension).

[0020] In some video compression techniques, the motion vector (MV) applied to a region of sample data can be predicted based on other MVs, such as those MVs associated with another region of sample data adjacent to the region being reconstructed and preceding that MV in the decoding order. This significantly reduces the amount of data required to encode the MV, thereby eliminating redundant information and increasing compression. MV prediction can be performed efficiently, for example, when encoding an input video signal derived from a camera (called natural video), there is a statistical probability that regions larger than the area applicable to a single MV will move in a similar direction. Therefore, in some cases, predictions can be made using similar motion vectors derived from MVs in neighboring regions. This results in MVs found for a given region being similar or identical to MVs predicted from surrounding MVs, and after entropy encoding, can be represented with fewer bits than when directly encoding the MV. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors that arise when calculating predictions based on several surrounding MVs.

[0021] H.265 / HEVC (ITU-T Rec. H.265, “Efficient Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this paper describes a technique referred to below as “spatial combining”.

[0022] Referring to Figure 1C, the current block (111) may include samples discovered by the encoder during motion search, which can be predicted based on previous blocks that have moved by the same size in space. Instead of directly encoding this MV, it is derived from metadata associated with one or more reference images, such as from the most recent (in decoding order) reference image, using the MV associated with any one of the five surrounding samples. The five surrounding samples are denoted as A0, A1 and B0, B1, B2 (112 to 116 respectively). In H.265, MV prediction can use predictions from the same reference image being used by adjacent blocks. Summary of the Invention

[0023] This disclosure provides an apparatus for video encoding / decoding. An apparatus includes processing circuitry that decodes prediction information for a current block in a current image, wherein the current image is part of an encoded video sequence. The prediction information indicates a cross-component filtering (CCF) process for the current block. The processing circuitry generates filtered reconstruction samples for the current block by applying the CCF process to at least one of prediction samples, residual values, or reconstructed samples of the current block. The filtered reconstruction samples of the current block are used for the reconstruction of subsequent blocks. The processing circuitry reconstructs the current block and the subsequent blocks based on the filtered reconstruction samples of the current block.

[0024] In one embodiment, the processing circuit generates filtered reconstructed samples of the current block by applying the CCF process to the output of at least one of the dequantization and inverse transform processes of the current block.

[0025] In one embodiment, the processing circuit reconstructs the current block by adding the output of the CCF process, which includes the offset value of the current block, to the chromaticity reconstruction sample of the current block.

[0026] In one embodiment, the processing circuit reconstructs the current block by adding the output of the CCF process, which includes the offset value of the current block, to the chromaticity residual value of the current block.

[0027] In one embodiment, the processing circuit reconstructs the current block by adding the output of the CCF process, which includes the offset value of the current block, to the chromaticity prediction sample of the current block.

[0028] In one embodiment, the prediction information includes the filter coefficients of the CCF process.

[0029] In one embodiment, the filter coefficients of the CCF process are predefined constants.

[0030] In one embodiment, the filter coefficients of the CCF process used in the current image are determined based on the filter coefficients of the CCF process used in another image and the offset values ​​of the filter coefficients of the CCF process used in the current image.

[0031] In one embodiment, the filter coefficients of the CCF process used in one color component are determined based on the filter coefficients of the CCF process used in another color component.

[0032] In one embodiment, the filter coefficients of the CCF process are determined based on the sample values ​​of the current block determined before the CCF process is applied.

[0033] In one embodiment, it is determined whether the CCF process is enabled based on one of the partitioning scheme, prediction mode, block width, block height, transform coefficients, and quantization parameters.

[0034] This disclosure provides a method for video encoding / decoding. In this method, prediction information for a current block in a current image is decoded, wherein the current image is part of an encoded video sequence. The prediction information indicates a cross-component filtering (CCF) process for the current block. A filtered reconstruction sample for the current block is generated by applying the CCF process to at least one of the prediction sample, residual value, or reconstructed sample of the current block. The filtered reconstruction sample of the current block is used for the reconstruction of subsequent blocks. Based on the filtered reconstruction sample of the current block, the current block and the subsequent blocks are reconstructed.

[0035] This disclosure also provides a non-volatile computer-readable medium storing instructions, wherein when the instructions are executed by at least one processor, the at least one processor causes the at least one processor to perform any one or a combination of video decoding methods. Attached Figure Description

[0036] Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, wherein:

[0037] Figure 1A is a schematic diagram illustrating an exemplary subset of intra-frame prediction modes.

[0038] Figure 1B is an illustration of an exemplary intra-frame prediction direction.

[0039] Figure 1C is a schematic diagram of the current block and its surrounding space merge candidates in an example.

[0040] Figure 2 This is a simplified block diagram of a communication system according to an embodiment.

[0041] Figure 3 This is a simplified block diagram of a communication system according to an embodiment.

[0042] Figure 4 This is a simplified block diagram of the decoder according to an embodiment.

[0043] Figure 5 This is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0044] Figure 6 A block diagram of an encoder according to another embodiment is shown.

[0045] Figure 7 A block diagram of a decoder according to another embodiment is shown.

[0046] Figures 8A-8B An exemplary diamond filter shape of an adaptive loop filter (ALF) according to an embodiment is shown.

[0047] Figures 9A-9D Exemplary secondary sampling positions for gradient calculation in the vertical, horizontal, and two diagonal directions, according to an embodiment, are shown.

[0048] Figure 10 An exemplary modified block classification applied to the luminance component according to an embodiment is shown.

[0049] Figure 11 An exemplary modified ALF for the luminance component at virtual boundaries is shown according to some embodiments.

[0050] Figure 12 An exemplary image quadtree segmentation aligned with the Largest Coding Unit (LCU) according to an embodiment is shown.

[0051] Figure 13 An exemplary quadtree partitioning flag encoded in z-order according to an embodiment is shown.

[0052] Figure 14A An exemplary placement of the Cross-Component Adaptive Loop Filter (CC-ALF) according to an embodiment is shown.

[0053] Figure 14B An exemplary linear diamond filter is shown, according to an embodiment, applied to the luminance channel of each chromaticity component during CC-ALF operation.

[0054] Figure 15 An exemplary direction search for an 8×8 block is shown according to an embodiment.

[0055] Figure 16 An exemplary subspace projection according to an embodiment is shown.

[0056] Figure 17 An exemplary cross-component filtering (CCF) is shown, applied after the inverse transform of the block.

[0057] Figure 18 An exemplary flowchart according to an embodiment of this disclosure is shown.

[0058] Figure 19 This is a schematic diagram of a computer system according to one embodiment. Detailed Implementation

[0059] I. Video Decoder and Encoder Systems

[0060] Figure 2 This is a simplified block diagram of a communication system (200) according to an embodiment of the present disclosure. The communication system (200) includes a plurality of terminal devices that can communicate with each other via, for example, a network (250). For example, the communication system (200) includes a first pair of terminal devices (210) and terminal device (220) interconnected via a network (250). Figure 2 In the example, the first pair of terminal devices (210) and (220) perform one-way data transmission. For example, terminal device (210) may encode video data (e.g., a video image stream captured by terminal device (210)) for transmission over a network (250) to another terminal device (220). The encoded video data is transmitted as one or more encoded video streams. Terminal device (220) may receive the encoded video data from the network (250), decode the encoded video data to recover the video data, and display video images based on the recovered video data. One-way data transmission is common in applications such as media services.

[0061] In another embodiment, the communication system (200) includes a second pair of terminal devices (230) and (240) performing bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bidirectional data transmission, in one example, each of the terminal devices (230) and (240) may encode video data (e.g., a stream of video images captured by the terminal device) for transmission over a network (250) to the other terminal device (230) and (240). Each of the terminal devices (230) and (240) may also receive encoded video data transmitted by the other terminal device (230) and (240), decode the encoded video data to recover the video data, and display the video images on an accessible display device based on the recovered video data.

[0062] exist Figure 2In the embodiments disclosed herein, terminal devices (210), (220), (230), and (240) may be servers, personal computers, and smartphones, but the principles disclosed herein are not limited thereto. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (250) refers to any number of networks that transmit encoded video data between terminal devices (210), (220), (230), and (240), including, for example, wired (connected) and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. This network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of the network (250) may be irrelevant to the operation of this application.

[0063] As an example, Figure 3 The diagram illustrates the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0064] The streaming system may include an acquisition subsystem (313) that may include a video source (301) such as a digital camera, which creates an uncompressed video image stream (302). In an embodiment, the video image stream (302) includes samples captured by a digital camera. The video image stream (302) is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data (304) (or encoded video bitstream). The video image stream (302) may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (302), the encoded video data (304) (or the encoded video bitstream (304)) is depicted as a thin line to emphasize the lower data volume of the encoded video data (304) (or the encoded video bitstream (304)), which can be stored on a streaming server (305) for future use. One or more streaming client subsystems, such as Figure 3Client subsystems (306) and (308) can access a streaming server (305) to retrieve copies (307) and (309) of encoded video data (304). Client subsystem (306) may include, for example, a video decoder (310) in an electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and produces an output video picture stream (311) that can be displayed on a display (312) (e.g., a screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (304), video data (307), and video data (309) (e.g., video streams) may be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T H.265. In embodiments, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and this application can be used in the context of the VVC standard.

[0065] It should be noted that the electronic devices (320) and (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0066] Figure 4 This is a block diagram of a video decoder (410) according to an embodiment disclosed in this application. The video decoder (410) may be disposed in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., receiving circuitry). The video decoder (410) may be used in place of... Figure 3 The video decoder (310) in the embodiment.

[0067] The receiver (431) may receive one or more encoded video sequences to be decoded by the video decoder (410); in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (431) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (431) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter referred to as "parser (420)"). In some applications, the buffer memory (415) is part of the video decoder (410). In other cases, the buffer memory (415) may be located external to the video decoder (410) (not indicated). In other cases, an external buffer (not shown) may be provided for the video decoder (410) to prevent network jitter, for example, and another buffer (415) may be configured internally for, for example, handling broadcast timing. When the receiver (431) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer (415) may not be necessary, or it may be made smaller. Of course, for use on packet networks such as the Internet, a buffer (415) may be required; this buffer may be relatively large and adaptive in size, and may be at least partially implemented in the operating system or a similar component (not shown) external to the video decoder (410).

[0068] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (410) and potential information for controlling a display device (412) (e.g., a display screen), which is not part of the electronic device (430) but may be coupled to it, such as... Figure 4As shown in the figure. The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (420) may parse / decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels in the encoded video sequence for use in the video decoder based on at least one parameter corresponding to a group. The subgroup may include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (420) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0069] The parser (420) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to create symbols (421).

[0070] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (421) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed by the parser (420) from the encoded video sequence. For brevity, the flow of such subgroup control information between the parser (420) and the various units described below is not described.

[0071] In addition to the functional blocks already mentioned, the video decoder (410) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0072] The first unit is the scaler / inverse transform unit (451). The scaler / inverse transform unit (451) receives quantization transform coefficients as symbols (421) and control information from the parser (420), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (451) can output a block containing sample values, which can be input into the aggregator (455).

[0073] In some cases, the output samples of the scaler / inverse transform unit (451) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) uses reconstructed information extracted from the current picture buffer (458) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (458) buffers partially reconstructed and / or fully reconstructed current images. In some cases, the aggregator (455) adds the predictive information generated by the intra-picture prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451) based on each sample.

[0074] In other cases, the output samples of the scaler / inverse transform unit (451) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (453) can access the reference image memory (457) to extract samples for prediction. After motion compensation of the extracted samples according to the symbols (421), these samples can be added by the aggregator (455) to the output of the scaler / inverse transform unit (451) (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit (453) can obtain the prediction samples from the address in the reference image memory (457) under motion vector control, and the motion vector is available to the motion compensation prediction unit (453) in the form of the symbols (421), which, for example, include X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (457) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0075] The output samples of the aggregator (455) can be employed by various loop filtering techniques in the loop filter unit (456). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (421) from the parser (420) in the loop filter unit (456). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0076] The output of the loop filter unit (456) can be a sample stream, which can be output to a display device (412) and stored in a reference image memory (457) for subsequent inter-frame image prediction.

[0077] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (420)) is identified as the reference image, the current image buffer (458) can become part of the reference image memory (457), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0078] The video decoder (410) can perform decoding operations according to a predetermined video compression technique, such as that specified in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0079] In this embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0080] Figure 5 This is a block diagram of a video encoder (503) according to an embodiment disclosed in this application. The video encoder (503) is disposed in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) can be used to replace... Figure 3 The video encoder (303) in the embodiment.

[0081] The video encoder (503) can obtain data from the video source (501) (not) Figure 5 In one embodiment, a portion of the electronic device (520) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (503). In another embodiment, the video source (501) is a portion of the electronic device (520).

[0082] A video source (501) can provide a sequence of source video samples encoded by a video encoder (503) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb4:2:0, YCrCb4:4:4). In a media service system, the video source (501) can be a storage device storing previously prepared video. In a video conferencing system, the video source (501) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0083] According to an embodiment, the video encoder (503) can encode and compress images of a source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (550). In some embodiments, the controller (550) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (550) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, reference area allowed for maximum motion vector, etc. The controller (550) may be used with other suitable functions related to the video encoder (503) optimized for a particular system design.

[0084] In some embodiments, the video encoder (503) operates within an encoding loop. As a simplified description, in an embodiment, the encoding loop may include a source encoder (530) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (533) embedded within the video encoder (503). The decoder (533) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (534). Since decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image memory (534) are also bit-precisely corresponding between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is also used in some related technologies.

[0085] The operation of the “local” decoder (533) can be combined with, for example, the above-described method. Figure 4 The video decoder (410) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 4 When symbols are available and the entropy encoder (545) and parser (420) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (410), including the buffer (415) and parser (420), may not be fully implemented in the local decoder (533).

[0086] It can be observed that any decoder technique other than parsing / entropy decoding present in the decoder must also exist in the corresponding encoder in essentially the same functional form. For this reason, this application focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0087] During operation, in some embodiments, the source encoder (530) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input image, referencing one or more previously encoded images from the video sequence designated as "reference images." In this manner, the encoding engine (532) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

[0088] The local video decoder (533) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (530). The operation of the encoding engine (532) can be a lossy process. When the encoded video data can be decoded by the video decoder (533), Figure 5 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (533) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image memory (534). In this way, the video encoder (503) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0089] The predictor (535) can perform a prediction search against the encoding engine (532). That is, for a new image to be encoded, the predictor (535) can search in the reference image memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (535) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (535), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (534).

[0090] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

[0091] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (545). The entropy encoder (545) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0092] The transmitter (540) can buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission via a communication channel (560), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (540) can combine the encoded video data from the video encoder (503) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0093] The controller (550) manages the operation of the video encoder (503). During encoding, the controller (550) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0094] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand variations of I-pictures and their corresponding applications and characteristics.

[0095] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0096] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0097] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0098] The video encoder (503) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (503) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0099] In this embodiment, the transmitter (540) may transmit additional data while transmitting encoded video. The source encoder (530) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0100] The acquired video can serve as multiple source images (video images) presented in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0101] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0102] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0103] According to some embodiments disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video image sequence are segmented into coding tree units (CTUs) for compression. The CTUs in the images have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Furthermore, each CTU can be further subdivided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be subdivided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In embodiments, each CU is analyzed to determine the prediction type used for the CU, such as inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. Taking a luma prediction block as an example, a prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0104] Figure 6 This is a diagram of a video encoder (603) according to another embodiment disclosed in this application. The video encoder (603) is used to receive processing blocks (e.g., prediction blocks) of sample values ​​within a current video image in a video image sequence, and to encode the processing blocks into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (603) is used instead of Figure 3The video encoder (303) in the embodiment.

[0105] In the HEVC embodiment, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as an 8×8 sample prediction block. The video encoder (603) uses, for example, rate-distortion (RD) optimization to determine whether to use an intra-frame mode, an inter-frame mode, or a bidirectional prediction mode to encode the processing block. When encoding the processing block in intra-frame mode, the video encoder (603) can use intra-frame prediction techniques to encode the processing block into an already encoded picture; and when encoding the processing block in inter-frame mode or bidirectional prediction mode, the video encoder (603) can use inter-frame prediction or bidirectional prediction techniques to encode the processing block into an already encoded picture, respectively. In some video coding techniques, the merging mode can be an inter-frame picture prediction sub-mode, in which motion vectors are derived from one or more motion vector prediction values ​​without relying on already encoded motion vector components outside the prediction values. In some other video coding techniques, motion vector components applicable to the subject block may exist. In the embodiment, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0106] exist Figure 6 In one embodiment, the video encoder (603) includes, as shown below: Figure 6 The inter-frame encoder (630), intra-frame encoder (622), residual calculator (623), switch (626), residual encoder (624), general controller (621) and entropy encoder (625) are shown coupled together.

[0107] An inter-frame encoder (630) is configured to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and later images), generate inter-frame prediction information (e.g., redundancy information description, motion vectors, merging mode information based on inter-frame coding techniques), and calculate inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference image is a decoded reference image based on encoded video information.

[0108] The intra encoder (622) is used to receive samples of the current block (e.g., the processing block), in some cases compare the block with previously encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases also (e.g., based on intra prediction direction information of one or more intra coding techniques) generate intra prediction information. In an embodiment, the intra encoder (622) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

[0109] A general-purpose controller (621) determines general-purpose control data and controls other components of the video encoder (603) based on the general-purpose control data. In an embodiment, the general-purpose controller (621) determines the mode of a block and provides control signals to a switch (626) based on the mode. For example, when the mode is an intra-frame mode, the general-purpose controller (621) controls the switch (626) to select an intra-frame mode result for use by the residual calculator (623) and controls the entropy encoder (625) to select intra-frame prediction information and add the intra-frame prediction information to the bitstream; and when the mode is an inter-frame mode, the general-purpose controller (621) controls the switch (626) to select an inter-frame prediction result for use by the residual calculator (623) and controls the entropy encoder (625) to select inter-frame prediction information and add the inter-frame prediction information to the bitstream.

[0110] A residual calculator (623) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (622) or the inter encoder (630). A residual encoder (624) is used to operate on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (624) is used to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (622) and the inter encoder (630). For example, the inter encoder (630) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (622) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and in some embodiments, the decoded image may be buffered in a memory circuit (not shown) and used as a reference image.

[0111] An entropy encoder (625) is used to format the bitstream to produce encoded blocks. The entropy encoder (625) generates various information according to a suitable standard such as HEVC. In an embodiment, the entropy encoder (625) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. It should be noted that, according to the disclosed subject matter, residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0112] Figure 7This is a diagram of a video decoder (710) according to another embodiment disclosed in this application. The video decoder (710) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (710) is used instead of Figure 3 The video decoder (310) in the embodiment.

[0113] exist Figure 7 In this embodiment, the video decoder (710) includes, as follows: Figure 7 The entropy decoder (771), inter-frame decoder (780), residual decoder (773), reconstruction module (774), and intra-frame decoder (772) are shown coupled together.

[0114] An entropy decoder (771) can be used to reconstruct certain symbols from an encoded image, these symbols representing the syntax elements constituting the encoded image. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can respectively identify certain samples or metadata used by the intra-frame decoder (772) or the inter-frame decoder (780) for prediction, residual information in the form of, for example, quantized transform coefficients, and so on. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (780); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (772). Residual information may be provided to the residual decoder (773) via inverse quantization.

[0115] The inter-frame decoder (780) is used to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0116] The intra-frame decoder (772) is used to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0117] The residual decoder (773) performs inverse quantization to extract the dequantized transform coefficients and processes the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require some control information (to obtain the quantizer parameter (QP)), and this information can be provided by the entropy decoder (771) (the data path is not indicated because this is only low-level control information).

[0118] The reconstruction module (774) is used to combine the residual output by the residual decoder (773) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which may be a part of a reconstructed image, which in turn may be a part of a reconstructed video. It should be noted that other suitable operations, such as deblocking, may be performed to improve visual quality.

[0119] It should be noted that any suitable technology can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). In one embodiment, one or more integrated circuits can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710).

[0120] II. Adaptive Loop Filter

[0121] In some relevant examples, such as VVC Draft 7, an adaptive loop filter (ALF) with block-based filter adaptation can be applied. For the luminance component, one of 25 filters can be selected for each 4×4 block based on the directionality and activity of each 4×4 block.

[0122] Figures 8A-8B An exemplary diamond filter shape for an ALF according to some embodiments of the present disclosure is shown. Figure 8A A 5×5 rhombus is shown that can be used for chromaticity components, and, Figure 8B A 7×7 rhombus is shown that can be used for the luminance component.

[0123] In some relevant examples, for the luminance component, each 4×4 block can be classified into one of 25 categories. This can be based on the directionality D and activity of the 4×4 block. The quantized value is used to derive the 4×4 block classification index C as follows.

[0124]

[0125] To calculate D and We can first use the 1-D Laplacian operator to calculate the gradients of the 4×4 block in the horizontal, vertical, and two diagonal directions as follows.

[0126]

[0127]

[0128]

[0129]

[0130] Here, indices i and j refer to the coordinates of the top-left sample within the 4×4 block, and R(i,j) represents the reconstructed sample at coordinates (i,j).

[0131] To reduce the complexity of block classification, a one-dimensional Laplace calculation with quadratic sampling can be applied. Figures 9A-9D Exemplary secondary sampling positions are shown for gradient calculations in the vertical, horizontal, and two diagonal directions, respectively.

[0132] Then, the maximum and minimum values ​​of the gradients in the horizontal and vertical directions can be set as follows:

[0133]

[0134]

[0135] The maximum and minimum values ​​of the gradients in the two diagonal directions can be set as follows:

[0136]

[0137]

[0138] To derive the values ​​of the directionality D, these values ​​can be compared with each other and with two thresholds t1 and t2.

[0139] Step 1: If and If all values ​​are true, set D to 0.

[0140] Step 2: If Continue from step 3; otherwise, continue from step 4.

[0141] Step 3: If Set D to 2; otherwise, set D to 1.

[0142] Step 4: If Set D to 4; otherwise, set D to 3.

[0143] Activity value A is calculated as follows:

[0144]

[0145] A is further quantized to a range of 0 to 4 (inclusive), and the quantized value is represented as...

[0146] No classification method was applied to the chromaticity components in the image; that is, a set of ALF coefficients could be applied to each chromaticity component.

[0147] Before filtering each 4×4 luminance block, geometric transformations such as rotation, diagonal flip, and vertical flip are applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l) based on the gradient values ​​calculated for the corresponding block. This is equivalent to applying these transformations to multiple samples in the filter support region to make the different blocks to which ALF is applied more similar by aligning their orientations.

[0148] Three geometric transformations, including diagonal flip, vertical flip, and rotation, can be described as follows:

[0149] Diagonal:f D (k,l)=f(l,k),c D Equation (k,l)=c(l,k). (11)

[0150] Vertical flip:f V (k,l)=f(k,Kl-1),c V Equation (k,l)=c(k,Kl-1). (12)

[0151] Rotation:f R (k,l)=f(Kl-1,k),c R Equation (k,l)=c(Gl-1,k). (13)

[0152] Where K is the size of the filter, and 0 ≤ k, l ≤ K⁻¹ are the coordinates of the transform coefficients; therefore, position (0, 0) is located in the upper left corner, and position (K⁻¹, K⁻¹) is located in the lower right corner. The transform is applied to the filter coefficients f(k, l) and the clipping values ​​c(k, l) based on the gradient values ​​calculated for the corresponding blocks. The relationship between the transform and the gradient in the four directions can be summarized in Table 1.

[0153] Table 1

[0154]

[0155]

[0156] In some relevant examples, such as those in VVC Draft 7, the filter parameters of the ALF are signaled in the Adaptation Parameter Set (APS). Within an APS, up to 25 sets of luma filter coefficients and clipping value indices, and up to 8 sets of chroma filter coefficients and clipping value indices can be signaled. To reduce bit overhead, filter coefficients from different categories of luma components can be merged. In the slice header, the index of the APS used for the current slice is signaled. ALF signaling in VVC Draft 7 is based on Coding Tree Units (CTUs).

[0157] The clipping value index decoded from APS allows the clipping values ​​to be determined using a clipping value table for the luma and chroma components. These clipping values ​​depend on the internal bit depth. For example, the clipping value table can be obtained using the following formula:

[0158] AlfClip = {round(2 B-α*n Equation (14) for 4∈[0..N-1]}

[0159] Where B equals the internal bit depth, α is a predefined constant value equal to 2.35, and N equals 4, which is the number of clipping values ​​allowed in VVC Draft 7. Table 2 is an example of the output of equation (14).

[0160] Table 2

[0161]

[0162] In the slice header, up to seven APS indices can be signaled to specify the luma filter group for the current slice. The filtering process can be further controlled at the Coding Tree Block (CTB) level. A flag can be signaled to indicate whether an ALF (Alternating Current Filter) is applied to the luma CTB. This luma CTB can select a filter group from 16 fixed filter groups and filter groups from the APS. A filter group index is signaled for this luma CTB to indicate which filter group is applied. These 16 fixed filter groups can be predefined and hard-coded in the encoder and decoder.

[0163] For a chroma component, an APS index can be signaled in the slice header to indicate the chroma filter group to be used for the current slice. At the CTB level, if there are multiple chroma filter groups in the APS, a filter index can be signaled for each chroma CTB.

[0164] The filter coefficients can be quantized using a norm equal to 128. To limit the multiplication complexity, bitstream consistency can be applied so that coefficient values ​​outside the center range from -27 to 27-1 (inclusive). The coefficients at the center position are not signaled in the bitstream and are assumed to be equal to 128.

[0165] In some relevant examples, such as VVC Draft 7, the syntax and semantics of clipping indices and their corresponding values ​​can be defined as follows.

[0166] `alf_luma_clip_idx[sfIdx][j]` specifies the clipping index of the clipping value to be used before multiplying by the j-th coefficient of the signaled luminance filter represented by `sfIdx`. `sfIdx` = 0..alf_luma_num_filters_signalled_minus1, and `j` = 0..11. The value of `alf_luma_clip_idx[sfIdx][j]` should range from 0 to 3 (inclusive) for bitstream consistency requirements.

[0167] Based on the bit depth (bitDepth) set to be equal to BitDepthY and the clipping index (clipIdx) set to be equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j], the luminance filter clipping value AlfClipL[adaptation_parameter_set_id][filtIdx][j] with element AlfClipL[adaptation_parameter_set_id][filtIdx][j], filtIdx = 0..NumAlfFilters-1 and j = 0..11 is derived from Table 2.

[0168] `alf_chroma_clip_idx[altIdx][j]` specifies the clipping index to be used before multiplying the j-th coefficient of the candidate chroma filter with index `altIdx`. The value of `alf_chroma_clip_idx[altIdx][j]` with `altIdx = 0..alf_chroma_num_alt_filters_minus1,j = 0..5` should be in the range of 0 to 3 (inclusive) for bitstream consistency requirements.

[0169] Based on the bit depth (bitDepth) set to equal BitDepthC and the clipping index (clipIdx) set to equal alf_chroma_clip_idx[altIdx][j], the chroma filter clipping values ​​of AlfClipC[adaptation_parameter_set_id][altIdx][j] with altIdx = 0..alf_chroma_num_alt_filters_minus1,j = 0..5 are derived from Table 2.

[0170] On the decoder side, when ALF is enabled for CTB, each sample R(i,j) in the coding unit (CU) is filtered to obtain the corresponding sample value R′(i,j), as shown below.

[0171]

[0172] Where f(k,l) represents the decoded filter coefficients, K(x,y) is a clipping function, and c(k,l) represents the decoded clipping parameters. The variables k and l... and The values ​​vary between L and L, where L represents a filter length. The clipping function K(x,y) = min(y,max(-y,x)) corresponds to a function Clip3(-y,y,x). By combining these clipping functions, this cyclic filtering method becomes a nonlinear process, called Non-Linear ALF. The selected clipping values ​​are encoded in the "alf_data" syntax element using the Golomb coding scheme corresponding to the indices of the clipping values ​​in Table 2. This coding scheme is the same as the coding scheme for the filter indices.

[0173] Figure 10 An exemplary modified block classification applied to the luminance component according to an embodiment of this disclosure is shown. By employing the modified block classification and filtering for samples near the horizontal CTU boundary, the line buffer requirements of the ALF can be reduced. Figure 10 As shown, the virtual boundary is defined as a line by shifting the horizontal CTU boundary with “N” samples, where N equals 4 for the luminance component and N equals 2 for the chrominance component.

[0174] For the 1D Laplacian gradient computation of the 4×4 block above the virtual boundary, only samples above the virtual boundary are used. Similarly, for the 1D Laplacian gradient computation of the 4×4 block below the virtual boundary, only samples below the virtual boundary are used. By taking into account the reduced number of samples used in the 1D Laplacian gradient computation, the quantization of the active value A is scaled accordingly.

[0175] Figure 11 An exemplary modified ALF is shown at a virtual boundary according to some embodiments. For the filtering process, a symmetrical padding operation at the virtual boundary can be applied to both the luminance and chrominance components. Figure 11 As shown, when the filtered sample is below the virtual boundary, the adjacent sample above the virtual boundary is filled. At the same time, the corresponding sample on the other side is symmetrically filled.

[0176] To enhance encoding and decoding efficiency, some related examples use ALP based on image quadtrees synchronized by coding units. A luminance image can be divided into several multi-level quadtree partitions, with each partition's boundary aligned with the boundary of the largest coding unit (LCU). Each partition has its own filtering process and can therefore be called a filter unit (FU).

[0177] The 2-channel coded stream is described below. In the first channel, the quadtree segmentation pattern and optimal filter for each function element (FU) are determined. During this determination process, filtering distortion is estimated using Fast Filtering Distortion Estimation (FFDE). The reconstructed image is filtered based on the determined quadtree segmentation pattern and the filters for all selected FUs. In the second channel, CU-synchronized ALF on / off control is performed. Based on the ALF on / off results, the filtered image from the first channel is partially recovered from the reconstructed image.

[0178] Figure 12 An exemplary LCU-aligned image quadtree segmentation according to an embodiment is illustrated. A top-down segmentation strategy is employed, dividing the image into multi-level quadtree partitions using a rate-distortion criterion. Each partition is referred to as a filter unit. The segmentation process aligns the quadtree partitions with the LCU boundaries. The encoding order of the FUs follows the z-scan order. For example, as... Figure 12 As shown, the image is divided into 10 FUs, and the encoding order is FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8 and FU9.

[0179] Figure 13 It shows the corresponding Figure 12An exemplary quadtree segmentation pattern. To indicate the quadtree segmentation pattern of an image, segmentation flags can be encoded and sent in z-order.

[0180] Based on the rate-distortion criterion, filters for each Functional Unit (FU) can be selected from two filter banks. The first bank contains newly derived 1 / 2 symmetric square and diamond filters for the current FU. The second bank comes from time-delayed filter buffers, which store filters derived for previous FUs of the previous image. The filter with the minimum rate-distortion cost from these two banks can be selected for the current FU. Similarly, if the current FU is not the minimum FU and can be further segmented into 4 sub-FUs, the rate-distortion cost of the 4 sub-FUs is calculated. By recursively comparing the rate-distortion costs of the segmented and unsegmented cases, the quadtree segmentation pattern of the image can be determined.

[0181] In some relevant examples, the maximum quadtree split level is 2, meaning the maximum number of functional units (FUs) is 16. During quadtree split determination, the correlation values ​​used to derive the Wiener coefficients of the 16 FUs at the bottom quadtree level (minimum FUs) can be reused. The remaining FUs can have their Wiener filters derived from the correlations of the 16 FUs at the bottom quadtree level. Therefore, only one framebuffer access is needed to derive the filter coefficients for all FUs.

[0182] After determining the quadtree partitioning pattern, to further reduce filtering distortion, CU-synchronized ALF on / off control can be performed. By comparing filtered distortion and non-filtered distortion, leaf CUs can explicitly switch ALF on / off in their local regions. The filter coefficients can be redesigned based on the ALF on / off result, thereby further improving encoding / decoding efficiency. However, the redesign process requires additional frame buffer access. In some relevant examples, after the CU-synchronized ALF on / off is determined, there is no redesign process to minimize the number of frame buffer accesses.

[0183] III. Cross-component adaptive loop filter

[0184] In some relevant examples, a cross-component adaptive loop filter (CC-ALF) is used. CC-ALF uses luminance sample values ​​to correct each chrominance component.

[0185] Figure 14A An exemplary placement of the CC-ALF according to an embodiment of this disclosure is shown. Figure 14B An exemplary linear diamond filter is shown applied to the luminance channel for each chromaticity component during CC-ALF operation. The filter coefficients can be sent in the APS, for example, in 2...10 The scaling factor is applied, and rounding is performed for the fixed-point representation. Filter application is controlled by a variable block size and is signaled by a context-encoded flag received for each block across multiple samples. For each chroma component, the block size and CC-ALF enable flag are received at the slice-level. In the example, the following block sizes are supported (in chroma samples): 16×16, 32×32, and 64×64.

[0186] Table 3 shows the syntax elements associated with CC-ALF.

[0187] Table 3

[0188]

[0189] In Table 3, the syntax element `alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]` equal to 0 indicates that the cross-component Cb filter is not applied to the sample block of the Cb color component located at the luminance position (xCtb, yCtb). The syntax element `alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]` not equal to 0 indicates that the first cross-component Cb filter is applied to the sample block of the Cb color component located at the luminance position (xCtb, yCtb).

[0190] The syntax element `alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]` equal to 0 indicates that the cross-component Cr filter is not applied to the sample block of the Cr color component at the luminance position (xCtb, yCtb). The syntax element `alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]` not equal to 0 indicates that the first cross-component Cr filter is applied to the sample block of the Cr color component at the luminance position (xCtb, yCtb).

[0191] IV. Constrained Directional Enhancement Filter

[0192] One goal of the in-loop constrained directional enhancement filter (CDEF) is to filter out coding artifacts while preserving image details. In some related examples of HEVC, such as the Samplead adaptive offset (SAO) algorithm, a similar goal can be achieved by defining signal offsets for different categories of pixels. Unlike SAO, CDEF is a nonlinear spatial filter. The filter design is constrained to be easily vectorized; for example, it can be implemented using Single Instruction Multiple Data (SIMD) operations, which is not the case for other nonlinear filters such as median filters and bilateral filters.

[0193] The CDEF design is based on the following observations: The number of ringing artifacts in an encoded image tends to be roughly proportional to the quantization step size. The amount of detail is a property of the input image, but even the smallest details preserved in the quantized image tend to be proportional to the quantization step size. For a given quantization step size, the amplitude of ringing is typically smaller than the amplitude of detail.

[0194] CDEF works by identifying the orientation of each block and then adaptively filtering along that orientation, with a smaller degree of filtering along the orientation rotated 45 degrees from the identified orientation. The filter strength is explicitly signaled, allowing for a high degree of control over blurring. An efficient encoder search is designed for the filter strength. CDEF is based on two previously proposed in-loop filters, and the emerging AOMedia Video1 (AV1) codec employs a combined filter.

[0195] Figure 15An exemplary direction search for an 8×8 block according to an embodiment of this disclosure is illustrated. The direction search operates on the reconstructed pixels after the deblocking filter. Since these pixels are available for the decoder, no signaling is required for the direction. The search operates on an 8×8 block, small enough to adequately handle non-linear edges, yet large enough to reliably estimate the direction when applied to a quantized image. Having a constant direction over an 8×8 region also facilitates the vectorization of the filter. For each block, the direction that best matches the pattern in the corresponding block is determined by minimizing the sum of squared differences (SSD) between the quantized block and the closest perfectly directional block. A perfectly directional block is a block in which all pixels along a line in one direction have the same value.

[0196] One reason for identifying the direction is to align the filter taps along that direction to reduce ringing while preserving the directional edges or pattern. However, directional filtering alone is sometimes insufficient to adequately reduce ringing. It is also desirable to use filter taps on pixels that are not along the main direction. To reduce the risk of blurring, these additional taps are handled more conservatively. For this purpose, CDEF defines primary taps and secondary taps. The complete two-dimensional (2-D) CDEF filter is represented as follows:

[0197]

[0198] Where D is the damping parameter, S (p) and S (s) These represent the strengths of the main tap and the secondary tap, respectively, and round(·) rounds to the nearest integer (ties away from zero). and Here, S is the filter weight, and f(d,S,D) is the constraint function that operates on the difference between the filtered pixel and each of its neighboring pixels. For small differences, f(d,S,D) = d, making the filter behave like a linear filter. When the difference is large, f(d,S,D) = 0, which effectively ignores the filter taps.

[0199] Loop recovery in IV and AV1

[0200] In addition to traditional deblocking operations, a set of in-loop recovery schemes can be used in post-encoded video deblocking to generally denoise and improve edge quality. These schemes can be switched within a frame for each appropriately sized tile. The specific schemes described are based on separable symmetric Wiener filters and dual self-guided filters with subspace projection. Because content statistics can vary significantly within a frame, these tools can be integrated within a switchable frame, where different tools can be triggered in different regions of the frame.

[0201] For a Wiener filter, each pixel in a degraded frame can be reconstructed as a non-causal filtered version of the corresponding pixel within a w×w window surrounding that pixel, where w = 2r + 1 is odd for integers r. If the 2D filter taps are in column vectorized form w... 2 The ×1 element vector F represents the filter parameters. Direct linear minimum mean square error (LMMSE) optimization leads to the filter parameters being derived from F = H. -1 M is given, where H = E[XX] T [ is the autocovariance of 1, w in a w×w window around the pixel] 2 The column vectorized version of the sample, and M = E[YX] T ] is the cross-correlation between x and the scalar source sample y to be estimated. The encoder can estimate H and M based on the deblocked frame and the source implementation, and send the resulting filter F to the decoder. However, this not only affects the transmission of w 2 Tapping incurs a significant bit rate cost and results in non-separable filtering, making decoding extremely complex. Therefore, several additional constraints are imposed on the properties of F. First, F is constrained to be separable, allowing filtering to be implemented as separable horizontal and vertical w-tap convolutions. Second, each of the horizontal and vertical filters is constrained to be symmetric. Third, it is assumed that the sum of the horizontal and vertical filter coefficients is 1.

[0202] The local linear model of the guided filter can be represented as follows:

[0203] Equation y = Fx + G. (17)

[0204] A local linear model is used to compute the filtered output y from an unfiltered sample x, where F and G are determined statistically based on the degraded image and the guidance image near the filtered pixels. If the guidance image is identical to the degraded image, the resulting so-called self-guided filtering has the effect of preserving smooth edges. The specific form of the self-guided filtering depends on two parameters: the radius r and the noise parameter e, as listed below.

[0205] (1) Obtain the mean μ and variance σ of the pixels in the (2r+1)×(2r+1) window surrounding each pixel. 2 This can be efficiently achieved using box filtering based on integrated imaging.

[0206] (2) Calculate for each pixel: f = σ 2 / (σ 2 +e); g=(1-f)μ.

[0207] (3) Calculate F and G for each pixel as the average of the f and g values ​​in a 3×3 window around the pixel for use.

[0208] The filtering is controlled by r and e, where a higher r means a higher spatial variance and a higher e means a higher range variance.

[0209] Figure 16 An exemplary subspace projection according to an embodiment is shown. Even if no inexpensive restorations X1 and X2 are close to the source Y, appropriate multipliers {α,β} can bring them closer to the source as long as they are moving in the right direction.

[0210] VI. Cross-Component Filtering for Block-Level Reconstruction

[0211] In some related examples such as VVC, a cross-component filtering (CCF) process has been proposed and adopted as an additional filtering process after SAO. That is, the CCF process is applied outside the transform, quantization, dequantization, and reconstruction loops. Therefore, the CCF process applied to the current block does not improve the prediction of subsequent coded blocks because the reconstructed samples of the current block, used as reference samples for subsequent blocks, are not affected by the CCF process. In such examples, the CCF process is applied at the image level. To enable the CCF process to affect the prediction of subsequent coded blocks, in embodiments of this disclosure, the CCF process can be applied at the block level.

[0212] This disclosure includes a method for applying the CCF process to block-level reconstruction. The CCF process can be defined as a filtering process that uses reconstructed samples of a first color component as input (e.g., Y, Cb, or Cr), and the output of this filtering process can be applied to a second color component different from the first color component. An example of a CCF process is the CC-ALF process described in Section III.

[0213] According to several aspects of this disclosure, the CCF process can be applied before SAO is completed. For example, the CCF process can be applied after dequantization and / or inverse transform. The filtered reconstructed samples of the current block output by the CCF process can be used as reference samples for subsequent blocks, and / or for generating prediction samples for subsequent blocks.

[0214] Figure 17 An exemplary CCF applied after the inverse transform of a block according to an embodiment of this disclosure is illustrated. The filtered samples can be used for intra-frame prediction and / or inter-frame prediction of subsequent blocks, and in-loop filtering can be applied on the filtered samples output from the CCF process.

[0215] In one embodiment, CCF can be applied only to the chromaticity color component. The input to the CCF process can be a reconstructed luminance sample, and the output can be an offset value added to the chromaticity sample of the current block.

[0216] In one embodiment, the CCF process can be applied to the residuals. The input to the CCF process can be the luminance residual value derived from the inverse transform, and the output can be an offset value added to the chrominance residual value of the current block.

[0217] In one embodiment, the CCF process can be applied to the predicted sample values. The input to the CCF process can be the predicted luminance sample values, and the output can be an offset value added to the predicted chrominance sample values ​​of the current block.

[0218] In one embodiment, the filter coefficients used in the CCF process can be signaled. For example, the filter coefficients can be signaled in the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), slice header, or tile header.

[0219] In one embodiment, the filter coefficients used in the CCF process are predefined constants.

[0220] In one embodiment, the filter coefficients used in the CCF process for the current image are derived from the filter coefficients used in another CCF process. For example, the filter coefficients used in the CCF process for the current image can be derived from the filter coefficients used in the CCF processes for different images and the offset value for signaling the current image.

[0221] In one embodiment, the filter coefficients used in the CCF process for one color component (e.g., chromaticity component) are derived from the filter coefficients used in the CCF process for another color component (e.g., luminance component).

[0222] In one embodiment, filter coefficients are selected for each sample based on some statistics, wherein the statistics are derived from sample values ​​determined prior to the application of the CCF procedure.

[0223] In some embodiments, it is determined whether the CCF process is enabled. The CCF process is applied based on the determination that the CCF process is enabled.

[0224] In one embodiment, the CCF process can be enabled or disabled for certain block sizes. In one example, the CCF process is disabled for blocks whose width and / or height are less than a given threshold.

[0225] In one embodiment, the CCF process can be enabled or disabled for certain partitioning schemes. In one example, the CCF process is disabled when the luma and chroma components have different block sizes, for example, when the luma and chroma components have different partitioning schemes or semi-decoupled partitioning schemes.

[0226] In one embodiment, the CCF process can be enabled or disabled for certain prediction modes. In one example, the CCF process is disabled for Direct Current (DC) mode. In one example, the CCF process is disabled for Planar mode. In one example, the CCF process is disabled for one or more of the SMOOTH, SMOOTH_H, and SMOOTH_V modes. In one example, the CCF process is disabled for Paeth predictor mode. In one example, the CCF process is disabled for SKIP mode. In one example, the CCF process is enabled for directional mode. In one example, the CCF process is enabled for inter-intra compound mode.

[0227] In one embodiment, the filter shape of the CCF process may depend on the width and / or height of the block.

[0228] In one embodiment, whether the CCF process is enabled depends on the transform coefficients of the inverse transform process. In one example, if all transform coefficients (before or after dequantization) are zero, the CCF process is not applied. In another example, if only the DC transform coefficients (before or after dequantization) are non-zero, the CCF process is not applied. In yet another example, if only the low-frequency transform coefficients (before or after dequantization) are non-zero, the CCF process is not applied. Low-frequency transform coefficients are transform coefficients located at coordinates (x, y), where x and / or y are less than a given threshold.

[0229] In one embodiment, whether the CCF process is enabled depends on the quantization parameters of the dequantization process.

[0230] VII. Flowchart

[0231] Figure 18A flowchart outlining an exemplary process (1800) according to an embodiment of the present disclosure is shown. In various embodiments, the process (1800) is executed by processing circuitry, such as processing circuitry in terminal devices (210), (220), (230), and (240), processing circuitry performing the functions of a video encoder (303), a video decoder (310), a video decoder (410), an intra-frame prediction module (452), a video encoder (503), a predictor (535), an intra-frame encoder (622), an intra-frame decoder (772), etc. In some embodiments, the process (1800) is implemented as software instructions, so that the processing circuitry executes the process (1800) when the software instructions are executed.

[0232] The process (1800) can typically begin at step (S1810), where the process (1800) decodes the prediction information for the current block in the current picture, which is part of an encoded video sequence. The prediction information indicates the CCF process for the current block. The process (1800) then proceeds to step (S1820).

[0233] In step (S1820), process (1800) generates a filtered reconstruction sample for the current block by applying the CCF process to at least one of the predicted sample, residual value, or reconstructed sample of the current block. The filtered reconstruction sample of the current block is used for the reconstruction of subsequent blocks. Then, process (1800) proceeds to step (S1830).

[0234] In step (S1830), process (1800) reconstructs the current block and subsequent blocks based on the filtered reconstruction samples of the current block. Then, process (1800) terminates.

[0235] In one embodiment, process (1800) generates filtered reconstructed samples of the current block by applying the CCF process to the output of at least one of the dequantization process and the inverse transform process of the current block.

[0236] In one embodiment, process (1800) reconstructs the current block by adding the output of a CCF process, which includes the offset value of the current block, to the chromaticity reconstruction sample of the current block.

[0237] In one embodiment, process (1800) reconstructs the current block by adding the output of a CCF process that includes the offset value of the current block to the chromaticity residual value of the current block.

[0238] In one embodiment, process (1800) reconstructs the current block by adding the output of a CCF process that includes the offset value of the current block to the chromaticity prediction sample of the current block.

[0239] In one embodiment, the prediction information includes the filter coefficients of the CCF process.

[0240] In one embodiment, the filter coefficients of the CCF process are predefined constants.

[0241] In one embodiment, the filter coefficients of the CCF process used in the current image are determined based on the filter coefficients of the CCF process used in another image and the offset values ​​of the filter coefficients of the CCF process used in the current image. The offset values ​​may be included in the prediction information.

[0242] In one embodiment, the filter coefficients of the CCF process used in one color component are determined based on the filter coefficients of the CCF process used in another color component.

[0243] In one embodiment, the filter coefficients of the CCF process are determined based on the sample values ​​of the current block determined before the CCF process is applied.

[0244] In one embodiment, the decision to enable the CCF process is determined based on one of the partitioning scheme, prediction mode, block width, block height, transform coefficients, and quantization parameters.

[0245] VIII. Computer Systems

[0246] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 19 A computer system (1900) is shown, which is adapted to implement certain embodiments of the disclosed subject matter.

[0247] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through decoding, microcode, etc.

[0248] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0249] Figure 19The components shown for the computer system (1900) are exemplary in nature and are not intended to limit the scope or functionality of the computer software used to implement the embodiments of this application. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of the computer system (1900).

[0250] The computer system (1900) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-computer interface device may also be used to capture certain media, which need not be directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0251] Human-machine interface input devices may include one or more of the following (only one is shown): keyboard (1901), mouse (1902), touchpad (1903), touch screen (1910), data glove (not shown), joystick (1905), microphone (1906), scanner (1907), camera (1908).

[0252] The computer system (1900) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (1910), data gloves (not shown), or joystick (1905), but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (1909), headphones (not shown)), visual output devices (e.g., screens including cathode ray tube screens, liquid crystal screens, plasma screens, organic light-emitting diode screens (1910), each with or without touchscreen input, each with or without tactile feedback—some of which may output two-dimensional or more three-dimensional visual outputs by means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown). These visual output devices (e.g., screens (1910)) may be connected to the system bus (1948) via a graphics adapter (1950).

[0253] Computer systems (1900) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) (1920) or similar media (1921), thumb drives (1922), removable hard disk drives or solid-state drives (1923), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), and so on.

[0254] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0255] The computer system (1900) may also include a network interface (1954) leading to one or more communication networks (1955). For example, the one or more communication networks (1955) may be wireless, wired, or optical. The one or more communication networks (1955) may also be local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), vehicular and industrial networks, real-time networks, delay-tolerant networks, and so on. Examples of the one or more communication networks (1955) include Ethernet, wireless LANs, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicular and industrial networks (including CANbus), and so on. Some networks typically require external network interface adapters for connection to certain general-purpose data ports or peripheral buses (1949) (e.g., the USB port of the computer system (1900)); other systems are typically integrated into the core of the computer system (1900) via a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer system (1900) can communicate with other entities. This communication can be unidirectional, used only for receiving (e.g., wireless television), unidirectional, used only for sending (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as via a local area or wide area digital network to other computer systems. Each of the aforementioned networks and network interfaces can use certain protocols and protocol stacks.

[0256] The aforementioned human-computer interface devices, human-accessible storage devices, and network interfaces can be connected to the core (1940) of the computer system (1900).

[0257] The core (1940) may include one or more central processing units (CPU) (1941), graphics processing units (GPUs) (1942), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (1943), task-specific hardware accelerators (1944), graphics adapters (1950), etc. These devices, along with read-only memory (ROM) (1945), random access memory (1946), and internal mass storage (e.g., internal non-user-accessible hard disk drives, solid-state drives, etc.) (1947), can be connected via a system bus (1948). In some computer systems, the system bus (1948) can be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (1948) or connected via a peripheral bus (1949). In one example, a screen (1910) may be connected to a graphics adapter (1950). Peripheral bus architectures include external controller interfaces (PCI), universal serial buses (USB), etc.

[0258] The CPU (1941), GPU (1942), FPGA (1943), and accelerator (1944) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (1945) or RAM (1946). Transitional data can also be stored in RAM (1946), while permanent data can be stored, for example, in internal mass storage (1947). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1941), GPUs (1942), mass storage (1947), ROM (1945), RAM (1946), etc.

[0259] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this application, or they may be media and code well-known and usable by those skilled in the art of computer software.

[0260] By way of example and not limitation, a computer system having an architecture (1900), particularly a core (1940), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the non-volatile core (1940), such as internal mass storage (1947) or ROM (1945). Software implementing various embodiments of this application can be stored in such a device and executed by the core (1940). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (1940), particularly the processor therein (including a CPU, GPU, FPGA, etc.), to execute a specific process or a specific portion of a specific process described herein, including defining data structures stored in RAM (1946) and modifying such data structures according to software-defined processes. Alternatively or as an alternative, a computer system may provide logic hardwired or otherwise incorporated into circuitry (e.g., an accelerator (1944)) that may replace or operate with the software to perform the specific process or a specific portion of the specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing the software (such as an integrated circuit (IC)), circuitry containing the logic for execution, or both. This application includes any suitable combination of hardware and software.

[0261] While this application has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are all within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are thus within its spirit and scope.

[0262] Appendix A: Acronyms

[0263] ALF: Adaptive Loop Filter

[0264] AMVP: Advanced Motion Vector Prediction

[0265] APS: Adaptation Parameter Set

[0266] ASIC: Application-Specific Integrated Circuit

[0267] ATMVP: Alternative / Advanced Temporal Motion Vector Prediction

[0268] AV1: Open Media Alliance Video 1

[0269] AV2: Open Media Alliance Video 2

[0270] BMS: Baseline

[0271] BV: Block Vector

[0272] CANBus: Controller Area Network Bus

[0273] CB: Encoded Block

[0274] CC-ALF: Transcomponent Adaptive Loop Filter

[0275] CD: CD-ROM

[0276] CDEF: Constrained Direction Enhancement Filter

[0277] CPR: Current image reference

[0278] CPUs: Central Processing Unit

[0279] CRT: Cathode Ray Tube

[0280] CTBs: Coded Tree Blocks

[0281] CTUs: Coding Tree Units

[0282] CU: Encoding Unit

[0283] DPB: Decoder Image Buffer

[0284] DPS: Decoding Parameter Set

[0285] DVD: Digital Video Disc

[0286] FPGA: Field Programmable Gate

[0287] JCCR: Joint CbCr Residual Encoding

[0288] JVET: Joint Video Exploration Team

[0289] GOPs: Image Group

[0290] GPUs: Graphics Processing Units

[0291] GSM: Global System for Mobile Communications

[0292] HDR: High Dynamic Range

[0293] HEVC: High-efficiency video encoding and decoding

[0294] HRD: Hypothetical Reference Decoder

[0295] IBC: Intra-Block Copy

[0296] IC: Integrated Circuit

[0297] ISP: Intra-Frame Sub-Partition

[0298] JEM: Joint Exploration Model

[0299] LAN: Local Area Network

[0300] LCD: Liquid Crystal Display

[0301] LR: Loop Recovery Filter

[0302] LTE: Long Term Evolution

[0303] MPM: Most Probable Pattern

[0304] MV: Motion Vector

[0305] OLED: Organic Light Emitting Diode

[0306] PBs: Predicted Blocks

[0307] PCI: Peripheral Component Interconnect

[0308] PDPC: Location-related prediction combination

[0309] PLD: Programmable Logic Device

[0310] PPS: Image Parameter Set

[0311] PUs: Prediction Units

[0312] RAM: Random Access Memory

[0313] ROM: Read-Only Memory

[0314] SAO: Sample Adaptive Offset

[0315] SCC: Screen Content Encoding

[0316] SDR: Standard Dynamic Range

[0317] SEI: Supplemental Enhancement Information

[0318] SNR: Signal-to-noise ratio

[0319] SPS: Sequence Parameter Set

[0320] SSD: Solid State Drive

[0321] TUs: Transformation Unit

[0322] USB: Universal Serial Bus

[0323] VPS: Video Parameter Set

[0324] VUI: Video Availability Information

[0325] VVC: Multi-functional Video Codec

[0326] WAIP: Wide-angle Intra-frame Prediction

Claims

1. A video decoding method, characterized in that, include: Decode the prediction information of the current block in the current image, wherein the current image is part of an encoded video sequence, and the prediction information indicates the cross-component filtering (CCF) process for the current block. Whether the CCF process is enabled is determined based on one of the partitioning scheme, prediction mode, transform coefficients, and quantization parameters. When the luminance component and chrominance component have different partitioning schemes or semi-decoupled partitioning schemes, the CCF process is disabled. When a directional mode or internal composite mode is used, the CCF process is enabled. When the transform coefficients before or after dequantization are all zero, the CCF process is disabled. By applying the CCF process to at least one of the predicted samples, residual values, or reconstructed samples of the current block, filtered reconstructed samples of the current block are generated, which are then used for the reconstruction of subsequent blocks; and, Based on the filtered reconstruction samples of the current block, reconstruct the current block and the subsequent blocks.

2. The method according to claim 1, characterized in that, The process of generating the filtered reconstruction samples for the current block includes: The filtered reconstruction sample of the current block is generated by applying the CCF process to the output of at least one of the dequantization and inverse transform processes of the current block.

3. The method according to claim 1, characterized in that, The reconstruction of the current block and the subsequent blocks includes: The current block is reconstructed by adding the output of the CCF process, which includes the offset value of the current block, to the chromaticity reconstruction sample of the current block.

4. The method according to claim 1, characterized in that, The reconstruction of the current block and the subsequent blocks includes: The current block is reconstructed by adding the output of the CCF process, which includes the offset value of the current block, to the chromaticity residual value of the current block.

5. The method according to claim 1, characterized in that, The reconstruction of the current block and the subsequent blocks includes: The current block is reconstructed by adding the output of the CCF process, which includes the offset value of the current block, to the chromaticity prediction sample of the current block.

6. The method according to claim 1, characterized in that, The prediction information includes the filter coefficients used in the CCF process.

7. The method according to claim 6, characterized in that, The filter coefficients used in the CCF process are predefined constants.

8. The method according to claim 6, characterized in that, The method further includes: determining a second filter coefficient used in the CCF process in the current image based on a first filter coefficient used in the CCF process used in another image and an offset value of the filter coefficient used in the CCF process used in the current image.

9. The method according to claim 6, characterized in that, The method further includes: determining a fourth filter coefficient used in the CCF process in one color component based on a third filter coefficient used in the CCF process used in another color component.

10. The method according to claim 6, characterized in that, The method further includes: determining the filter coefficients used in the CCF process based on the sample values ​​of the current block determined before applying the CCF process.

11. A video encoding method applied to an encoder including a local decoder, characterized in that, include: Decode the prediction information of the current block in the current image, wherein the current image is part of an encoded video sequence, and the prediction information indicates the cross-component filtering (CCF) process for the current block. Whether the CCF process is enabled is determined based on one of the partitioning scheme, prediction mode, transform coefficients, and quantization parameters. When the luminance component and chrominance component have different partitioning schemes or semi-decoupled partitioning schemes, the CCF process is disabled. When a directional mode or internal composite mode is used, the CCF process is enabled. When the transform coefficients before or after dequantization are all zero, the CCF process is disabled. By applying the CCF process to at least one of the predicted samples, residual values, or reconstructed samples of the current block, filtered reconstructed samples of the current block are generated, which are then used for the reconstruction of subsequent blocks; and, Based on the filtered reconstruction samples of the current block, reconstruct the current block and the subsequent blocks.

12. A video decoding device, characterized in that, include: The decoding module is used to decode the prediction information of the current block in the current image, wherein the current image is part of an encoded video sequence, and the prediction information indicates the cross-component filtering (CCF) process for the current block. Whether the CCF process is enabled is determined based on one of the partitioning scheme, prediction mode, transform coefficients, and quantization parameters. When the luminance component and chrominance component have different partitioning schemes or semi-decoupled partitioning schemes, the CCF process is disabled. When the directional mode or internal composite mode is used, the CCF process is enabled. When the transform coefficients before or after dequantization are all zero, the CCF process is disabled. A generation module is configured to generate filtered reconstruction samples for the current block by applying the CCF process to at least one of the predicted samples, residual values, or reconstructed samples of the current block, the filtered reconstruction samples of the current block being used for the reconstruction of subsequent blocks; and, The reconstruction module reconstructs the current block and the subsequent blocks based on the filtered reconstruction samples of the current block.

13. The apparatus according to claim 12, characterized in that, The generation module is further used for: The filtered reconstruction sample of the current block is generated by applying the CCF process to the output of at least one of the dequantization and inverse transform processes of the current block.

14. The apparatus according to claim 12, characterized in that, The reconstruction module is further used for: The current block is reconstructed by adding the output of the CCF process, which includes the offset value of the current block, to the chromaticity reconstruction sample, chromaticity residual value, or chromaticity prediction sample of the current block.

15. The apparatus according to claim 12, characterized in that, The prediction information includes the filter coefficients used in the CCF process.

16. The apparatus according to claim 15, characterized in that, The filter coefficients used in the CCF process are predefined constants.

17. The apparatus according to claim 15, characterized in that, The device further includes: A determining module is configured to determine, based on a first filter coefficient used in the CCF process used in another image and an offset value of the filter coefficient used in the CCF process used in the current image, a second filter coefficient used in the current image; or, Based on the third filter coefficients used in the CCF process in one color component, determine the fourth filter coefficients used in the CCF process in one color component; or, The filter coefficients used in the CCF process are determined based on the sample values ​​of the current block determined before the CCF process is applied.

18. A video decoding device, characterized in that, include: One or more computer-readable non-volatile storage media for storing computer program code; as well as, One or more computer processors are configured to access the computer program code and operate in accordance with the instructions of the computer program code to perform the method of any one of claims 1 to 10.

19. A non-volatile computer-readable storage medium for storing instructions, characterized in that, When the instructions are executed by at least one processor, the at least one processor performs the method according to any one of claims 1 to 11.

20. A method for storing video bitstreams, characterized in that, The video encoding method of claim 11 is used to generate a video bitstream and to store the video bitstream.

21. A method for transmitting video bitstreams, characterized in that, The video encoding method of claim 11 is used to generate a video bitstream and to transmit the video bitstream.

Images