Method, apparatus and readable storage medium for video decoding

Abstract

Aspects of the disclosure include methods, apparatuses, and readable storage media of video decoding. An apparatus includes processing circuitry that determines reconstructed samples of a first color component in a filter support region of a current picture, the current picture being part of a coded video sequence. The processing circuitry performs a filtering process on the reconstructed samples to obtain an output value of the filtering process. A quantization step size of the filtering process is set separately for each of a current coding block, the current picture, and the coded video sequence. The processing circuitry reconstructs samples of a second color component in the filter support region based on the output value. The second color component is different from the first color component.

Description

[0001] INCORPORATION BY REFERENCE

[0002] This application claims priority to U.S. Patent Application No. 17 / 360,865, “METHOD AND APPARATUS FOR VIDEO CODING” filed on June 28, 2021, which claims priority to U.S. Provisional Application No. 63 / 113,125, “IMPROVED CROSS-COMPONENT SAMPLE OFFSET USING ADAPTIVE QUANTIZATION AND OFFSET SCALING” filed on November 12, 2020. The entire disclosures of these prior applications are hereby incorporated by reference in their entirety. TECHNICAL FIELD

[0003] This disclosure describes embodiments relating generally to video coding. BACKGROUND

[0004] The background description provided herein is intended to present the context of the present application. As such, work of the inventors in this regard, to the extent it is currently known, is considered to be part of the state of the art. The background description is provided for the purpose of generally presenting the context of the application. The work of the inventors in this regard is not, however, admitted to be prior art to the present application.

[0005] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having a spatial dimension of, for example, 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (also known as frame rate), of, for example, 60 pictures per second or 60 Hz. Uncompressed video has a very high data rate. For example, 1080p60 4:2:0 (1920x1080 luminance resolution, 60 Hz frame rate) video at 8 bit per sample (luma and chroma) requires close to 1.5 Gbit / s bandwidth. An hour of such video requires more than 600 GB of storage space.

[0006] One purpose of video encoding and decoding is to reduce redundant information in the input video signal through compression. Video compression can help reduce the requirement for the aforementioned bandwidth or storage space, in some cases by orders of magnitude. Both lossless and lossy compression, as well as a combination of the two, can be employed. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal can not be identical to the original signal but the distortion between the original and reconstructed signal is small enough to make the reconstructed signal useful for the intended application. Lossy compression is widely employed in video. The amount of distortion tolerated depends on the application. For example, some users of consumer streaming applications can tolerate higher distortion than users of a television application. The achievable compression ratio reflects that higher allowed / tolerated distortion can result in higher compression ratios.

[0007] Video encoders and decoders can utilize several broad categories of techniques, including, for example, motion compensation, transform, quantization, and entropy coding.

[0008] Video codec techniques can include known intra-coding techniques. In intra-coding, sample values are represented without reference to samples or other data of previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all blocks of a picture are coded in intra-mode, that picture can be an intra picture. Intra pictures and their derivations, e.g. independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and in a video session, or as a still image. Samples of an intra block can be subjected to a transform, and transform coefficients can be quantized before entropy coding. Intra prediction can be a technique that minimizes sample values in the pre-transform domain. In certain cases, the smaller the DC value after a transform is, and the smaller AC coefficients are, the fewer the bits that are needed to represent the block after entropy coding, at a given quantization step size.

[0009] Conventional intra coding does not use intra prediction, as known from e.g. MPEG-2 generation coding techniques. However, some newer video compression techniques include techniques that attempt to derive a data block from, e.g., surrounding sample data and / or metadata that was obtained during encoding and / or decoding, spatially adjacent, and prior in decoding order. Such techniques are later referred to as "intra prediction" techniques. It is noted that, at least in some cases, intra prediction only uses reference data of the current picture that is being reconstructed, and not reference data of reference pictures.

[0010] There can be many different forms of intra-prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be coded in intra-prediction modes. In some cases, a mode may have sub-modes and / or parameters, and these modes may be encoded individually or contained in mode codewords. Which codeword is used for a given mode, the combination of sub-modes and / or parameters will affect the coding efficiency gain through intra-prediction, and this also applies to entropy coding techniques used to convert codewords into bitstreams.

[0011] H.264 introduced an intra-frame prediction mode, which was improved in H.265 and further refined in newer coding techniques such as Joint Exploration Model (JEM), Universal Video Coding (VVC), and Baseline Set (BMS). Prediction blocks are formed by using neighboring sample values ​​belonging to already available samples. The sample values ​​of neighboring samples are copied into the prediction block in a specific direction. References to the direction used can be encoded in the bitstream or can be predicted themselves.

[0012] Referring to Figure 1A, the lower right corner depicts a subset of nine known prediction directions from the 33 possible prediction directions of H.265 (corresponding to 33 angular patterns out of 35 internal patterns). The point (101) where the arrows converge represents the sample being predicted. The arrow indicates the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples at a 45-degree angle to the horizontal direction from the upper right. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples at a 22.5-degree angle to the horizontal direction from the lower left.

[0013] Referring again to Figure 1A, a square block (104) comprising 4×4 samples is shown in the upper left (represented by a thick dashed line). The square block (104) consists of 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from top to bottom) and the first sample in the X dimension (from left to right). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block is a 4×4 size sample, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. Reference samples are labeled with "R" and their Y position (e.g., row index) and X position (e.g., column index) relative to block (104). In H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, so negative values ​​are not required.

[0014] Intra-frame image prediction can be performed by copying reference sample values ​​from adjacent samples occupied by the prediction direction indicated by the signal. For example, suppose the encoded video bitstream includes signaling that, for this block, the signaling indicates a prediction direction consistent with arrow (102), i.e., predicting samples based on one or more prediction samples at a 45-degree angle to the horizontal direction from the upper right. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Then, sample S44 is predicted based on reference sample R08.

[0015] In some cases, such as through interpolation, the values ​​of multiple reference samples can be combined to compute a reference sample, especially when the direction is not divisible by 45 degrees.

[0016] With the development of video coding technology, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013) and JEM / VVC / BMS, and at the time of this application, up to 65 directions could be supported. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding have been used to represent those possible directions using a small number of bits, while accepting some cost for less likely directions. Furthermore, the direction itself can sometimes be predicted based on the adjacent directions used in adjacent, already decoded blocks.

[0017] Figure 1B is a schematic diagram (105) illustrating 65 intra-frame prediction directions based on JEM to show how the number of prediction directions increases over time.

[0018] The mapping of intra-predicted direction bits in a coded video bitstream can vary depending on the video coding technique, and can range from a simple, direct mapping of intra-predicted modes to predicted directions in codewords to complex adaptive schemes that incorporate the most probable modes and similar techniques. However, in all cases, there may be certain directions in the video content that are statistically less likely to occur than others. Since the purpose of video compression is to reduce redundancy, in well-functioning video coding techniques, less probable directions will be represented using a greater number of bits compared to the more probable directions.

[0019] Motion compensation can be a lossy compression technique and can involve using sample data blocks from a previously reconstructed image or a portion thereof (the reference image), spatially shifted in a direction indicated by a motion vector (MV), for prediction of the newly reconstructed image or a portion thereof. In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference image in use (indirectly, the latter could be a temporal dimension).

[0020] In some video compression techniques, a video feature (MV) applicable to a region of sample data can be predicted from other MVs, such as those MVs preceding the one being reconstructed, which are spatially adjacent to the region being reconstructed. This significantly reduces the amount of data required to encode the MV, eliminating redundancy and increasing compression. MV prediction can work effectively, for example, because when encoding an input video signal derived from a camera (called natural video), regions larger than the area applicable to a single MV are statistically likely to move in similar directions. Therefore, in some cases, these regions (regions larger than the area applicable to a single MV) can be predicted using similar MVs derived from neighboring regions. This results in MVs found for a given region being similar to or identical to MVs predicted from surrounding MVs, and, conversely, after entropy encoding, the MV can be represented with fewer bits than when directly encoding the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself may be lossy, for example due to rounding errors when calculating predictions from several surrounding MVs.

[0021] H.265 / HEVC (ITU-T H.265 Recommendation, “High Efficiency Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this application describes the technique hereinafter referred to as “spatial combining”.

[0022] Referring to Figure 1C, the current block (111) includes samples discovered by the encoder during the motion search process, which can be predicted based on previous blocks of the same size that have generated spatial offsets. Alternatively, the MV can be derived from metadata associated with one or more reference images, rather than being directly encoded. For example, using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 112 to 116 respectively), the MV is derived from the metadata of the nearest reference image (in decoding order). In H.265, MV prediction can use predictions from the same reference image that is also being used in adjacent blocks. Summary of the Invention

[0023] This disclosure provides various aspects of a video decoding apparatus. One video decoding apparatus includes processing circuitry that determines reconstructed samples of a first color component in a filter support region of a current image, said current image being part of an encoded video sequence. The processing circuitry performs a filtering process on the reconstructed samples to obtain an output value of the filtering process. A quantization step size for the filtering process is set for each of the encoded block, video frame, and video sequence. Based on the output value, the processing circuitry reconstructs samples of a second color component in the filter support region, said second color component being different from the first color component.

[0024] In this embodiment, the quantization step size is a predefined constant for one of the current coding block, the current image, and the encoded video sequence.

[0025] In this embodiment, the quantization step size is an integer that is a power of 2.

[0026] In an embodiment, the quantization step size is one of a plurality of quantization steps, which are determined based on a first index included in the bitstream, the first index corresponding to one of the current coding block, the current image, and the encoded video sequence.

[0027] In an embodiment, the quantization step size is included in the bitstream, and the quantization step size corresponds to one of the current coding block, the current image, and the encoded video sequence.

[0028] In one embodiment, the processing circuit determines whether the output value needs to be scaled based on the reconstructed sample of the first color component. In response to the output value needing scaling, the processing circuit reconstructs the sample of the second color component in the filter support region based on the scaled output value.

[0029] In one embodiment, the processing circuit determines whether the output value of the filtering process needs to be scaled based on the reconstructed sample of the first color component and the first syntax element, wherein the first syntax element indicates whether scaling of the output value is enabled.

[0030] In one embodiment, the processing circuitry determines the scaled output value based on the reconstructed sample of the first color component and the quantization step size.

[0031] This disclosure provides methods for video decoding. In this method, reconstructed samples of a first color component in a filter support region of a current image, which is part of an encoded video sequence, are determined. A filtering process is performed on the reconstructed samples to obtain output values ​​of the filtering process, wherein the quantization step size of the filtering process is set separately for each of the current coding block, the current image, and the encoded video sequence. Based on the output values, samples of a second color component in the filter support region are reconstructed, the second color component being different from the first color component.

[0032] Various aspects of this disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform any one or more combinations of video decoding methods. Attached Figure Description

[0033] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0034] Figure 1A is a schematic diagram of an exemplary subset of intra-prediction modes.

[0035] Figure 1B is an illustration of an exemplary intra-frame prediction direction.

[0036] Figure 1C is a schematic diagram of the current block and its surrounding space merge candidates in an example.

[0037] Figure 2 This is a simplified block diagram of a communication system according to an embodiment;

[0038] Figure 3 This is a simplified block diagram of a communication system according to an embodiment;

[0039] Figure 4 This is a simplified block diagram of the decoder according to an embodiment;

[0040] Figure 5 This is a simplified block diagram of the encoder according to an embodiment;

[0041] Figure 6 A block diagram of an encoder according to another embodiment is shown;

[0042] Figure 7 A block diagram of a decoder according to another embodiment is shown;

[0043] Figures 8A-8B An exemplary diamond filter shape for an adaptive loop filter (ALF) according to some embodiments is shown;

[0044] Figures 9A-9DExemplary subsampling positions for gradient calculation in the vertical, horizontal and two diagonal directions, respectively, are shown according to some embodiments;

[0045] Figure 10 An exemplary block classification applied to the luminance component according to an embodiment is shown;

[0046] Figure 11 An exemplary modified ALF of the luminance component at a virtual boundary according to some embodiments is shown;

[0047] Figure 12 An exemplary image quadtree segmentation aligned with maximum coding unit (LCU) according to an embodiment is shown;

[0048] Figure 13 An exemplary quadtree segmentation flag encoded in z-order according to an embodiment is shown;

[0049] Figure 14A An exemplary arrangement of a cross-component adaptive loop filter (CC-ALF) according to an embodiment is illustrated;

[0050] Figure 14B An exemplary linear diamond filter is shown for the luminance channel applied to each chromaticity component during CC-ALF operation according to an embodiment;

[0051] Figure 15 Exemplary positions of chromaticity samples relative to luminance samples are shown according to some embodiments;

[0052] Figure 16 An exemplary directional search of an 8×8 block according to an embodiment is shown;

[0053] Figure 17 An exemplary subspace projection according to an embodiment is shown;

[0054] Figure 18 An exemplary filter support region for a Cross Component Sample Offset (CCSO) filter according to an embodiment is shown;

[0055] Figures 19A-19B A table is shown illustrating 81 exemplary combinations of nonlinear mappings in a CCSO filter according to an embodiment;

[0056] Figure 20 An exemplary flowchart according to an embodiment is shown; and

[0057] Figure 21 This is a schematic diagram of a computer system according to an embodiment. Detailed Implementation

[0058] I. Video Decoder and Encoder Systems

[0059] Figure 2 This is a simplified block diagram of a communication system (200) according to an embodiment disclosed in this application. The communication system (200) includes a plurality of terminal devices that can communicate with each other via, for example, a network (250). For example, the communication system (200) includes a first terminal device (210) and a second terminal device (220) interconnected via a network (250). Figure 2 In this embodiment, the first terminal device (210) and the second terminal device (220) perform unidirectional data transmission. For example, the first terminal device (210) may encode video data (e.g., a video image stream captured by the first terminal device (210)) for transmission over a network (250) to the second terminal device (220). The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device (220) may receive the encoded video data from the network (250), decode the encoded video data to recover the video data, and display video images based on the recovered video data. Unidirectional data transmission is common in applications such as media services.

[0060] In another embodiment, the communication system (200) includes a third terminal device (230) and a fourth terminal device (240) that perform bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (230) and the fourth terminal device (240) may encode video data (e.g., a stream of video images captured by the terminal device) for transmission over a network (250) to the other terminal device. Each of the third terminal device (230) and the fourth terminal device (240) may also receive encoded video data transmitted by the other terminal device and may decode the encoded video data to recover the video data, and may display the video images on an accessible display device based on the recovered video data.

[0061] exist Figure 2In the embodiments disclosed herein, the first terminal device (210), the second terminal device (220), the third terminal device (230), and the fourth terminal device (240) may be servers, personal computers, and smartphones, but the principles disclosed herein are not limited thereto. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (250) refers to any number of networks that transmit encoded video data between the first terminal device (210), the second terminal device (220), the third terminal device (230), and the fourth terminal device (240), including, for example, wired (connected) and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. The network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of the network (250) may be irrelevant to the operation of this application.

[0062] As an example, Figure 3 The diagram illustrates the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0063] The streaming system may include an acquisition subsystem (313) that may include a video source (301) such as a digital camera, which creates an uncompressed video image stream (302). In an embodiment, the video image stream (302) includes samples captured by a digital camera. The video image stream (302) is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data (304) (or encoded video bitstream). The video image stream (302) may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (302), the encoded video data (304) (or the encoded video bitstream (304)) is depicted as a thin line to emphasize the lower data volume of the encoded video data (304) (or the encoded video bitstream (304)), which can be stored on a streaming server (305) for future use. One or more streaming client subsystems, such as Figure 3Client subsystems (306) and (308) can access a streaming server (305) to retrieve copies (307) and (309) of encoded video data (304). Client subsystem (306) may include, for example, a video decoder (310) in an electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and produces an output video picture stream (311) that can be displayed on a display (312) (e.g., a screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (304), video data (307), and video data (309) (e.g., video streams) may be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T H.265. In embodiments, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and this application can be used in the context of the VVC standard.

[0064] It should be noted that the electronic devices (320) and (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0065] Figure 4 This is a block diagram of a video decoder (410) according to an embodiment disclosed in this application. The video decoder (410) may be disposed in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., receiving circuitry). The video decoder (410) may be used in place of... Figure 3 The video decoder (510) in the embodiment.

[0066] The receiver (431) may receive one or more encoded video sequences to be decoded by the video decoder (410); in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (431) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (431) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter referred to as "parser (420)"). In some applications, the buffer memory (415) is part of the video decoder (410). In other cases, the buffer memory (415) may be located external to the video decoder (410) (not indicated). In other cases, an external buffer (not shown) may be provided for the video decoder (410) to prevent network jitter, for example, and another buffer (415) may be configured internally for, for example, handling broadcast timing. When the receiver (431) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer (415) may not be necessary, or it may be made smaller. Of course, for use on packet networks such as the Internet, a buffer (415) may be required; this buffer may be relatively large and adaptive in size, and may be at least partially implemented in the operating system or a similar component (not shown) external to the video decoder (410).

[0067] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (410) and potential information for controlling a display device (412) (e.g., a display screen), which is not part of the electronic device (430) but may be coupled to it, such as... Figure 4As shown in the figure. The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (420) may parse / decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels in the encoded video sequence for use in the video decoder based on at least one parameter corresponding to a group. The subgroup may include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (420) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, MV, etc.

[0068] The parser (420) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to create symbols (421).

[0069] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (421) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed by the parser (420) from the encoded video sequence. For brevity, the flow of such subgroup control information between the parser (420) and the various units described below is not described.

[0070] In addition to the functional blocks already mentioned, the video decoder (410) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0071] The first unit is the scaler / inverse transform unit (451). The scaler / inverse transform unit (451) receives quantization transform coefficients as symbols (421) and control information from the parser (420), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (451) can output a block containing sample values, which can be input into the aggregator (455).

[0072] In some cases, the output samples of the scaler / inverse transform unit (451) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) uses reconstructed information extracted from the current picture buffer (458) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (458) buffers partially reconstructed and / or fully reconstructed current images. In some cases, the aggregator (455) adds the predictive information generated by the intra-picture prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451) based on each sample.

[0073] In other cases, the output samples of the scaler / inverse transform unit (451) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (453) can access the reference image memory (457) to extract samples for prediction. After motion compensation of the extracted samples according to the symbols (421), these samples can be added by the aggregator (455) to the output of the scaler / inverse transform unit (451) (referred to as residual samples or residual signals in this case) to generate output sample information. The motion compensation prediction unit (453) can obtain the prediction samples from the address in the reference image memory (457) under motion vector control, and the motion vector is available to the motion compensation prediction unit (453) in the form of the symbols (421), which, for example, include X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (457) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0074] The output samples of the aggregator (455) can be employed by various loop filtering techniques in the loop filter unit (454). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (421) from the parser (420) in the loop filter unit (456). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0075] The output of the loop filter unit (456) can be a sample stream, which can be output to a display device (412) and stored in a reference image memory (457) for subsequent inter-frame image prediction.

[0076] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (420)) is identified as the reference image, the current image buffer (458) can become part of the reference image memory (457), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0077] The video decoder (410) can perform decoding operations according to a predetermined video compression technique, such as that specified in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0078] In this embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0079] Figure 5 This is a block diagram of a video encoder (503) according to an embodiment disclosed in this application. The video encoder (503) is disposed in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) can be used to replace... Figure 3 The video encoder (303) in the embodiment.

[0080] The video encoder (503) can obtain data from the video source (501) (not) Figure 5 In one embodiment, a portion of the electronic device (520) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (503). In another embodiment, the video source (501) is a portion of the electronic device (520).

[0081] A video source (501) can provide a sequence of source video samples encoded by a video encoder (503) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (501) can be a storage device storing previously prepared video. In a video conferencing system, the video source (501) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0082] According to an embodiment, the video encoder (503) can encode and compress images of a source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (550). In some embodiments, the controller (550) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (550) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum allowed motion vector reference area, etc. The controller (550) may be used with other suitable functions related to the video encoder (503) optimized for a particular system design.

[0083] In some embodiments, the video encoder (503) operates within an encoding loop. As a simplified description, in an embodiment, the encoding loop may include a source encoder (530) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (533) embedded within the video encoder (503). The decoder (533) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (534). Since decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image memory (534) are also bit-precisely corresponding between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is also used in some related technologies.

[0084] The operation of the “local” decoder (533) can be combined with, for example, the above-described method. Figure 4 The video decoder (410) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 4 When symbols are available and the entropy encoder (545) and parser (420) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (410), including the buffer (415) and parser (420), may not be fully implemented in the local decoder (533).

[0085] It can be observed that any decoder technique other than parsing / entropy decoding present in the decoder must also exist in the corresponding encoder in essentially the same functional form. For this reason, this application focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0086] During operation, in some embodiments, the source encoder (530) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input image, referencing one or more previously encoded images from the video sequence designated as "reference images." In this manner, the encoding engine (532) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

[0087] The local video decoder (533) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (530). The operation of the encoding engine (532) can be a lossy process. When the encoded video data can be decoded by the video decoder (533), Figure 5 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (533) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image cache (534). In this way, the video encoder (503) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0088] The predictor (535) can perform a prediction search against the encoding engine (532). That is, for a new image to be encoded, the predictor (535) can search in the reference image memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (535) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (535), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (534).

[0089] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

[0090] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (545). The entropy encoder (545) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0091] The transmitter (540) can buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission via a communication channel (560), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (540) can combine the encoded video data from the video encoder (503) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0092] The controller (550) manages the operation of the video encoder (503). During encoding, the controller (550) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0093] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand variations of I-pictures and their corresponding applications and characteristics.

[0094] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0095] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0096] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0097] The video encoder (503) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (503) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0098] In this embodiment, the transmitter (540) may transmit additional data while transmitting encoded video. The source encoder (530) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0099] The acquired video can serve as multiple source images (video images) presented in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0100] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0101] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0102] According to some embodiments disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video image sequence are segmented into coding tree units (CTUs) for compression. The CTUs in the images have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Furthermore, each CTU can be further subdivided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be subdivided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In embodiments, each CU is analyzed to determine the prediction type used for the CU, such as inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. Taking a luma prediction block as an example, a prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0103] Figure 6 This is a diagram of a video encoder (603) according to another embodiment disclosed in this application. The video encoder (603) is used to receive processing blocks (e.g., prediction blocks) of sample values ​​within a current video image in a video image sequence, and to encode the processing blocks into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (603) is used instead of Figure 3The video encoder (303) in the embodiment.

[0104] In the HEVC embodiment, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as an 8×8 sample prediction block. The video encoder (603) uses, for example, rate-distortion optimization to determine whether to use intra-frame mode, inter-frame mode, or bidirectional prediction mode to encode the processing block. When encoding the processing block in intra-frame mode, the video encoder (603) can use intra-frame prediction techniques to encode the processing block into an encoded picture; and when encoding the processing block in inter-frame mode or bidirectional prediction mode, the video encoder (603) can use inter-frame prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some video coding techniques, the merging mode can be an inter-frame picture prediction sub-mode, in which motion vectors are derived from one or more motion vector prediction values ​​without relying on encoded motion vector components outside the prediction values. In some other video coding techniques, motion vector components applicable to the subject block may exist. In the embodiment, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0105] exist Figure 6 In one embodiment, the video encoder (603) includes, as shown below: Figure 6 The inter-frame encoder (630), intra-frame encoder (622), residual calculator (623), switch (626), residual encoder (624), general controller (621) and entropy encoder (625) are shown coupled together.

[0106] An inter-frame encoder (630) is configured to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and later images), generate inter-frame prediction information (e.g., redundancy information description, motion vectors, merging mode information based on inter-frame coding techniques), and calculate inter-frame prediction results (e.g., prediction blocks) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference image is a decoded reference image based on encoded video information.

[0107] The intra encoder (622) is used to receive samples of the current block (e.g., the processing block), in some cases compare the block with previously encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases also (e.g., based on intra prediction direction information of one or more intra coding techniques) generate intra prediction information. In an embodiment, the intra encoder (622) also calculates intra prediction results (e.g., prediction blocks) based on the intra prediction information and reference blocks in the same image.

[0108] A general-purpose controller (621) determines general-purpose control data and controls other components of the video encoder (603) based on the general-purpose control data. In an embodiment, the general-purpose controller (621) determines the mode of a block and provides control signals to a switch (626) based on the mode. For example, when the mode is an intra-frame mode, the general-purpose controller (621) controls the switch (626) to select an intra-frame mode result for use by the residual calculator (623) and controls the entropy encoder (625) to select intra-frame prediction information and add the intra-frame prediction information to the bitstream; and when the mode is an inter-frame mode, the general-purpose controller (621) controls the switch (626) to select an inter-frame prediction result for use by the residual calculator (623) and controls the entropy encoder (625) to select inter-frame prediction information and add the inter-frame prediction information to the bitstream.

[0109] A residual calculator (623) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (622) or the inter encoder (630). A residual encoder (624) is used to operate on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (624) is used to transform the residual data from the time domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (622) and the inter encoder (630). For example, the inter encoder (630) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (622) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and in some embodiments, the decoded image may be buffered in a memory circuit (not shown) and used as a reference image.

[0110] An entropy encoder (625) is used to format the bitstream to produce encoded blocks. The entropy encoder (625) generates various information according to a suitable standard such as HEVC. In an embodiment, the entropy encoder (625) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. It should be noted that, according to the disclosed subject matter, residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0111] Figure 7This is a diagram of a video decoder (710) according to another embodiment disclosed in this application. The video decoder (710) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (710) is used instead of Figure 3 The video decoder (310) in the embodiment.

[0112] exist Figure 7 In this embodiment, the video decoder (710) includes, as follows: Figure 7 The entropy decoder (771), inter-frame decoder (780), residual decoder (773), reconstruction module (774), and intra-frame decoder (772) are shown coupled together.

[0113] An entropy decoder (771) can be used to reconstruct certain symbols from an encoded image, these symbols representing the syntax elements constituting the encoded image. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can respectively identify certain samples or metadata used by the intra-frame decoder (772) or the inter-frame decoder (780) for prediction, residual information in the form of, for example, quantized transform coefficients, and so on. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (780); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (772). Residual information may be provided to the residual decoder (773) via inverse quantization.

[0114] The inter-frame decoder (780) is used to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0115] The intra-frame decoder (772) is used to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0116] The residual decoder (773) performs inverse quantization to extract the dequantized transform coefficients and processes the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require some control information (to obtain the quantizer parameters QP), and this information may be provided by the entropy decoder (771) (the data path is not indicated because this is only low-level control information).

[0117] The reconstruction module (774) is used to combine the residual output by the residual decoder (773) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which may be a part of a reconstructed image, which in turn may be a part of a reconstructed video. It should be noted that other suitable operations, such as deblocking, may be performed to improve visual quality.

[0118] It should be noted that any suitable technology can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). In one embodiment, one or more integrated circuits can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710).

[0119] II. Adaptive Loop Filter

[0120] In some relevant examples, such as VVC Draft 8, an adaptive loop filter (ALF) with block-based filter adaptation can be applied. For the luminance component, a filter can be selected from 25 filters for each 4×4 block based on the directivity and activity of each 4×4 block.

[0121] Figures 8A-8B An exemplary diamond filter shape for an ALF according to some embodiments of the present disclosure is shown. Figure 8A A 5×5 rhombus shape that can be applied to the chromaticity component is shown, and Figure 8B A 7×7 rhombus shape that can be applied to the luminance component is shown.

[0122] In some relevant examples, for the luminance component, each 4×4 block can be classified into one of 25 categories. The activity of the 4×4 block can be used as a basis for classification. The quantization value and directionality D are used to derive the 4×4 block classification index C as follows:

[0123]

[0124] To calculate D and The gradients in the horizontal, vertical, and two diagonal directions of the 4×4 block can be calculated first using the 1-D Laplacian operator as follows:

[0125]

[0126]

[0127]

[0128]

[0129] Here, indices i and j refer to the coordinates of the top-left sample within the 4×4 block, and R(i, j) indicates the reconstructed sample at coordinates (i, j).

[0130] To reduce the complexity of block classification, a 1-D Laplace operator for subsampling can be used for computation. Figures 9A-9D Exemplary subsampling positions for gradient calculation in the vertical, horizontal, and two diagonal directions are shown respectively.

[0131] Then, the maximum and minimum values ​​of the gradients in the horizontal and vertical directions can be set as follows:

[0132]

[0133]

[0134] The maximum and minimum values ​​of the gradients in the two diagonal directions can be set as:

[0135]

[0136]

[0137] To derive the values ​​of the directionality D, these values ​​can be compared with each other and with two thresholds t1 and t2.

[0138] Step 1: If and If both are true, then set D to 0.

[0139] Step 2: If Continue from step 3; otherwise, continue from step 4.

[0140] Step 3: If If so, set D to 2; otherwise, set D to 1.

[0141] Step 4: If If so, set D to 4; otherwise, set D to 3.

[0142] Activity value A is calculated as follows:

[0143]

[0144] A is further quantized to the range of 0 to 4 (inclusive), and the quantized value is represented as

[0145] Classification methods should not be applied to the chromaticity components in the image. For example, a single set of ALF coefficients can be applied to each chromaticity component.

[0146] Before filtering each 4×4 luminance block, geometric transformations such as rotation, diagonal flip, and vertical flip are applied to the filter coefficients f(k, l) and the corresponding filter threshold values ​​c(k, l), depending on the gradient values ​​calculated for each block. This is equivalent to applying these transformations to samples in the filter support region (or filter support area) to make the different blocks to which ALF is applied more similar by aligning the orientation of ALF.

[0147] Three geometric transformations (including diagonal flip, vertical flip, and rotation) can be described as follows:

[0148] Diagonal: f D (k, l) = f(l, k), c D (k, l) = c(l, k) Equation (11)

[0149] Vertical flip: f V (k, l) = f(k, Kl-1), c V (k, l) = c(k, Kl-1) Equation (12)

[0150] Rotation: f R (k, l) = f(Kl-1, k), c R (k, l) = c(Kl-1, k) Equation (13)

[0151] Where K is the size of the filter, and 0≤k, l≤K-1 are the coordinates of the transform coefficients, such that position (0,0) is at the top left corner and position (K-1,K-1) is at the bottom right corner. The filter coefficients f(k,l) and the limiting value c(k,l) are transformed based on the gradient values ​​calculated for the corresponding blocks. The relationship between the transform and the gradients in the four directions can be summarized in Table 1.

[0152] Table 1

[0153] Gradient value Transform g d2 g d1 and g h g v ]]> No transform g d2 g d1 and g v g h ]] Diagonal g d1 g d2 and g h g v ]] Vertical flip g d1 g d2 and g v g h ]]> Rotation

[0154] In some relevant examples, such as VVC Draft 8, the filter parameters of the ALF are signaled in the Adaptive Parameter Set (APS). Within an APS, up to 25 sets of luma filter coefficients and limiting indexes, and up to 8 sets of chroma filter coefficients and limiting indexes, can be signaled. To reduce bit overhead, filter coefficients from different classifications of the luma components can be combined. In the slice header, the index of the APS used for the current slice is signaled. In VVC Draft 8, the ALF signaling is based on the CTU.

[0155] The limiting index decoded from the APS allows the use of limiting tables for the luma and chroma components to determine the limiting value. These limiting values ​​depend on the internal bit depth. For example, the limiting table can be obtained using the following formula:

[0156] AlfClip = {round(2 B-α*n Equation (14) for n∈[0..N-1]}

[0157] Where B equals the internal bit depth, α is a predefined constant of 2.35, and N equals 4, where N is the number of clipping values ​​allowed in VVC draft 8. Table 2 shows an example of the output of equation (14).

[0158] Table 2

[0159]

[0160] In the slice header, up to seven APS indices can be signaled to specify the luma filter set used for the current slice. The filtering process can be further controlled at the CTB level. Flags can be signaled to indicate whether the ALF is applied to the luma CTB. The luma CTB can select a filter set from 16 fixed filter sets and filter sets from the APS. The filter set index used for the luma CTB is signaled to indicate which filter set is applied. The 16 fixed filter sets are predefined and hardcoded in the encoder and decoder.

[0161] For chroma components, the APS index can be signaled in the slice header to indicate the set of chroma filters used for the current slice. At the CTB level, if there is more than one set of chroma filters in the APS, the filter index for each chroma CTB can be signaled.

[0162] Filter coefficients can be quantized using a norm equal to 128. To limit the complexity of multiplication operations, stream consistency can be applied so that coefficient values ​​at non-center positions can be in the range of -27 to 27-1 (inclusive). Center position coefficients are not signaled in the stream, and they are considered equal to 128.

[0163] In some relevant examples such as VVC Draft 8, the syntax and semantics of the limit index and its corresponding value can be defined as follows.

[0164] `alf_luma_clip_idx[sfIdx][j]` specifies the clipping index to be used before multiplying by the j-th coefficient of the luminance filter indicated by `sfIdx`. The requirement for bitstream consistency is that the value of `alf_luma_clip_idx[sfIdx][j]`, where `sfIdx = 0..alf_luma_num_filters_signalled_minus1` and `j = 0..11`, should be in the range of 0 to 3 (inclusive).

[0165] Based on bitDepth being set to equal BitDepthY and clipIdx being set to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j], the luminance filter limiting value AlfClipL[adaptation_parameter_set_id][filtIdx][j] with the element AlfClipL[adaptation_parameter_set_id][filtIdx][j] (where filtIdx = 0..NumAlfFilters-1 and j = 0..11) is derived in Table 2.

[0166] `alf_chroma_clip_idx[altIdx][j]` specifies the clipping index to use before multiplying the j-th coefficient of the optional chroma filter with index `altIdx`. The requirement for bitstream consistency is that the value of `alf_chroma_clip_idx[altIdx][j]` (where `altIdx = 0..alf_chroma_num_alt_filters_minus1`, `j = 0..5`) should be in the range of 0 to 3 (inclusive).

[0167] Based on bitDepth being set to equal BitDepthC and clipIdx being set to equal alf_chroma_clip_idx[altIdx][j], the chroma filter limiting value AlfClipC[adaptation_parameter_set_id][altIdx][j] with the element AlfClipC[adaptation_parameter_set_id][altIdx][j] (where altIdx = 0..alf_chroma_num_alt_filters_minus1, j = 0..5) is derived in Table 2.

[0168] At the decoder end, when ALF is enabled for CTB, each sample R(i,j) in the CU is filtered to produce the corresponding sample value R′(i,j) as shown below.

[0169]

[0170]

[0171] Where f(k, l) represents the decoded filter coefficients, K(x, y) is the limiting function, and c(k, l) represents the decoded limiting parameters. The variables k and l are... and The values ​​vary between L and L, where L represents the filter length. The limiting function K(x, y) = min(y, max(-y, x)), which corresponds to the function Clip3(-y, y, x). By incorporating this limiting function, the loop filtering method becomes a nonlinear process, known as the nonlinear ALF. The selected limiting value is encoded in the "alf_data" syntax element using the Golomb coding scheme corresponding to the index of the limiting values ​​in Table 2. This coding scheme is the same as the one used for the filter index.

[0172] Figure 10 An exemplary modified block classification applied to the luminance component according to an embodiment of this disclosure is shown. Modified block classification and filtering, for samples near the horizontal CTU boundary, can reduce the line buffer requirements of the ALF. Figure 10 As shown, the virtual boundary is defined as a line by shifting the horizontal CTU boundary with “N” samples, where N equals 4 for the luminance component and N equals 2 for the chrominance component.

[0173] For the computation of the 1D Laplacian gradient of a 4×4 block above the virtual boundary, only samples above the virtual boundary are used. Similarly, for the computation of the 1D Laplacian gradient of a 4×4 block below the virtual boundary, only samples below the virtual boundary are used. To account for the reduced number of samples used in the 1D Laplacian gradient computation, the quantization of the activity value A is scaled accordingly.

[0174] Figure 11 An ALF diagram showing a modified luminance component at an exemplary virtual boundary according to some embodiments is shown. For filtering, a symmetric padding operation at the virtual boundary can be applied to both the luminance and chrominance components. Figure 11 As shown, when the filtered sample is below the virtual boundary, the adjacent samples above the virtual boundary are filled. At the same time, the corresponding samples on the other side are also filled symmetrically.

[0175] To improve encoding and decoding efficiency, some related examples use ALP based on a synchronized image quadtree of coding units. The luminance image is divided into several multi-level quadtree partitions, and the boundaries of each partition are aligned with the boundaries of the maximum coding unit (LCU). Each partition has its own filtering process and can therefore be called a filter unit (FU).

[0176] The two-stage encoding process is described below. In the first stage, the quadtree segmentation pattern and optimal filter for each function element (FU) are determined. During the decision-making process, filtering distortion is estimated using Fast Filter Distortion Estimation (FFDE). Based on the determined quadtree segmentation pattern and the filters for all selected FUs, the reconstructed image is filtered. In the second stage, CU-synchronized ALF on / off control is performed. Based on the ALF on / off result, the filtered image from the first stage is partially recovered from the reconstructed image.

[0177] Figure 12 An exemplary LCU-aligned image quadtree segmentation according to an embodiment is illustrated. A top-down segmentation strategy is employed to divide the image into multi-level quadtree partitions using a rate-distortion criterion. Each partition is called a filter unit. The segmentation process aligns the quadtree partitions with the LCU boundaries. The encoding order of the FUs follows the z-scan order. For example, as... Figure 12 As shown, the image is divided into 10 FUs, and the encoding order is FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8 and FU9.

[0178] Figure 13 It shows the corresponding Figure 12 An exemplary quadtree segmentation pattern. To indicate the quadtree segmentation pattern of an image, segmentation flags can be encoded and transmitted in z-order.

[0179] The filter for each FU can be selected from two filter sets based on a rate-distortion criterion. The first set contains 1 / 2 symmetric square filters and diamond filters newly derived for the current FU. The second set comes from a time-delay filter buffer that stores filters previously derived for FUs of previous images. The filter with the minimum rate-distortion cost from these two sets can be selected for the current FU. Similarly, if the current FU is not the minimum FU and can be further segmented into 4 sub-FUs, the rate-distortion cost of the 4 sub-FUs is calculated. By recursively comparing the rate-distortion costs of the segmented and unsegmented cases, the quadtree segmentation pattern of the image can be determined.

[0180] In some relevant examples, the maximum quadtree split level is 2, meaning the maximum number of functional units (FUs) is 16. During the quadtree split decision, correlation values ​​can be reused to derive the Wiener coefficients for the 16 FUs (minimum FUs) at the bottom quadtree level. The remaining FUs can be derived from the correlations of the 16 FUs at the bottom quadtree level. Therefore, only one frame buffer access is needed to derive the filter coefficients for all FUs.

[0181] After determining the quadtree partitioning pattern, to further reduce filtering distortion, CU-synchronized ALF on / off control can be performed. By comparing filtered distortion and non-filtered distortion, leaf CUs can explicitly turn the ALF on / off in their local regions. Encoding / decoding efficiency can be further improved by redesigning the filter coefficients based on the ALF on / off results. However, the redesign process requires additional frame buffer accesses. In some relevant examples, there is no redesign process after the CU-synchronized ALF on / off decision to minimize the number of frame buffer accesses.

[0182] III. Cross-component adaptive loop filter

[0183] In some relevant examples, a cross-component adaptive loop filter (CC-ALF) is used. CC-ALF uses luminance sample values ​​to correct each chrominance component.

[0184] Figure 14A An exemplary arrangement of a CC-ALF according to an embodiment of the present disclosure is illustrated. Figure 14B An exemplary linear rhombus filter is shown, which is applied to the luminance channel of each chroma component during CC-ALF operation. The filter coefficients can be transmitted in the APS, for example, according to a factor of 2. 10Scaling and rounding are performed for use in fixed-point representation. The application of the filter is controlled on a variable block size and signaled by a context-coded flag received for each sample block. At the slice level, the block size and CC-ALF enable flag are received for each chroma component. In the example, the following block sizes (in chroma samples) are supported: 16×16, 32×32, and 64×64.

[0185] Table 3 shows the syntax elements associated with CC-ALF.

[0186] Table 3

[0187]

[0188]

[0189] In Table 3, syntax elements

[0190] `alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]` equal to 0 indicates that the cross component Cb filter is not applied to the sample block in the Cb color component at the luminance location (xCtb, yCtb). Syntax element

[0191] alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, indicating that the alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] cross component Cb filter is applied to the sample block of the Cb color component at the luminance position (xCtb, yCtb).

[0192] The syntax element `alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]` equal to 0 indicates that the cross-component Cr filter is not applied to the sample block of the Cr color component at the luminance position (xCtb, yCtb). The syntax element `alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]` not equal to 0 indicates that the 1st cross-component Cr filter is applied to the sample block of the Cr color component at the luminance position (xCtb, yCtb).

[0193] IV. Chromaticity Sampling Format

[0194] Figure 15 This shows an exemplary relative position of the top-left chroma sample when chroma_format_idc equals 1 (4:2:0 chroma format) and chroma_sample_loc_type_top_field or chroma_sample_loc_type_bottom_field equals the value of the variable ChromaLocType. The area represented by the top-left 4:2:0 chroma sample is shown relative to the area represented by the top-left luminance sample.

[0195] V. Constrained Directional Enhancement Filter

[0196] One goal of the In-Loop Constrained Directional Enhancement Filter (CDEF) is to filter out coded artifacts while preserving image details. In some related examples such as HEVC, the Sample Adaptive Offset (SAO) algorithm can achieve a similar goal by defining signal offsets for pixels of different classes. Unlike SAO, CDEF is a nonlinear spatial filter. The design of this filter is constrained to be easily vectorized (e.g., implemented using Single Instruction Multiple Data (SIMD) operations), which is not the case for other nonlinear filters such as median filters and bilateral filters.

[0197] The CDEF design is based on the following observations: The amount of ringing artifacts in an encoded image tends to be roughly proportional to the quantization step size. The amount of detail is a property of the input image, but even the smallest details preserved in the quantized image tend to be proportional to the quantization step size. For a given quantization step size, the amplitude of ringing is typically smaller than the amplitude of detail.

[0198] CDEF works by identifying the orientation of each block and then adaptively filtering along that orientation, with a smaller degree of filtering along the orientation rotated 45 degrees from the identified orientation. The filter strength is explicitly signaled, allowing for a high degree of control over blurring. An efficient encoder search is designed for the filter strength. CDEF is based on two previously proposed in-loop filters and combines them for the emerging AOMedia Video 1 (AV1) codec.

[0199] Figure 16 An exemplary orientation search for an 8×8 block according to an embodiment of this disclosure is illustrated. The orientation search is performed on the reconstructed pixels immediately after the deblocking filter. Since these pixels are available for the decoder, orientation signaling is not required. The search is performed on an 8×8 block, which is small enough to adequately handle non-linear edges, yet large enough to reliably estimate the orientation when applied to a quantized image. Having a constant orientation over the 8×8 area also makes vectorization of the filter easier. For each block, the orientation that best matches the pattern in the corresponding block is determined by minimizing the sum of squared differences (SSD) between the quantized block and the nearest fully oriented block. A fully oriented block is a block in which all pixels along a line in one direction have the same value.

[0200] One reason for identifying the direction is to align filter taps along that direction to reduce ringing while preserving directional edges or patterns. However, directional filtering alone is sometimes insufficient to reduce ringing. It is also desirable to use filter taps on pixels that are not along the main direction. To reduce the risk of blurring, these additional taps are handled more carefully. For this reason, CDEF defines major and minor taps. The complete 2-D CDEF filter is represented as follows:

[0201]

[0202] Where D is the damping parameter, S (p) and S (s) These are the strengths of the main tap and the secondary tap, respectively, and round(·) rounds ties away from zero. and Here, f(d, S, D) represents the filter weights, and f(d, S, D) is a constraint function that operates on the difference between the filtered pixel and each of its neighboring pixels. For small differences, f(d, S, D) = d, making the filter function like a linear filter. When the differences are large, f(d, S, D) = 0, which effectively ignores the filter taps.

[0203] Loop recovery in VI.AV1

[0204] In addition to traditional deblocking operations, a set of in-loop recovery schemes can be used in post-encoding deblocking to improve overall denoising and edge quality. These schemes can be switched within a frame for each appropriately sized tile. The specific scheme described is based on separable symmetric Wiener filters and dual self-guided filters with subspace projection. Because content statistics can vary significantly within a frame, these tools are integrated within a switchable frame. Different tools within the switchable frame are triggered in different regions of the frame.

[0205] For a Wiener filter, each pixel in a degraded frame can be reconstructed as a noncausal filtered version of each pixel within a w×w window surrounding that pixel, where w = 2r + 1 is odd for integers r. If the 2D filter taps are in column vector form by w... 2 If the filter parameters are represented by a vector F with 1 × 1 elements, then the direct linear minimum mean square error (LMMSE) optimization makes the filter parameters change from F = H. -1 M is given, where H = E[XX] T ] is the autocovariance of x (the w×w window around the pixel). 2 (a column vectorized version of each sample), and M = E[YX T [ ] represents the cross-correlation between the x to be estimated and the scalar source sample y. The encoder can estimate H and M based on the implementation in the deblocked frame and the source, and send the resulting filter F to the decoder. However, this not only affects the transmission w 2 Tapping incurs a significant bit rate cost and results in non-separable filtering, which makes decoding overly complex. Therefore, several additional constraints are imposed on the properties of F. First, F is constrained to be separable, allowing filtering to be implemented as separable horizontal and vertical w-tapped convolutions. Second, each horizontal and vertical filter is constrained to be symmetric. Third, it is assumed that the sum of both the horizontal and vertical filter coefficients is 1.

[0206] The local linear model of the guided filter can be represented as follows:

[0207] Equation (17) is y = Fx + G.

[0208] A local linear model is used to compute the filtered output y from an unfiltered sample x, where F and G are determined based on statistics of the guiding image in the neighborhood of the degraded image and the filtered pixels. If the guiding image is the same as the degraded image, the resulting so-called self-guided filtering has the effect of preserving smooth edges. The specific form of self-guided filtering depends on two parameters: radius r and noise parameter e, and is listed below.

[0209] (1) Obtain the mean μ and variance σ of the pixels in the (2r+1)×(2r+1) window surrounding each pixel. 2This can be achieved efficiently using box filtering based on integral imaging.

[0210] (2) Calculate for each pixel: f = σ 2 / (σ 2 +e); g=(1-f)μ.

[0211] (3) Calculate the F and G values ​​of each pixel as the average of the f and g values ​​in a 3×3 window around the pixel for use.

[0212] The filter is controlled by r and e, where a higher r means a higher spatial variation and a higher e means a higher range variation.

[0213] Figure 17 An exemplary subspace projection according to an embodiment is shown. Even if neither of the inexpensive recoveries X1 and X2 is close to the source Y, appropriate multipliers {α, β} can make them closer to the source, as long as they are slightly moved in the correct direction.

[0214] VII. Cross-component sample offset filter

[0215] Cross-Component Sample Shift (CCSO) filters are loop filtering methods that reduce distortion in reconstructed samples. In a CCSO filter, given a processed input reconstructed sample for a first color component, an output shift can be derived using a nonlinear mapping, and this output shift can be added to the reconstructed sample for the second color component during the CCSO filter's filtering process. The input reconstructed sample for the first color component lies within the filter's support region (or filter support area).

[0216] Figure 18 An exemplary filter support region according to an embodiment of this disclosure is shown. The filter support region includes four reconstructed samples: p0, p1, p2, and p3. The four input reconstructed samples follow a cross shape in the vertical and horizontal directions. The center sample (denoted by c) in the first color component is located at the same position as the sample to be filtered in the second color component. When processing the input reconstructed samples, the following steps are applied:

[0217] Step 1: First, calculate the incremental values ​​between p0-p3 and c, denoted as m0, m1, m2 and m3.

[0218] Step 2: Further quantize the increment values ​​m0-m3. The quantized values ​​are represented as d0, d1, d2, and d3. Based on the following quantization process, the quantized values ​​can be -1, 0, and 1: (a) if m < -N, then d = -1; (b) if -N <= m <= N, then d = 0; and (c) if m > N, then d = 1, where N is called the quantization step size, and example values ​​for N include 4, 8, 12, and 16.

[0219] Quantization values ​​d0-d3 can be used to identify a combination of nonlinear mappings. For example, Figure 18 The CCSO filter in the example has four filter taps d0-d3, and each filter tap can have one of three quantization values ​​(e.g., -1, 0, and 1), so a total of 3... 4 =81 combinations.

[0220] Figures 19A-19B A table according to an embodiment is shown, listing 81 exemplary combinations of nonlinear mappings in a CCSO filter. The last column of the table represents the output offset value for each combination. The example offset values ​​are integers, such as 0, 1, -1, 3, -3, 5, -5, and -7.

[0221] The final filtering process of the CCSO filter can be applied as follows:

[0222] f′=clip(f+s) Equation (18)

[0223] Where f is the reconstructed sample to be filtered, and s is from Figures 19A-19B The output offset value is retrieved from the table. The filtered sample value f′ is further clipped to a range associated with the bit depth.

[0224] VIII. Improved Cross-Component Sample Offset Filter

[0225] In relevant CCSO filters, the quantization step size is a fixed value when processing input reconstructed samples, which may limit the performance of the CCSO filter. Furthermore, the output offset value for each combination of processed input reconstructed samples is a fixed value, which may further limit the performance of the CCSO filter.

[0226] In this disclosure, a CCSO filter can be defined as a filtering process that uses a reconstructed sample of a first color component of a block as input (e.g., Y, Cb, or Cr) and applies the output to a second color component of the block that is different from the first color component of the block.

[0227] The input to the nonlinear mapping of the CCSO filter is one or more processed reconstructed samples located in the filter support region (or filter support area) of the first color component of the block. A quantization process can be applied while processing the reconstructed samples of the input, and this quantization process can be controlled by a quantization step size N. Once the input reconstructed samples have been processed, the output offset value can be retrieved from the nonlinear mapping of the CCSO filter. The output offset value can then be applied to the second color component of the block.

[0228] This disclosure includes methods for improving the performance of CCSO filters. In this disclosure, a filter support region (or area) is defined as the region where a filtering process such as a CCSO filter is performed. During quantization, the range of possible sample values ​​is decomposed into a finite set of intervals (or steps), which may be called quantization intervals. The difference between two quantization intervals may be called the quantization step size.

[0229] According to various aspects of this disclosure, when processing input reconstructed samples within the filter support region (or area) of a CCSO filter, an adaptive quantization step size (e.g., N as described in Section VII (the section on cross-component sample offset filters) can be used). In some embodiments, the quantization step size can be varied based on attributes of the coded block, video frame, and / or video sequence. Variations in the quantization step size can improve overall encoding / decoding efficiency.

[0230] In one embodiment, the quantization step size N can be a predefined constant integer. Example values ​​for N include 4, 8, 12, and 16. N can vary for different coded blocks, video frames, or video sequences.

[0231] In one embodiment, the quantization step size N can be an integer 2. N That is, powers of 2, such as 2, 4, 8 or 16.

[0232] In one embodiment, the quantization step size N can be signaled. The quantization step size N can be signaled at block-level parameters, slice-level parameters, tile-level parameters, or frame-level parameters.

[0233] In one embodiment, given an array of possible quantization step sizes, an index can be signaled to indicate which quantization step size to select. This signaling can be used to notify the index at block-level parameters, slice-level parameters, plot-level parameters, CTU-level parameters, video frame-level parameters, or sequence-level parameters.

[0234] In one example, the possible array of quantization steps could be {4, 8, 12, 16}, and the indices in [0, 3] could be signaled to indicate which value in the array to select.

[0235] In one example, a fixed-length codeword such as {00, 01, 10, 11} can be used to signal the index.

[0236] In one example, the possible array of quantization steps could be {4, 8, 16}, and the indices in [0, 2] could be signaled to indicate which value in the array to select.

[0237] In one example, variable-length codewords such as {0, 10, 11} or {1, 00, 01} can be used to signal the index.

[0238] In one embodiment, the quantization step size N or the index of N is signaled in the high-level parameter set. High-level parameter sets include, for example, a decoder parameter set (DPS), a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptive parameter set (APS), a slice header, or a picture header.

[0239] According to various aspects of this disclosure, adaptive offset values ​​can be used based on properties of the filter support region. In some embodiments, the offset values ​​derived from the nonlinear mapping of the CCSO filter can be further scaled based on properties of the filter support region.

[0240] In one embodiment, the scaling of the offset value may depend on the increment value (e.g., m0-m3) and / or the quantization step size N, where m0-m3 is the increment value between the surrounding samples p0-p3 and the center sample c, as described in Section VII (the section on cross-component sample offset filters). In one example, the scaling of the offset value may depend on the ratio r between the average of the absolute values ​​of m0-m3 and N.

[0241] In one embodiment, the average of the absolute values ​​of m0-m3 can be calculated first, denoted as m'. The ratio r can be derived as r = m' / N. Then, the offset value derived from the nonlinear mapping can be scaled by the maximum value among 1 and r, such as max(1, m' / N).

[0242] In one example, d0 = 1, d1 = -1, d2 = 0, d3 = 0, and m0 = 8, m1 = -8, m2 = 3, m3 = 1. The quantization step size N is 4, and the output offset s is 3. Therefore, m' = avg(abs(m0) + abs(m1) + abs(m2) + abs(m3)) = avg(8 + 8 + 3 + 1) = 5, where avg and abs represent the averaging operation and the absolute value operation, respectively. The ratio r = m' / N = 5 / 4 = 1, and the final offset is s × r = 3 × 1 = 3.

[0243] In one embodiment, the average value m' can be calculated using a subset of the absolute values ​​of m0-m3, where the corresponding quantized values ​​d0-d3 are non-zero (e.g., if the quantized value d is zero, the corresponding m is excluded from the calculation of m'). The ratio r can be derived as m' / N. The output offset value derived from the nonlinear mapping can then be scaled by the maximum value between 1 and r, such as max(1, m' / N).

[0244] In one example, d0 = 1, d1 = -1, d2 = 0, d3 = 0, and m0 = 8, m1 = -8, m2 = 3, m3 = 1. The quantization step size N is 4, and the output offset s is 3. Therefore, m' = avg(abs(m0) + abs(m1)) = avg(8 + 8) = 8, where avg and abs represent the averaging operation and the absolute value operation, respectively. The ratio r = m' / N = 8 / 4 = 2, and the final offset is s × r = 3 × 2 = 6.

[0245] In one embodiment, adaptive output offset can always be enabled (or turned on). That is, scaling of the output offset can always be used in the CCSO filter.

[0246] In one embodiment, a syntax element indicating whether adaptive output offset values ​​are enabled can be signaled at parameters such as block-level parameters, video frame-level parameters, slice-level parameters, tile-level parameters, CTU-level parameters, or sequence-level parameters. The syntax element can also be signaled in a high-level parameter set (e.g., DPS, VPS, SPS, PPS, or APS), slice header, picture header, or SEI message.

[0247] IX. Flowchart

[0248] Figure 20 A flowchart outlining an exemplary method (2000) according to embodiments of the present disclosure is shown. In various embodiments, the method (2000) is executed by processing circuitry, such as processing circuitry in terminal devices (210), (220), (230), and (240), processing circuitry performing the functions of a video encoder (303), processing circuitry performing the functions of a video decoder (310), processing circuitry performing the functions of a video decoder (410), processing circuitry performing the functions of an intra-frame prediction module (452), processing circuitry performing the functions of a video encoder (503), processing circuitry performing the functions of a predictor (535), processing circuitry performing the functions of an intra-frame encoder (622), processing circuitry performing the functions of an intra-frame decoder (772), and so on. In some embodiments, the method (2000) is implemented with software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes the method (2000).

[0249] The method (2000) generally begins with step (S2010), in which the method (2000) determines a reconstructed sample of the first color component in the filter support region of the current image, which is part of an encoded video sequence. Then, the method (2000) proceeds to step (S2020).

[0250] At step (S2020), the method (2000) performs a filtering process on the reconstructed sample to obtain the output value of the filtering process, wherein the quantization step size of the filtering process is set separately for each of the current coding block, the current image, and the encoded video sequence. Then, the method (2000) proceeds to step (S2030).

[0251] At step (S2030), the method (2000) reconstructs samples of the second color component in the filter support region based on the output value. The second color component is different from the first color component. Then, the method (2000) ends.

[0252] In this embodiment, the quantization step size is a predefined constant for one of the current coding block, the current image, and the encoded video sequence.

[0253] In this embodiment, the quantization step size is an integer that is a power of 2.

[0254] In an embodiment, the quantization step size is one of a plurality of quantization steps, which are determined based on a first index included in the bitstream, the first index corresponding to one of the current coding block, the current image, and the encoded video sequence.

[0255] In an embodiment, the quantization step size is included in the bitstream, and the quantization step size corresponds to one of the current coding block, the current image, and the encoded video sequence.

[0256] In an embodiment, the method (2000) determines whether the output value needs to be scaled based on the reconstructed sample of the first color component. In response to the output value needing to be scaled, the method (2000) reconstructs the sample of the second color component in the filter support region based on the scaled output value.

[0257] In an embodiment, the method (2000) determines whether the output value of the filtering process needs to be scaled based on the reconstructed sample of the first color component and a first syntax element, wherein the first syntax element indicates whether scaling of the output value is enabled.

[0258] In one embodiment, the method (2000) determines the scaled output value of the filtering process based on the reconstructed sample of the first color component in the filter support region and the quantization step size. In one embodiment, to reconstruct a sample of the second color component in the filter support region, the method (2000) may determine whether the output value needs to be scaled; in response to the output value needing to be scaled, scale the output value to obtain the scaled output value; and reconstruct the sample of the second color component in the filter support region based on the scaled output value. In one embodiment, the reconstructed sample is a central reconstructed sample c in the filter support region, the filter support region including four surrounding reconstructed samples p0, p1, p2, and p3 surrounding the central reconstructed sample c, the four surrounding reconstructed samples following a cross shape in the vertical and horizontal directions, and the quantization step size is denoted as N. The step of performing a filtering process on the reconstructed sample to obtain the output value of the filtering process includes: calculating the incremental values ​​m0, m1, m2, and m3 between the surrounding reconstructed samples p0-p3 and the central reconstructed sample c; quantizing the incremental values ​​m0, m1, m2, and m3 into quantized values ​​d0, d1, d2, and d3 respectively based on the quantization step size N; deriving an offset value from a predetermined nonlinear mapping based on the quantized values ​​d0, d1, d2, and d3, and using the offset value as the output value, wherein the nonlinear mapping is used to represent the mapping relationship between the quantized values ​​d0, d1, d2, and d3 and the offset value. In one embodiment, quantizing the incremental values ​​m0, m1, m2, and m3 into quantized values ​​d0, d1, d2, and d3 based on the quantization step size N includes: determining di = -1 when mi < -N, where i is an integer ranging from 0 to 3; determining di = 0 when -N <= mi <= N; and determining d = 1 when mi > N. In one embodiment, the quantization step size N ranges from 2, 4, 8, 12, and 16. In one embodiment, scaling the output value to obtain a scaled output value includes: scaling the output value according to the incremental values ​​m0, m1, m2, and m3 and the quantization step size N to obtain the scaled output value.

[0259] X. Computer System

[0260] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 21 A computer system (2100) is shown, which is adapted to implement certain embodiments of the disclosed subject matter.

[0261] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through decoding, microcode, etc.

[0262] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0263] Figure 21 The components shown for the computer system (2100) are exemplary in nature and are not intended to limit the scope or functionality of the computer software used to implement the embodiments of this application. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of the computer system (2100).

[0264] The computer system (2100) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media, which need not be directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0265] Human-machine interface input devices may include one or more of the following (only one is shown): keyboard (2101), mouse (2102), touchpad (2103), touch screen (2110), data glove (not shown), joystick (2105), microphone (2106), scanner (2107), camera (2108).

[0266] The computer system (2100) may also include certain human-machine interface (HMI) output devices. Such HMI output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such HMI output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (2110), data gloves (not shown), or joystick (2105), but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (2109), headphones (not shown)), visual output devices (e.g., screens (2110) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light-emitting diode screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which may output two-dimensional or more three-dimensional visual outputs by means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown). These visual output devices (e.g., screens (2110)) may be connected to the system bus (2148) via a graphics adapter (2150).

[0267] The computer system (2100) may also include human-accessible storage devices and related media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) (2120) or similar media (2121), thumb drives (2122), removable hard disk drives or solid-state drives (2123), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), and so on.

[0268] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0269] The computer system (2100) may also include a network interface (2154) leading to one or more communication networks (2155). The one or more communication networks (2155) may be wireless, wired, or optical. The one or more communication networks (2155) may also be local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), vehicular and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of the one or more communication networks (2155) include Ethernet, wireless LANs, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicular and industrial networks (including CANbus), etc. Some networks typically require external network interface adapters for connection to certain general-purpose data ports or peripheral buses (2149) (e.g., a USB port on the computer system (2100)); other systems are typically integrated into the core of the computer system (2100) via a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer system (2100) can communicate with other entities. This communication can be unidirectional, for receiving only (e.g., wireless television), unidirectional, for sending only (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as via a local area or wide area digital network to other computer systems. Each of the aforementioned networks and network interfaces can use certain protocols and protocol stacks.

[0270] The aforementioned human-computer interface device, human-accessible storage device, and network interface can be connected to the core (2140) of the computer system (2100).

[0271] The core (2140) may include one or more central processing units (CPU) (2141), graphics processing units (GPUs) (2142), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (2143), task-specific hardware accelerators (2144), etc. These devices, along with read-only memory (ROM) (2145), random access memory (2146), internal mass storage (e.g., internal non-user-accessible hard disk drives, solid-state drives, etc.) (2147), etc., can be connected via a system bus (2148). In some computer systems, the system bus (2148) may be accessed as one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (2148) or connected via a peripheral bus (2149). In one example, a screen (2110) may be connected to a graphics adapter (2150). Peripheral bus architectures include external controller interfaces (PCI), universal serial buses (USB), etc.

[0272] The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (2145) or RAM (2146). Transient data can also be stored in RAM (2146), while permanent data can be stored, for example, in internal mass storage (2147). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage (2147), ROM (2145), RAM (2146), etc.

[0273] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this application, or they may be media and code well-known and usable by those skilled in the art of computer software.

[0274] By way of example and not limitation, a computer system having an architecture (2100), particularly a core (2140), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the non-volatile core (2140), such as internal mass storage (2147) or ROM (2145). Software implementing various embodiments of this application can be stored in such a device and executed by the core (2140). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (2140), particularly the processor therein (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (2146) and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide logic hardwired or otherwise incorporated into circuitry (e.g., an accelerator (2144)) that may replace or operate with the software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing the execution of software (such as an integrated circuit (IC)), circuitry containing execution logic, or both. This application includes any suitable combination of hardware and software.

[0275] While this application has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are all within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are thus within its spirit and scope.

[0276] Appendix A: Acronyms

[0277] ALF: Adaptive Loop Filter

[0278] AMVP: Advanced Motion Vector Prediction

[0279] APS: Adaptive Parameter Set

[0280] ASIC: Application-Specific Integrated Circuit

[0281] ATMVP: Alternative / Advanced Temporal Motion Vector Prediction

[0282] AV1: Open Media Alliance Video 1 (AOMedia Video 1)

[0283] AV2: AOMedia Video 2

[0284] BMS: Benchmark set

[0285] BV: Block Vector

[0286] CANBus: Controller Area Network Bus

[0287] CB: Coding Block

[0288] CC-ALF: Cross-Component Adaptive Loop Filter

[0289] CD: Compact Disc

[0290] CDEF: Constrained Directional Enhancement Filter

[0291] CPR: Current Picture Referencing

[0292] CPU: Central Processing Unit

[0293] CRT: Cathode Ray Tube

[0294] CTB: Coding Tree Block

[0295] CTU: Coding Tree Unit

[0296] CU: Coding Unit

[0297] DPB: Decoder Picture Buffer

[0298] DPCM: Differential Pulse Code Modulation

[0299] DPS: Decoding Parameter Set

[0300] DVD: Digital Video Disc

[0301] FPGA: Field Programmable Gate Array

[0302] JCCR: Joint CbCr Residual Coding

[0303] JVET: Joint Video Exploration Team

[0304] GOP: Group of Pictures

[0305] GPU: Graphics Processing Unit

[0306] GSM: Global System for Mobile Communications

[0307] HDR: High Dynamic Range

[0308] HEVC: High Efficiency Video Coding

[0309] HRD: Hypothetical Reference Decoder

[0310] IBC: Intra Block Copy

[0311] IC: Integrated Circuit

[0312] ISP: Intra Sub-Partitions

[0313] JEM: Joint Exploration Model

[0314] LAN: Local Area Network

[0315] LCD: Liquid Crystal Display

[0316] LR: Loop Restoration Filter

[0317] LRU: Loop Restoration Unit

[0318] LTE: Long-Term Evolution

[0319] MPM: Most Probable Mode

[0320] MV: Motion Vector

[0321] OLED: Organic Light-Emitting Diode

[0322] PBs: Prediction Blocks

[0323] PCI: Peripheral Component Interconnect

[0324] PDPC: Position Determined Prediction Combination

[0325] PLD: Programmable Logic Device

[0326] PPS: Picture Parameter Set

[0327] PU: Prediction Unit

[0328] RAM: Random Access Memory

[0329] ROM: Read-Only Memory

[0330] SAO: Sample Adaptive Offset

[0331] SCC: Screen Content Coding

[0332] SDR: Standard Dynamic Range

[0333] SEI: Supplementary Enhancement Information

[0334] SNR: Signal-to-Noise Ratio

[0335] SPS: Sequence Parameter Set

[0336] SSD: Solid-state drive

[0337] TU: Transform Unit

[0338] USB: Universal Serial Bus

[0339] VPS: Video Parameter Set

[0340] VUI: Video Usability Information

[0341] VVC: Versatile Video Coding

[0342] WAIP: Wide-Angle Intra Prediction

Claims

1. A method for video decoding, characterized in that, include: Determine the reconstructed sample of the first color component in the filter support region of the current image, which is part of an encoded video sequence; The reconstructed samples of the first color component are subjected to a cross-component sample offset (CCSO) filtering process, wherein the quantization step size of the filtering process is set separately for each coding block, each video frame, or each video sequence. as well as Based on the output offset value obtained by performing a cross component sample offset (CCSO) filtering process on the reconstructed sample of the first color component, a sample of the second color component in the filter support region is reconstructed, wherein the second color component is different from the first color component.

2. The method according to claim 1, wherein, The quantization step size is a predefined constant for the corresponding coded block, the corresponding video frame, or the corresponding video sequence.

3. The method according to claim 1, wherein, The quantization step size is an integer that is a power of 2.

4. The method according to claim 1, wherein, The quantization step size is one of a plurality of quantization step sizes, which are determined based on a first index included in the bitstream, the first index corresponding to a corresponding coded block, a corresponding video frame, or a corresponding video sequence.

5. The method according to claim 1, wherein, The quantization step size is included in the bitstream, and the quantization step size corresponds to the corresponding coding block, the corresponding video frame, or the corresponding video sequence.

6. The method according to claim 1, wherein, The process of reconstructing the samples of the second color component in the filter support region includes: Based on the reconstructed sample of the first color component, determine whether the output offset value needs to be scaled; and In response to the output offset value being scaled, Based on the scaled output offset value, the sample of the second color component in the filter support region is reconstructed.

7. The method according to claim 6, wherein, Determining whether the output offset value needs to be scaled includes: determining whether the output offset value of the filtering process needs to be scaled based on the reconstructed sample of the first color component and the first syntax element, wherein the first syntax element indicates whether scaling of the output offset value is enabled.

8. The method according to claim 6, wherein, The process of reconstructing samples of the second color component in the filter support region further includes: determining the scaled output offset value based on the reconstructed samples of the first color component and the quantization step size.

9. The method according to claim 1, wherein, The process of reconstructing the samples of the second color component in the filter support region includes: Determine whether the output offset value needs to be scaled; In response to the output offset value being scaled, the output offset value is scaled to obtain the scaled output offset value; Based on the scaled output offset value, the sample of the second color component in the filter support region is reconstructed.

10. The method according to claim 1, wherein, The reconstructed sample is the central reconstructed sample c located in the filter support region. The filter support region includes four surrounding reconstructed samples p0, p1, p2, and p3 surrounding the central reconstructed sample c. The four surrounding reconstructed samples follow a cross shape in the vertical and horizontal directions. The quantization step size is represented by N. The step of performing a filtering process on the reconstructed sample to obtain the output offset value of the filtering process includes: Calculate the incremental values ​​m0, m1, m2, and m3 between the surrounding reconstructed samples p0-p3 and the central reconstructed sample c; Based on the quantization step size N, the incremental values ​​m0, m1, m2 and m3 are quantized into quantized values ​​d0, d1, d2 and d3 respectively; Based on the quantization values ​​d0, d1, d2, and d3, offset values ​​are derived from a predetermined nonlinear mapping, and these offset values ​​are used as the output offset values. The nonlinear mapping is used to represent the mapping relationship between the quantization values ​​d0, d1, d2, and d3 and the offset values.

11. The method according to claim 10, wherein, The step of quantizing the incremental values ​​m0, m1, m2, and m3 into quantized values ​​d0, d1, d2, and d3 based on the quantization step size N includes: When mi < -N, then di = -1 is determined, where i takes the value of an integer from 0 to 3; When -N <= mi <= N, determine that di = 0; When mi > N, d = 1 is determined.

12. The method according to claim 10, wherein, The quantization step size N can be 2, 4, 8, 12, or 16.

13. The method according to claim 9, wherein, The reconstructed sample is the central reconstructed sample c located in the filter support region. The filter support region includes four surrounding reconstructed samples p0, p1, p2, and p3 surrounding the central reconstructed sample c. The four surrounding reconstructed samples follow a cross shape in the vertical and horizontal directions. The quantization step size is represented by N. The step of scaling the output offset value in response to the need for scaling the output offset value to obtain the scaled output offset value includes: The output offset value is scaled according to the increment values ​​m0, m1, m2 and m3 and the quantization step size N to obtain the scaled output offset value, wherein the increment values ​​m0, m1, m2 and m3 represent the increment values ​​between the surrounding reconstructed samples p0-p3 and the central reconstructed sample c, respectively.

14. A video decoding apparatus, characterized in that, Includes a processing circuit, which is configured to perform the method as described in any one of claims 1-13.

15. A video encoding method, characterized in that, include: Determine the reconstructed sample of the first color component in the filter-supported region of the current image, which is part of a video sequence; Determine the quantization step size for the Cross Component Sample Offset (CCSO) filtering process corresponding to the filter support region, wherein the quantization step size is set individually for each coding block, each video frame, or each video sequence, and the quantization step size is used to perform the Cross Component Sample Offset (CCSO) filtering process on the reconstructed samples of the first color component; and Encoding the current image to generate an encoded video stream, wherein the determined quantization step size is signaled in the encoded video stream and is used to: obtain an output offset value by performing a cross component sample offset (CCSO) filtering process on the reconstructed samples of the first color component, and reconstruct samples of the second color component in the filter support region, the second color component being different from the first color component.

16. A non-volatile computer-readable storage medium storing instructions, having stored thereon a computer program / instructions and a video stream, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the video encoding method of claim 15 to generate the video stream.

17. A method for storing video streams, characterized in that, The video encoding method of claim 15 is used to generate a video stream and store the video stream.

18. A method for transmitting a video stream, characterized in that, The video encoding method of claim 15 is used to generate a video stream and transmit the video stream.

Images