Video decoding and coding method and device, video code stream processing method and device and storage medium

Abstract

The embodiment of the invention provides a video decoding method, a video coding method, a video code stream processing method, a video coding device and a storage medium, and the method comprises the steps: decoding the coding information of a reconstruction sample of a current component in a current image according to a coded video code stream, and enabling the coding information to indicate a sample offset filter to be applied to the reconstruction sample, the filtering shape of the sample offset filter comprises a first reconstruction sample of a first component in the current image; determining an output value of a sample offset filter based on a first reconstructed sample in a filtering shape indicated by the encoding information; and determining a filtered sample value of the reconstructed sample of the current component based on the reconstructed sample and an output value of the sample offset filter.

Description

[0001] This application is a divisional application of an actively modified version of a patent application with an application date of October 8, 2021, Chinese patent application number 202180027129.X, and invention name “Method and device for video decoding”. Technical Field

[0002] The present application describes methods and apparatus generally related to video decoding and encoding. Background Art

[0003] The background description provided herein is for the purpose of presenting the content of the present application in general. To the extent described in this background section, the works of the currently named inventors and aspects of this description that may not otherwise qualify as prior art at the time of filing are neither explicitly nor implicitly admitted as prior art to the present application.

[0004] Video encoding and decoding can be performed using inter-frame image prediction with motion compensation. Uncompressed digital video can include a series of images, each image having spatial dimensions of, for example, 1920x1080 luma samples and associated chroma samples. The series of images can have a fixed or variable image rate (also informally referred to as a frame rate), such as 60 images per second or 60Hz. Uncompressed video has very high bit rate requirements. For example, at 8 bits per sample, 1080p60 4:2:0 video (with a resolution of 1920x1080 luma samples at a 60Hz frame rate) requires a bandwidth of nearly 1.5Gbit / s. One hour of such video requires more than 600 gigabytes of storage space.

[0005] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal by compression. Compression can help reduce the above-mentioned bandwidth and / or storage space requirements, in some cases by two orders of magnitude or more. Lossless compression and lossy compression and their combinations can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal usable for the intended application. In the case of video, lossy compression is widely used. The amount of tolerable distortion depends on the application, for example, users of some consumer streaming applications can tolerate higher distortion than users of television distribution applications. The achievable compression rate can reflect that higher allowable / tolerable distortion can produce higher compression rates.

[0006] Video encoders and decoders may employ techniques from several broad categories including, for example, motion compensation, transforms, quantization, and entropy coding.

[0007] Video codec techniques may include a variety of techniques known as intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference image. In some video codecs, an image is spatially subdivided into blocks of samples. When all sample blocks are encoded in intra-frame mode, the image may be an intra-frame image. Intra-frame images and their derivatives (e.g., independent decoder refresh images) may be used to reset the decoder state, and may therefore be used as the first image in an encoded video bitstream and video session, or as a static image. The samples of the intra-frame block may be transformed, and the transform coefficients may be quantized before entropy coding. Intra-frame prediction may be a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficient, the fewer bits are needed to represent the block after entropy coding at a given quantization step size.

[0008] Conventional intra-frame coding (e.g., known from coding techniques such as MPEG-2) does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to use intra-frame prediction based on, for example, surrounding sample data and / or metadata obtained during the encoding / decoding process of spatially adjacent and earlier data blocks in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. It is worth noting that, at least in some cases, intra-frame prediction uses only reference data from the current image being reconstructed, not reference images.

[0009] There can be many different forms of intra-frame prediction. When more than one such technique is used in a given video coding technique, the techniques used can be encoded in intra-frame prediction modes. In some cases, a mode can have multiple sub-modes and / or multiple parameters, and these sub-modes and parameters can be encoded separately or included in the mode codeword. Which codeword is used for a given mode / sub-mode / parameter combination can have an impact on the coding efficiency gain through intra-frame prediction, and the entropy coding techniques used to convert the codewords into bitstreams are similar.

[0010] An intra prediction mode was introduced in H.264, improved in H.265, and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). The prediction block can be formed using the values ​​of neighboring samples that belong to samples that are already available. Sample values ​​of neighboring samples are copied to the prediction block according to the direction. The reference to the direction in use can be encoded in the bitstream or it can itself be predicted.

[0011] refer to Figure 1A ,exist Figure 1AThe lower right side of the diagram depicts a subset of 9 known prediction directions from the 33 possible prediction directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction of the predicted sample. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples to the upper right at an angle of 45 degrees to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples to the lower left of sample (101) at an angle of 22.5 degrees to the horizontal.

[0012] Still reference Figure 1A ,exist Figure 1A A square block (104) of 4×4 samples is depicted at the upper left of (indicated by the dashed bold line). The square block (104) includes 16 samples, each sample labeled "S", its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top side) and the first sample in the X dimension (starting from the left side). Similarly, sample S44 is the fourth sample in the Y dimension and the X dimension in the block (104). Since the size of the block is 4×4 samples, sample S44 is located at the lower right. Reference samples following a similar numbering scheme are also shown. The reference samples are labeled R, which are labeled with a Y position (e.g., row index) and an X position (column index) relative to the block (104). In H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; therefore, there is no need to use negative values.

[0013] Intra-frame prediction works by copying reference sample values ​​from neighboring samples occupied by a signaled prediction direction. For example, assume that the encoded video bitstream includes signaling indicating a prediction direction consistent with arrow (102) for the block, i.e., the sample is predicted based on one or more prediction samples to the upper right at an angle of 45 degrees to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0014] In some cases, especially when the prediction direction is not divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding technology has developed, the number of possible prediction directions has increased. In H.264 (2003), 9 different prediction directions can be represented. This was increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions when disclosed. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those possible directions with a small number of bits, thereby accepting a certain penalty for less likely directions. In addition, the direction itself can sometimes be predicted based on the neighboring directions used in adjacent decoded blocks.

[0016] Figure 1B A schematic diagram (180) depicting 65 intra prediction directions according to JEM is shown to illustrate that the number of prediction directions increases over time.

[0017] The mapping of the intra-prediction direction bit representing the direction in the coded video bitstream can vary depending on the video coding technique; and the mapping of the intra-prediction direction bit can range from, for example, a simple direct mapping from prediction direction to intra-prediction mode, to codewords, to complex adaptive schemes involving the most probable mode (and similar techniques). However, in all cases, some directions are statistically less likely to appear in the video content than other directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, those unlikely directions will need to be represented using more bits than more likely directions.

[0018] Motion compensation may be a lossy compression technique, and may involve the following technique: a block of sample data from a previously reconstructed image or portion thereof (reference image) is used to predict a newly reconstructed image or image portion after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference image may be the same as the image currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third dimension indicating the reference image being used (the latter may indirectly be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to sample data of certain regions can be predicted based on other MVs (e.g., based on sample data of another region that is spatially adjacent to the region being reconstructed and those MVs that precede the MV in decoding order). Doing so can greatly reduce the amount of data required to encode the MV, thereby eliminating redundancy and increasing the compression rate. MV prediction can work effectively, for example, because when encoding an input video signal obtained from a camera (called natural video), there is a statistical probability that a larger area than the area applicable to a single MV moves in a similar direction, so in some cases, similar motion vectors derived from MVs of adjacent regions can be used to predict the larger area. This makes the MV found for a given region similar or identical to the MV predicted from the surrounding MVs, and then after entropy coding, the MV found for the given region can be represented with fewer bits than the number of bits used when the MV is directly encoded. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself can be lossy, for example due to rounding errors when calculating the predicted value based on multiple surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T H.265 Recommendation, "High Efficiency Video Coding", December 2016). Among the various MV prediction mechanisms provided by H.265, this application describes a technique referred to below as "spatial merging".

[0021] Referring to Figure 1, the current block (201) includes samples that have been discovered by the encoder during the motion search process, and the samples can be predicted based on the previous block of the same size that has generated a spatial offset. In addition, the MV can be derived from metadata associated with one or more reference images instead of encoding the MV directly. For example, the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206, respectively) is used to derive the MV from the metadata of the nearest reference image (in decoding order). In H.265, MV prediction can use the prediction value of the same reference image that the neighboring blocks are also using. Summary of the invention

[0022] The embodiments of the present application provide a method, device and storage medium for video decoding, encoding and processing of video code streams.

[0023] In some embodiments, an embodiment of the present application provides a method for video decoding, the method comprising:

[0024] Decoding, according to the encoded video bitstream, encoding information of a reconstructed sample of a current component in a current image, the encoding information indicating a sample offset filter to be applied to the reconstructed sample, the filter shape of the sample offset filter including a first reconstructed sample of a first component in the current image;

[0025] determining an output value of the sample offset filter based on a first reconstructed sample in the filter shape indicated by the encoding information; and

[0026] Based on the reconstructed samples and an output value of the sample offset filter, filtered sample values ​​of the reconstructed samples of the current component are determined.

[0027] In some embodiments, the present application provides a device for video decoding, including:

[0028] A processing circuit, wherein the processing circuit is configured to execute the video decoding method provided in an embodiment of the present application.

[0029] In some embodiments, the present application provides a video encoding method, the method comprising:

[0030] determining a filter shape to be applied to reconstructed samples of a current component in a current picture from a plurality of filter shapes of a sample offset filter, the filter shape of the sample offset filter comprising a first reconstructed sample of a first component in the current picture;

[0031] A video code stream is generated based on samples of the current picture and the multiple filter shapes, the video code stream indicating encoding information of the filter shape of the sample offset filter applied to the reconstructed samples of the current component, the first reconstructed sample in the filter shape indicated by the encoding information is used to determine the output value of the sample offset filter; and the output value is used to determine the filtered sample value of the reconstructed sample of the current component.

[0032] In some embodiments, an embodiment of the present application provides a device for video encoding, including: a processing circuit, wherein the processing circuit is configured to execute the video encoding method provided by the embodiment of the present application.

[0033] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores instructions. When the instructions are executed by a computer for video decoding, the computer executes the video decoding method provided by the embodiment of the present application to decode the video code stream, or the computer executes the video encoding method provided by the embodiment of the present application to generate a video code stream.

[0034] In some embodiments, an embodiment of the present application provides a method for processing a video stream, and the video stream can be decoded based on the video decoding method provided by the embodiment of the present application, or the video stream can be generated based on the video encoding method provided by the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Further features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0036] Figure 1A It is a schematic diagram of an exemplary subset of intra-frame prediction modes according to an embodiment of the present application.

[0037] Figure 1B It is a schematic diagram of an exemplary intra-frame prediction direction according to an embodiment of the present application.

[0038] Figure 2 This is a schematic diagram of spatial merging candidates of a current block and its surroundings according to an example of an embodiment of the present application.

[0039] Figure 3 It is a schematic diagram of a simplified block diagram of a communication system according to one embodiment of the present application.

[0040] Figure 4 It is a schematic diagram of a simplified block diagram of a communication system according to another embodiment of the present application.

[0041] Figure 5 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment of the present application.

[0042] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment of the present application.

[0043] Figure 7 A block diagram of an encoder according to another embodiment of the present application is shown.

[0044] Figure 8 A block diagram of a decoder according to another embodiment of the present application is shown.

[0045] Fig. 9 An exemplary adaptive loop filter according to an embodiment of the present application is shown.

[0046] Figures 10A to 10D An example of sub-sampling positions for calculating gradients in a vertical direction, a horizontal direction, and two diagonal directions, respectively, according to an embodiment of the present application is shown.

[0047] Fig.11A and 11B FIG. 4 shows modified block classification at a virtual boundary according to an embodiment of the present application.

[0048] Figures 12A to 12F An exemplary adaptive loop filter with padding operation at various virtual boundaries according to an embodiment of the present application is shown.

[0049] Fig.13 An example of image quadtree splitting with maximum coding unit alignment according to an embodiment of the present application is shown.

[0050] Fig.14 The embodiment corresponding to the embodiment of the present application is shown Fig.13 The quadtree splitting pattern.

[0051] Fig.15 A cross-component filter for generating a chroma component according to an embodiment of the present application is shown.

[0052] Fig.16 An example of a filter according to an embodiment of the present application is shown.

[0053] Fig.17A and Fig. 17B FIG. 4 shows exemplary positions of chroma samples relative to luma samples according to an embodiment of the present application.

[0054] Fig.18 An example of directional search of blocks according to an embodiment of the present application is shown.

[0055] Fig.19 An example of subspace projection according to an embodiment of the present application is shown.

[0056] Fig. 20 An example of a filter support region in a cross-component sample offset (CCSO) filter according to an embodiment of the present application is shown.

[0057] Figures 21A to 21C An exemplary mapping for a CCSO filter according to an embodiment of the present application is shown.

[0058] Fig. 22 An example implementation of CCSO according to an embodiment of the present application is shown.

[0059] Fig.23 Four patterns of pixel classification in edge offset according to an embodiment of the present application are shown.

[0060] Fig.24 Several exemplary filter shapes used in the sample shift filter according to an embodiment of the present application are shown.

[0061] Fig.25Several exemplary filter shapes used in the sample shift filter according to an embodiment of the present application are shown.

[0062] Fig.26 Several exemplary filter shapes used in the sample shift filter according to an embodiment of the present application are shown.

[0063] Fig. 27 Several exemplary filter shapes used in the sample shift filter according to an embodiment of the present application are shown.

[0064] Fig.28 Several exemplary filter shapes used in the sample shift filter according to an embodiment of the present application are shown.

[0065] Fig.29 Several exemplary filter shapes used in the sample shift filter according to an embodiment of the present application are shown.

[0066] Fig.30 An example is shown in which one or more filter taps are moved according to an embodiment of the present application.

[0067] Fig.31 A flow chart showing a process according to one embodiment of the present application.

[0068] Fig.32 A flowchart showing a process according to another embodiment of the present application.

[0069] Fig.33 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION

[0070] Figure 3 3 is a simplified block diagram of a communication system (300) according to an embodiment disclosed in the present application. The communication system (300) includes a plurality of terminal devices, which can communicate with each other through, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected through the network (350). Figure 3 In an embodiment of the present invention, the first terminal device (310) and the second terminal device (320) perform unidirectional data transmission. For example, the first terminal device (310) may encode video data (e.g., a video image stream captured by the terminal device (310)) for transmission to the second terminal device (320) via the network (350). The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video data, and display the video image based on the restored video data. Unidirectional data transmission is more common in applications such as media services.

[0071] In another embodiment, the communication system (300) includes a third terminal device (330) and a fourth terminal device (340) that perform bidirectional transmission of encoded video data, which bidirectional transmission may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (330) and the fourth terminal device (340) may encode video data (e.g., a video image stream captured by the terminal device) to transmit to the other of the third terminal device (330) and the fourth terminal device (340) through a network (350). Each of the third terminal device (330) and the fourth terminal device (340) may also receive the encoded video data transmitted by the other of the third terminal device (330) and the fourth terminal device (340), and may decode the encoded video data to restore the video data, and may display the video image on an accessible display device based on the restored video data.

[0072] exist Figure 3 In the embodiment of the present invention, the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340) may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (350) represents any number of networks that transmit encoded video data between the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purpose of the present application, unless explained below, the architecture and topology of the network (350) may be irrelevant to the operation disclosed in the present application.

[0073] As an example, Figure 4 The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0074] The streaming system may include an acquisition subsystem (413), which may include a video source (401) such as a digital camera, which creates an uncompressed video image stream (402). In an embodiment, the video image stream (402) includes samples captured by a digital camera. Compared to the encoded video data (404) (or the encoded video code stream), the video image stream (402) is depicted as a thick line to emphasize the high data volume of the video image stream, and the video image stream (402) can be processed by an electronic device (420), which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video bitstream (404)), which can be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as Figure 3 The client subsystem (406) and the client subsystem (408) in the streaming server (405) can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an output video image stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0075] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0076] Figure 5is a block diagram of a video decoder (510) according to an embodiment disclosed in the present application. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 3 A video decoder (410) of an embodiment.

[0077] A receiver (531) may receive one or more encoded video sequences to be decoded by a video decoder (510); in the same or another embodiment, one encoded video sequence is decoded at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not shown). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and an entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be provided external to the video decoder (510) (not shown). In other cases, a buffer memory (not shown) is provided outside the video decoder (510) to prevent network jitter, for example, and another buffer memory (515) may be configured inside the video decoder (510) to handle broadcast timing, for example. When the receiver (531) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (515), or the buffer memory may be made smaller. Of course, in order to use on a service packet network such as the Internet, a buffer memory (515) may also be required, and the buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or a similar element (not shown) outside the video decoder (510).

[0078] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The types of symbols include information used to manage the operation of the video decoder (510) and potential information used to control a display device such as a display device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 4As shown in . The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0079] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0080] Depending on the type of coded video picture or part of coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser (520) and the multiple units below is not described.

[0081] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0082] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as symbols (521) from the parser (520) and control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) can output a block including sample values, which can be input into the aggregator (555).

[0083] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed image, but may use predictive information from a previously reconstructed portion of a current image. Such predictive information may be provided by an intra-image prediction unit (552). In some cases, the intra-image prediction unit (552) generates surrounding blocks of the same size and shape as the block being reconstructed using reconstructed information extracted from a current image buffer (558). For example, the current image buffer (558) buffers a partially reconstructed current image and / or a fully reconstructed current image. In some cases, the aggregator (555) adds the prediction information generated by the intra-frame prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0084] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference image memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbol (521), these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signals) by the aggregator (555) to generate output sample information. The acquisition of the predicted samples by the motion compensated prediction unit (553) from the address in the reference image memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbol (521), which symbol (521) includes, for example, X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0085] The output samples of the aggregator (555) may be used by various loop filtering techniques in a loop filter unit (556). The video compression technique may include a loop filter technique that is controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and that are available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression technique may also be responsive to meta information obtained during decoding of a previous (in decoding order) portion of the encoded image or encoded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0086] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0087] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting reconstruction of a subsequent coded picture.

[0088] The video decoder (510) may perform decoding operations according to a predetermined video compression technology, such as in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference image size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata of the HRD buffer management signaled in the encoded video sequence.

[0089] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0090] Figure 6 6 is a block diagram of a video encoder (603) according to an embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (603) in an embodiment.

[0091] The video encoder (603) can be used to obtain the video source (601) (not Figure 6 In another embodiment, the video source (601) is a part of the electronic device (620) to receive video samples, and the video source can collect video images to be encoded by the video encoder (603). In another embodiment, the video source (601) is a part of the electronic device (620).

[0092] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate images that are given motion when viewed sequentially. The image itself may be constructed as a spatial pixel array, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0093] According to an embodiment, the video encoder (603) may encode and compress the images of the source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, the coupling is not marked in the figure. The parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, lambda value of rate distortion optimization technology, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0094] In some embodiments, the video encoder (603) operates in an encoding loop. As a simple description, in embodiments, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input image to be encoded and a reference image) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression technology considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference image memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference image synchronicity (and the drift that occurs when synchronicity cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0095] The operation of the "local" decoder (633) can be combined with the above Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0096] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operation. The description of encoder technology can be simplified because encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only needed in certain areas and is provided below.

[0097] During operation, in some embodiments, the source encoder (630) may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes an input image with reference to one or more previously encoded images from a video sequence designated as "reference images." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of a reference image that may be selected as a prediction reference for the input image.

[0098] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 6 When the video sequence is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0099] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new image to be encoded, the predictor (635) may search the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new image. The predictor (635) may operate on a pixel-by-pixel-block basis based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input image may have prediction references taken from a plurality of reference images stored in the reference image memory (634).

[0100] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0101] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0102] The transmitter (640) may buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0103] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0104] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow for different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0105] Predictive pictures (P pictures), which may be pictures that may be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0106] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0107] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8 or 16×16 samples) and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, a block of an I picture may be non-predictively coded, or the block may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. A block of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. A block of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0108] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0109] In an embodiment, the transmitter (640) may transmit additional data when transmitting the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include time / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0110] The captured video may be taken as a plurality of source images (video images) in a temporal sequence. Intra-image prediction (often simplified to intra-frame prediction) exploits spatial correlations in a given image, while inter-image prediction exploits (temporal or other) correlations between images. In an embodiment, a particular image being encoded / decoded is divided into blocks, and the particular image being encoded / decoded is referred to as the current image. When a block in the current image is similar to a reference block in a reference image that was previously encoded in the video and is still buffered, the block in the current image may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference image, and in the case where multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0111] In some embodiments, bidirectional prediction techniques may be used in inter-frame image prediction. According to the bidirectional prediction technique, two reference images are used, for example, a first reference image and a second reference image, both of which are before the current image in the video in decoding order (but may be in the past and future in display order, respectively). A block in the current image may be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block may be predicted by a combination of the first reference block and the second reference block.

[0112] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0113] According to some embodiments disclosed in the present application, predictions such as inter-frame image prediction and intra-frame image prediction are performed in units of blocks. For example, according to the HEVC standard, an image in a video image sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the image have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type for the CU, such as an inter-frame prediction type or an intra-frame prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Taking the luminance prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luminance values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0114] Figure 7 is a diagram of a video encoder (703) according to another embodiment disclosed in the present application. The video encoder (703) is used to receive a processing block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence, and encode the processing block into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used to replace Figure 4A video encoder (403) in an embodiment.

[0115] In an HEVC embodiment, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to use intra mode, inter mode, or bidirectional prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded image; and when encoding the processing block in inter mode or bidirectional prediction mode, the video encoder (703) may use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded image, respectively. In some video encoding techniques, the merge mode may be an inter-image prediction submode, in which a motion vector is derived from one or more motion vector predictors without the aid of an encoded motion vector component external to the predictor. In some other video encoding techniques, there may be a motion vector component applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0116] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes: Figure 7 Shown are an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0117] The inter-frame encoder (730) is used to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in a previous image and a subsequent image), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference image is a decoded reference image decoded based on the encoded video information.

[0118] The intra-frame encoder (722) is used to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same image in some cases, generate quantization coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information according to one or more intra-frame coding techniques). In an embodiment, the intra-frame encoder (722) also calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same image.

[0119] The general controller (721) is used to determine the general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select the intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select the inter prediction information and add the inter prediction information to the bitstream.

[0120] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to operate based on the residual data to encode the residual data to generate a transform coefficient. In an embodiment, the residual encoder (724) is used to convert the residual data from the time domain to the frequency domain and generate a transform coefficient. The transform coefficient is then processed by quantization to obtain a quantized transform coefficient. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and the intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

[0121] The entropy encoder (725) is used to format the code stream to produce an encoded block. The entropy encoder (725) generates various information according to a suitable standard such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (such as intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. It should be noted that according to the disclosed subject matter, when the block is encoded in the inter-frame mode or the merge sub-mode of the bidirectional prediction mode, there is no residual information.

[0122] Figure 8FIG. 8 is a diagram of a video decoder (810) according to another embodiment disclosed in the present application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In an embodiment, the video decoder (810) is used to replace Figure 4 A video decoder (410) of an embodiment.

[0123] exist Figure 8 In one embodiment, the video decoder (810) includes: Figure 7 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874) and an intra-frame decoder (872) coupled together are shown in FIG.

[0124] The entropy decoder (871) can be used to reconstruct certain symbols from the encoded image, which represent syntax elements that constitute the encoded image. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata for prediction by the intra-frame decoder (872) or the inter-frame decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be inverse quantized and provided to the residual decoder (873).

[0125] The inter-frame decoder (880) is used to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0126] The intra-frame decoder (872) is used to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0127] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to obtain quantizer parameters (QP)), and this information can be provided by the entropy decoder (871) (the data path is not indicated because this is only low-volume control information).

[0128] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which can be part of the reconstructed image, which in turn can be part of the reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve visual quality.

[0129] It should be noted that the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using any suitable technology. In an embodiment, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using one or more processors that execute software instructions.

[0130] Embodiments of the present application include a set of video coding techniques, such as low-complexity filters or filter shapes for sample offset. For example, a low-complexity filter shape for a sample offset filter is disclosed herein.

[0131] The video coding format may include any suitable video coding format, for example, an open video coding format designed for video transmission over the Internet, such as Alliance for Open Media (AOMedia) Video 1 (AV1), or a next-generation AOMedia video format beyond AV1. The video coding standard may include the High Efficiency Video Coding (HEVC) standard, the next-generation video coding beyond HEVC (e.g., Versatile Video Coding (VVC)), and the like.

[0132] The encoder / decoder may apply an adaptive loop filter (ALF) with block-based filter adaptation to reduce artifacts. For the luma component, for example, one of multiple filters (e.g., 25 filters) may be selected for a 4×4 luma block based on the direction and activity of the local gradient.

[0133] The ALF may have any suitable shape and size. Fig. 9, ALF (910) to (911) has a shape of a rhombus, for example, ALF (910) has a shape of a 5×5 rhombus, and ALF (911) has a shape of a 7×7 rhombus. In ALF (910), elements (920) to (932) can be used for the filtering process and form a rhombus. Seven values ​​(e.g., C0 to C6) can be used for elements (920) to (932). In ALF (911), elements (940) to (964) can be used for the filtering process and form a rhombus. 13 values ​​(e.g., C0 to C12) can be used for elements (940) to (964).

[0134] Reference Fig. 9 In some examples, two ALFs (910) to (911) having diamond-shaped filter shapes are used. A 5×5 diamond filter (910) may be applied to chroma components (e.g., chroma blocks, chroma CBs), and a 7×7 diamond filter (911) may be applied to luma components (e.g., luma blocks, luma CBs). Other suitable shapes and sizes may be used in the ALFs. For example, a 9×9 diamond filter may be used.

[0135] The filter coefficients at the positions indicated by the values ​​(e.g., C0 to C6 in (910) or C0 to C12 in (920)) may be non-zero. In addition, when the ALF includes a clipping function, the clipping values ​​at these positions may be non-zero.

[0136] For block classification of the luma component, a 4×4 block (or luma block, luma CB) can be divided or classified into one of a plurality of (e.g., 25) categories. The quantized values ​​of the directionality parameter D and the activity value A can be used as the basis. The classification index C is derived using equation (1).

[0137]

[0138] In order to calculate the directivity parameter D and the quantization value The gradient g in the vertical direction, horizontal direction, and two diagonal directions (e.g., d1 and d2) can be calculated using the following one-dimensional Laplace method: v , g h , g d1 and g d2 .

[0139]

[0140]

[0141]

[0142]

[0143] Where the indices i and j refer to the coordinates of the top left sample within the 4×4 block, and R(k, l) represents the reconstructed sample at the coordinates (k, l). The directions (eg, d1 and d2) may refer to two diagonal directions.

[0144] In order to reduce the complexity of the above block classification, a sub-sampled one-dimensional Laplace calculation can be applied. Figures 10A to 10D Shown are used to calculate the vertical direction ( Fig. 10A ), horizontal direction( Fig. 10B ) and two diagonal directions d1( Fig. 10C ) and d2( Fig. 10D )’s gradient g v , g h , g d1 and g d2 The same subsampling position can be used for gradient calculations in different directions. Fig. 10A The label "V" shows the method used to calculate the vertical gradient g. v The sub-sampling position of Fig. 10B The label "H" shows the calculation method for the horizontal gradient g. h The sub-sampling position of Fig. 10C The label "d1" shows the calculation method used to calculate the diagonal gradient g of d1. d1 The sub-sampling position of Fig. 10D The label "d2" shows the calculation method used to calculate the diagonal gradient g of d2. d2 The subsampling position.

[0145] Horizontal and vertical gradients g v and g h The maximum value of and minimum value Can be set to:

[0146]

[0147] Two diagonal gradients g d1 and g d2 The maximum value of and minimum value Can be set to:

[0148] Step 1: If (1) and (2) is true, D is set to 0.

[0149] Step 2: If Proceed to step 3; otherwise proceed to step 4.

[0150] Step 3: If D is set to 2; otherwise D is set to 1.

[0151] Step 4: If D is set to 4; otherwise D is set to 3.

[0152] The activity value A can be calculated as:

[0153]

[0154] A can be further quantized to the range of 0 to 4, and the quantized value is represented as

[0155] For the chrominance components in the image, no block classification is applied, therefore, a single set of ALF coefficients may be applied for each chrominance component.

[0156] The geometric transformation may be applied to the filter coefficients and the corresponding filter clipping values ​​(also called clipping values). Before filtering a block (e.g., a 4×4 luma block), for example, depending on the gradient values ​​used for block calculation (e.g., g v , g h , g d1 and / or d2 ), a geometric transformation such as rotation or diagonal flipping and vertical flipping can be applied to the filter coefficients f(k, l) and the corresponding filter clipping values ​​c(k, l). The geometric transformation applied to the filter coefficients f(k, l) and the corresponding filter clipping values ​​c(k, l) can be equivalent to the geometric transformation applied to the samples in the area supported by the filter. By aligning the respective directionality, the geometric transformation can make different blocks to which the ALF is applied more similar.

[0157] Three geometric transformations, including diagonal flipping, vertical flipping, and rotation, can be performed as described in equations (9) to (11):

[0158] f D (k, l) = f(l, k), c D (k, l) = c(l, k), Equation (9)

[0159] f V (k, l) = f(k, Kl-1), c V (k, l) = c(k, Kl-1) Equation (10)

[0160] f R (k, l) = f(Kl-1, k), c R (k, l) = c(Kl-1, k) Equation (11)

[0161] Where K is the size of the ALF or filter, and 0≤k, l≤K-1 are the coordinates of the coefficients. For example, position (0, 0) is located at the upper left corner, and position (K-1, K-1) is located at the lower right corner of the filter f or the cropping value matrix (or cropping matrix) c. Depending on the gradient value used for block calculation, a transform can be applied to the filter coefficients f(k, l) and the cropping value c(k, l). An example of the relationship between the transform and the four gradient values ​​is summarized in Table 1.

[0162] Table 1: Mapping and transformation of gradient values ​​for block computation

[0163] Gradient Value Transform <![CDATA[g d2 <g d1 and g h <g v ]]> No transformation <![CDATA[g d2 <g d1 and g v <g h ]]> Diagonal Flip <![CDATA[g d1 <g d2 and g h <g v ]]> Flip Vertically <![CDATA[g d1 <g d2 and g v <g h ]]> Rotation

[0164] In some embodiments, such as in VVC, the ALF filter parameters are signaled in the Adaptation Parameter Set (APS) of the image. In the APS, one or more groups (e.g., up to 25 groups) of luminance filter coefficients and cropping value indices can be signaled. In one example, one of the one or more groups may include luminance filter coefficients and one or more cropping value indices. One or more groups (e.g., up to 8 groups) of chrominance filter coefficients and cropping value indices can be signaled. In order to reduce signaling overhead, filter coefficients for different classifications of luminance components (e.g., with different classification indices) can be merged. In the slice header, the APS index for the current slice can be signaled. In one example, such as in VVC, the signaling of ALF is based on CTU.

[0165] In one embodiment, a cropping value index (also referred to as a cropping index) may be decoded from the APS. The cropping value index may be used to determine a corresponding cropping value, for example, based on a relationship between the cropping value index and the corresponding cropping value. The relationship may be predefined and stored in the decoder. In one example, the relationship is described by a table, for example, a table of cropping value indexes and corresponding cropping values ​​for luma components (e.g., for luma CBs), a table of cropping value indexes and corresponding cropping values ​​for chroma components (e.g., for chroma CBs). The cropping value may depend on the bit depth B. The bit depth B may refer to an internal bit depth, a bit depth of a reconstructed sample in a CB to be filtered, etc. In some examples, a table of cropping values ​​(e.g., for luma and / or for chroma) is obtained using equation (12).

[0166] AlfClip = {round(2 B-α*n )for n∈[0..N-1]}, Equation (12)

[0167] -wherein, AlfClip is the clipping value, B is the bit depth (e.g., bitDepth), N (e.g., N=4) is the number of allowed clipping values, and α is a predefined constant value. In one example, α is equal to 2.35. n is the clipping value index (also called clipping index or clipIdx). Table 2 shows an example of a table obtained using equation (12) and N=4. In Table 2, the clipping index n can be 0, 1, 2, and 3. Table 2 can be used for luma blocks or chroma blocks.

[0168] Table 2 - AlfClip can depend on bit depth B and clipIdx

[0169]

[0170] In the slice header of the current slice, one or more APS indexes (e.g., up to 7 APS indexes) can be signaled to specify the luma filter group that can be used for the current slice. The filtering process can be controlled at one or more suitable levels, such as image level, slice level, CTB level and / or similar levels. In one embodiment, the filtering process can be further controlled at the CTB level. A flag can be signaled to indicate whether the ALF is applied to the luma CTB. The luma CTB can select a filter group from multiple fixed filter groups (e.g., 16 fixed filter groups) and filter groups signaled in the APS (also referred to as signaling filter groups). The filter group index can be signaled to the luma CTB to indicate the filter group to be applied (e.g., one of multiple fixed filter groups and the signaled filter group). Multiple fixed filter groups can be predefined and hard-coded in the encoder and decoder and can be referred to as predefined filter groups.

[0171] For chroma components, the APS index may be signaled in the slice header to indicate the chroma filter set to be used for the current slice. At the CTB level, if multiple chroma filter sets exist in the APS, the filter set index may be signaled for each chroma CTB.

[0172] The filter coefficients may be quantized with a norm equal to 128. To reduce multiplication complexity, code stream consistency may be applied so that the coefficient values ​​at non-center positions may be in the range of -27 to 27-1, inclusive. In one example, the center position coefficients are not signaled in the code stream and may be considered equal to 128.

[0173] In some embodiments, the syntax and semantics of the clipping index and clipping value are defined as follows: alf_luma_clip_idx[sfIdx][j] may be used to specify the clipping index of the clipping value to be used before multiplying the jth coefficient of the signal brightness filter indicated by sfIdx. The requirements for code stream consistency may include: the value of alf_luma_clip_idx[sfIdx][j] with sfIdx=0 to alf_luma_num_filters_signalled_minus1 and j=0 to 11 shall be in the range of 0 to 3, inclusive.

[0174] The luma filter clipping values ​​AlfClipL[adaptation_parameter_set_id] have elements AlfClipL[adaptation_parameter_set_id][filtIdx][j], where filtIdx = 0 to NumAlfFilters-1 and j = 0 to 11, and the luma filter clipping values ​​AlfClipL[adaptation_parameter_set_id] can be derived as specified in Table 2, depending on bitDepth being set equal to BitDepthY, and clipIdx being set equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j].

[0175] alf_chroma_clip_idx[altIdx][j] may be used to specify the clip index of the clip value to be used before multiplying the jth coefficient of the alternative chroma filter with altIdx index. Codestream conformance requirements may include that the value of alf_chroma_clip_idx[altIdx][j] for altIdx=0 to alf_chroma_num_alt_filters_minus1 and j=0 to 5 shall be in the range of 0 to 3, inclusive.

[0176] The chroma filter clipping values ​​AlfClipC[adaptation_parameter_set_id][altIdx] have elements AlfClipC[adaptation_parameter_set_id][altIdx][j], where altIdx = 0 to alf_chroma_num_alt_filters_minus1 and j = 0 to 5, and the chroma filter clipping values ​​AlfClipC[adaptation_parameter_set_id][altIdx] can be derived as specified in Table 2, depending on bitDepth being set equal to BitDepthC and clipIdx being set equal to alf_chroma_clip_idx[altIdx][j].

[0177] In one embodiment, the filtering process can be described as follows. At the decoder side, when ALF is enabled for CTB, the sample R(i, j) within the CU (or CB) can be filtered to produce the filtered sample value R'(i, j), as shown below using equation (13). In one example, filtering is performed for each sample in the CU.

[0178]

[0179] Where f(k, l) represents the decoded filter coefficients, K(x, y) is the clipping function, and c(k, l) represents the decoded clipping parameters (or clipping values). The variables k and l can vary between -L / 2 and L / 2, where l represents the filter length. The clipping function K(x, y) = min(y, max(-y, x)) corresponds to the clipping function Clip3(-y, y, x). By combining the clipping function K(x, y), the loop filtering method (e.g., ALF) is a nonlinear process and can be referred to as a nonlinear ALF.

[0180] The selected cropping value may be encoded in the "alf_data" syntax element in the following manner: The cropping index corresponding to the selected cropping value may be encoded using a suitable encoding scheme (e.g., Golomb encoding scheme), as shown in Table 2. The encoding scheme may be the same encoding scheme used to encode the filter bank index.

[0181] In one embodiment, a virtual boundary filtering process may be used to reduce the line buffer requirements of the ALF. Thus, modified block classification and filtering may be employed for samples near a CTU boundary (e.g., a horizontal CTU boundary). By offsetting the horizontal CTU boundary (1120) by "Nsamples" samples, a virtual boundary (1130) may be defined as a line, such as Fig.11AAs shown, Nsamples may be a positive integer. In one example, for the luminance component, Nsamples is equal to 4, and for the chrominance component, Nsamples is equal to 2.

[0182] refer to Fig.11A , the modified block classification can be applied to the luminance component. In one example, for the one-dimensional Laplacian gradient calculation of the 4×4 block (1110) above the virtual boundary (1130), only the samples above the virtual boundary (1130) are used. Similarly, referring to Fig. 11B , for the one-dimensional Laplacian gradient calculation of the 4×4 block (1111) below the virtual boundary (1131) offset from the CTU boundary (1121), only the samples below the virtual boundary (1131) are used. By considering the reduced number of samples used in the one-dimensional Laplacian gradient calculation, the quantization of the activity value A can be scaled accordingly.

[0183] For the filtering process, symmetric padding operations at virtual boundaries can be used for luma and chroma components. FIG. 12A to FIG. 12F An example of such modified ALF filtering for luma components at virtual boundaries is shown. When a filtered sample is below a virtual boundary, neighboring samples above the virtual boundary may be padded. When a filtered sample is above a virtual boundary, neighboring samples below the virtual boundary may be padded. Fig. 12A , the adjacent sample C0 can be filled with the sample C2 located below the virtual boundary (1210). Fig. 12B , the adjacent sample C0 can be filled with the sample C2 located above the virtual boundary (1220). Fig. 12C , adjacent samples C1 to C3 may be filled with samples C5 to C7 located below the virtual boundary, respectively (1230). Fig.12D , adjacent samples C1 to C3 may be filled with samples C5 to C7 located above the virtual boundary, respectively (1240). Fig.12E , adjacent samples C4 to C8 may be filled with samples C10, C11, C12, C11, and C10 located below the virtual boundary, respectively (1250). Fig.12F , adjacent samples C4 to C8 may be filled with samples C10, C11, C12, C11, and C10 located above the virtual boundary, respectively (1260).

[0184] In some examples, when the sample and the adjacent sample are located on the left (or right) and the right (or left) side of the virtual boundary, the above description may be appropriately adjusted.

[0185] The image quadtree splitting aligned with the largest coding unit (LCU) can be used. In order to improve coding efficiency, an adaptive loop filter based on the coding unit synchronized image quadtree can be used in video coding. In one example, the luminance image is split into multiple multi-level quadtree partitions, and each partition boundary is aligned with the boundary of multiple largest coding units (LCUs). Each partition can have a filtering process and can therefore be referred to as a filtering unit or filtering unit (FU).

[0186] The 2-pass encoding process is described as follows. In the first pass, the quadtree splitting pattern and the optimal filter (or the most suitable filter) for each FU can be determined. During the determination process, the filtering distortion can be estimated by fast filtering distortion estimation (FFDE). According to the determined quadtree splitting pattern and the selected filters of multiple FUs (e.g., all FUs), the reconstructed image can be filtered. In the second pass, CU synchronous ALF on / off control can be performed. According to the ALF on / off result, the first filtered image is partially restored by reconstructing the image.

[0187] A top-down splitting strategy can be adopted to split the image into multi-level quadtree partitions by using a rate-distortion criterion. Each partition can be called an FU. The splitting process can align the quadtree partitions with the LCU boundaries, such as Fig.13 shown. Fig.13 An example of LCU-aligned image quadtree splitting according to an embodiment of the present application is shown. In one example, the encoding order of multiple FUs follows the Z scan order. For example, referring to Fig.13 , the image is split into 10 FUs (e.g., FU0 to FU9), and the encoding order is from FU0 to FU9, e.g., FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8 and FU9.

[0188] To indicate the image quadtree splitting pattern, the split flags may be encoded and transmitted in z-scan order. Fig.14 The embodiment according to the embodiment of the present application is shown. Fig.13 The quadtree splitting pattern. Fig.14 As shown in the example in , the quadtree split flags are encoded in z-scan order.

[0189] The filter for each FU can be selected from two filter groups based on the rate-distortion criterion. The first filter group can have a 1 / 2 symmetrical square filter and a diamond filter newly derived for the current FU. The second filter group can come from a delay filter buffer. The delay filter buffer can store filters that were previously derived for FUs in previous images. For the current FU, the filter with the minimum rate-distortion cost in the two filter groups can be selected. Similarly, if the current FU is not the smallest FU and can be further split into four sub-FUs, the rate-distortion costs of the four sub-FUs can be calculated. By recursively comparing the rate-distortion costs in the split and non-split cases, the image quadtree splitting pattern can be determined.

[0190] In some examples, the maximum quadtree split level is 2, so the maximum number of FUs is 16. During the quadtree split determination, the correlation values ​​used to derive the Wiener coefficients of the 16 FUs at the bottom quadtree level (the smallest FU) can be reused. The remaining FUs can derive the Wiener filters of the remaining FUs from the correlations of the 16 FUs at the bottom quadtree level. Therefore, in one example, only one frame buffer is accessed to derive the filter coefficients for all FUs.

[0191] After determining the quadtree splitting pattern, in order to further reduce the filtering distortion, CU synchronous ALF on / off control can be performed. By comparing the filtering distortion and non-filtering distortion, the leaf CU can explicitly turn on / off the ALF in the corresponding local area. Redesigning the filter coefficients according to the ALF on / off results can further improve the coding efficiency. In one example, the redesign process requires additional frame buffer accesses. Therefore, in some examples, such as the Coding unit Synchronous Picture Quadtree-based AdaptiveLoop Filter (CS-PQALF) encoder design, no redesign process is required after the CU synchronous ALF on / off determination to minimize the number of frame buffer accesses.

[0192] The cross-component filtering process may apply a cross-component filter, such as a cross-component adaptive loop filter (CC-ALF). The cross-component filter may use the luma sample value of the luma component (e.g., luma CB) to refine the chroma component (e.g., the chroma CB corresponding to the luma CB). In one example, the luma CB and the chroma CB are included in the CU.

[0193] Fig.15A cross-component filter (e.g., CC-ALF) for generating a chrominance component according to an embodiment of the present application is shown. In some examples, Fig.15 The filtering process of a first chroma component (e.g., first chroma CB), a second chroma component (e.g., second chroma CB), and a luma component (e.g., luma CB) is shown. The luma component may be filtered by a sample adaptive offset (SAO) filter (1510) to produce an SAO filtered luma component (1541). The SAO filtered luma component (1541) may be further filtered by an ALF luma filter (1516) to become a filtered luma CB (1561) (e.g., 'Y').

[0194] The first chroma component may be filtered by an SAO filter (1512) and an ALF chroma filter (1518) to generate a first intermediate component (1552). In addition, the SAO filtered luma component (1541) may be filtered by a cross-component filter (e.g., CC-ALF) (1521) for the first chroma component to generate a second intermediate component (1542). Subsequently, a filtered first chroma component (1562) (e.g., 'CB') may be generated based on at least one of the second intermediate component (1542) and the first intermediate component (1552). In one example, the filtered first chroma component (1562) (e.g., 'CB') may be generated by combining the second intermediate component (1542) and the first intermediate component (1552) using an adder (1522). The cross-component adaptive loop filtering process for the first chroma component may include steps performed by the CC-ALF (1521) and steps performed by, for example, the adder (1522).

[0195] The above description may be applicable to the second chroma component. The second chroma component may be filtered by the SAO filter (1514) and the ALF chroma filter (1518) to generate a third intermediate component (1553). In addition, the SAO filtered luma component (1541) may be filtered by a cross-component filter (e.g., CC-ALF) (1531) for the second chroma component to generate a fourth intermediate component (1543). Subsequently, a filtered second chroma component (1563) (e.g., 'CR') may be generated based on at least one of the fourth intermediate component (1543) and the third intermediate component (1553). In one example, a filtered second chroma component (1563) (e.g., 'CR') may be generated by combining the fourth intermediate component (1543) and the third intermediate component (1553) with an adder (1532). In one example, the cross-component adaptive loop filtering process for the second chroma component may include steps performed by the CC-ALF (1531) and steps performed by, for example, an adder (1532).

[0196] The cross-component filter (e.g., CC-ALF (1521), CC-ALF (1531)) can operate by applying a linear filter with any suitable filter shape to the luma component (or luma channel) to refine each chroma component (e.g., first chroma component, second chroma component).

[0197] Fig.16 An example of a filter (1600) according to an embodiment of an embodiment of the present application is shown. The filter (1600) may include non-zero filter coefficients and zero filter coefficients. The filter (1600) has a diamond (1620) (represented by a circle with black fill) formed by the filter coefficients (1610). In one example, the non-zero filter coefficients in the filter (1600) are included in the filter coefficients (1610), and the filter coefficients not included in the filter coefficients (1610) are zero. Therefore, the non-zero filter coefficients in the filter (1600) are included in the diamond (1620), and the filter coefficients not included in the diamond (1620) are zero. In one example, the number of filter coefficients of the filter (1600) is equal to the number of filter coefficients (1610), Fig.14 In the example shown, this is 18.

[0198] CC-ALF may include any suitable filter coefficients (also referred to as CC-ALF filter coefficients). Return to Reference Fig.15 , CC-ALF (1521) and CC-ALF (1531) may have the same filter shape, for example Fig.14The diamond shape (1620) shown, and the same number of filter coefficients. In one example, the values ​​of the filter coefficients in CC-ALF (1521) are different from the values ​​of the filter coefficients in CC-ALF (1531).

[0199] Typically, filter coefficients (e.g., non-zero filter coefficients) in CC-ALF can be transmitted in, for example, an APS. In one example, the filter coefficients can be scaled by a factor (e.g., 210) and can be rounded for a fixed point representation. The application of CC-ALF can be controlled on a variable block size and signaled by a context coding flag (e.g., a CC-ALF enable flag) received for each sample block. Context coding flags, such as a CC-ALF enable flag, can be signaled at any suitable level (e.g., a block level). For each chroma component, the block size and the CC-ALF enable flag can be received at the slice level. In some examples, 16×16, 32×32, and 64×64 block sizes (in chroma samples) can be supported.

[0200] In an example, the syntax changes of CC-ALF are shown in Table 3 below.

[0201] Table 3: Syntax changes of CC-ALF

[0202]

[0203] The semantics of CC-ALF related syntax can be described as follows:

[0204] alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] equal to 0 may indicate that the cross-component Cb filter is not applied to the Cb color component sample block at the luma position (xCtb, yCtb).

[0205] alf_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] not equal to 0 may indicate that the alf_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cb filter is applied to the Cb color component sample block at the luma position (xCtb, yCtb).

[0206] alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] equal to 0 may indicate that a cross-component Cr filter is not applied to the block of Cr color component samples at the luma position (xCtb, yCtb).

[0207] alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] not equal to 0 may indicate that the alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cr filter is applied to the Cr color component sample block at luma position (xCtb, yCtb).

[0208] An example of a chroma sampling format is described below. In general, one luma block may correspond to one or more chroma blocks, such as two chroma blocks. The number of samples in each chroma block may be less than the number of samples in the luma block. The chroma subsampling format (also referred to as a chroma subsampling format, for example, specified by chroma_format_idc) may represent a chroma horizontal subsampling factor (e.g., SubWidthC) and a chroma vertical subsampling factor (e.g., SubHeightC) between each chroma block and the corresponding luma block. In one example, the chroma subsampling format is 4:2:0, so the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 2, and in one example, the chroma subsampling format is 4:2:2, so the chroma horizontal subsampling factor (e.g., SubWidthC) is 2 and the chroma vertical subsampling factor (e.g., SubHeightC) is 1. In one example, the chroma subsampling format is 4:4:4, and therefore, the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 1. The chroma sample type (also referred to as the chroma sample position) may indicate a relative position of a chroma sample in a chroma block relative to at least one corresponding luma sample in a luma block.

[0209] Fig.17A and Fig. 17B FIG. 2 shows an exemplary position of chrominance samples relative to luma samples according to an embodiment of the present application. Fig.17A , the brightness samples (1701) are located in rows (1711) to (1718). Fig.17AThe luma sample (1701) shown in the figure may represent a portion of an image. In one example, a luma block (e.g., luma CB) includes luma sample (1701). The luma block may correspond to two chroma blocks having a 4:2:0 chroma subsampling format. In one example, each chroma block includes chroma sample (1703). Each chroma sample (e.g., chroma sample (1703(1)) corresponds to four luma samples (e.g., luma samples (1701(1)) to (1701(4)). In one example, the four luma samples are upper left sample (1701(1)), upper right sample (1701(2)), lower left sample (1701(3)), and lower right sample (1701(4)). The chroma sample (e.g., (1703(1))) is located between the upper left sample (1701(1)) and the lower right sample (1701(4)). The chroma sample type of the chroma block having the chroma sample (1703) at the left center position between the upper left sample (1701(3)) and the lower left sample (1701(3)) may be referred to as chroma sample type 0. Chroma sample type 0 represents a relative position 0 corresponding to the left center position between the upper left sample (1701(1)) and the lower left sample (1701(3)). Four luma samples (e.g., (1701(1)) to (1701(4))) may be referred to as adjacent luma samples of the chroma sample (1703)(1).

[0210] In one example, each chroma block includes a chroma sample (1704). The above description of the chroma sample (1703) may be applicable to the chroma sample (1704), so for the sake of brevity, a detailed description may be omitted. Each chroma sample (1704) may be located at a center position of four corresponding luma samples, and a chroma sample type of a chroma block having the chroma sample (1704) may be referred to as a chroma sample type 1. The chroma sample type 1 represents a relative position 1 corresponding to a center position of four luma samples (e.g., (1701(1)) to (1701(4))). For example, one of the plurality of chroma samples (1704) may be located at a center portion of the luma samples (1701(1)) to (1701(4)).

[0211] In one example, each chroma block includes a chroma sample (1705). Each chroma sample (1705) may be located at an upper left position co-located with an upper left sample of four corresponding luma samples (1701), and a chroma sample type of a chroma block having the chroma sample (1705) may be referred to as a chroma sample type 2. Thus, each chroma sample (1705) is co-located with an upper left sample of four luma samples (1701) corresponding to the corresponding chroma sample. Chroma sample type 2 indicates a relative position 2 corresponding to an upper left position of the four luma samples (1701). For example, one of the plurality of chroma samples (1705) may be located at an upper left position of luma samples (1701(1)) to (1701(4)).

[0212] In one example, each chroma block includes a chroma sample (1706). Each chroma sample (1706) may be located at a top center position between a corresponding top left sample and a corresponding top right sample, and a chroma sample type of a chroma block having the chroma sample (1706) may be referred to as a chroma sample type 3. Chroma sample type 3 indicates a relative position 3 corresponding to a top center position between a top left sample (and a top right sample). For example, one of the plurality of chroma samples (1706) may be located at a top center position of luma samples (1701(1)) to (1701(4)).

[0213] In one example, each chroma block includes a chroma sample (1707). Each chroma sample (1707) may be located at a lower left position co-located with a lower left sample of four corresponding luma samples (1701), and a chroma sample type of a chroma block having the chroma sample (1707) may be referred to as a chroma sample type 4. Thus, each chroma sample (1707) is co-located with a lower left sample of four luma samples (1701) corresponding to the corresponding chroma sample. Chroma sample type 4 indicates a relative position 4 corresponding to a lower left position of the four luma samples (1701). For example, one of the plurality of chroma samples (1707) may be located at a lower left position of luma samples (1701(1)) to (1701(4)).

[0214] In one example, each chroma block includes a chroma sample (1708). Each chroma sample (1708) is located at a bottom center position between a bottom left sample and a bottom right sample, and a chroma sample type of a chroma block having the chroma sample (1708) may be referred to as a chroma sample type 5. The chroma sample type 5 indicates a relative position 5 corresponding to a bottom center position between a bottom left sample and a bottom right sample of four luma samples (1701). For example, one of the plurality of chroma samples (1708) may be located between a bottom left sample and a bottom right sample of luma samples (1701(1)) to (1701(4)).

[0215] In general, any suitable chroma sample type may be used for a chroma subsampling format. Chroma sample types 0 to 5 are exemplary chroma sample types described using the chroma subsampling format 4:2:0. Other chroma sample types may be used for the chroma subsampling format 4:2:0. In addition, other chroma sample types and / or variants of chroma sample types 0 to 5 may be used for other chroma subsampling formats, such as 4:2:2, 4:4:4, etc. In one example, a chroma sample type combining chroma samples (1705) and (1707) is used for the chroma subsampling format 4:2:2.

[0216] In one example, a luma block is considered to have alternating rows, such as rows (1711) to (1712), which respectively include the top two samples (e.g., (1701(3)) to (1701(4))) of four luma samples (e.g., (1701(1)) to (1701(4))), and the bottom two samples (e.g., (1701(1)) to (1701(2))) of four luma samples (e.g., (1701(1)) to (1701(4))). Therefore, rows (1711), (1713), (1715), and (1717) can be referred to as current rows (also referred to as top fields), and rows (1712), (1714), (1716), and (1718) can be referred to as next rows (also referred to as bottom fields). Four brightness samples (e.g., (1701(1)) to (1701(4))) are located at the current row (e.g., (1711)) and the next row (e.g., (1712)). Relative positions 2 to 3 are located in the current row, relative positions 0 to 1 are located between each current row and the corresponding next row, and relative positions 4 to 5 are located in the next row.

[0217] Chroma samples (1703), (1704), (1705), (1706), (1707), or (1708) are located in rows (1751) to (1754) in each chroma block. The specific positions of rows (1751) to (1754) may depend on the chroma sample types of the chroma samples. For example, for chroma samples (1703) to (1704) having corresponding chroma sample types 0 to 1, row (1751) is located between rows (1711) to (1712). For chroma samples (1705) to (1706) having chroma sample types 2 to 3, respectively, row (1751) is located at the same position as the current row (1711). For chroma samples (1707) to (1708) having corresponding chroma sample types 4 to 5, row (1751) is located at the same position as the next row (1712). The above description may be appropriately applied to rows (1752) to (1754), and a detailed description is omitted for the sake of brevity.

[0218] Any suitable scanning method may be used to display, store and / or transmit the above Fig.17A In one example, a progressive scan is used.

[0219] like Fig. 17B As shown, interlaced scanning may be used. As described above, the chroma subsampling format is 4:2:0 (e.g., chroma_format_idc is equal to 1). In one example, a variable chroma location type (e.g., ChromaLocType) indicates a current row (e.g., ChromaLocType is chroma_sample_loc_type_top_field) or a next row (e.g., ChromaLocType is chroma_sample_loc_type_bottom_field). The current row (1711), (1713), (1715), and (1717) and the next row (1712), (1714), (1716), and (1718) may be scanned, respectively. For example, the current row (1711), (1713), (1715), and (1717) may be scanned first, and then the next row (1712), (1714), (1716), and (1718) may be scanned. The current row may include luma samples (1701) and the next row may include luma samples (1702).

[0220] Similarly, the corresponding chroma blocks may be scanned interlaced. Rows (1751) and (1753) including chroma samples (1703), (1704), (1705), (1706), (1707), or (1708) that are not filled may be referred to as current rows (or current chroma rows), and rows (1752) and (1754) including chroma samples (1703), (1704), (1705), (1706), (1707), or (1708) filled with grayscale samples may be referred to as next rows (or next chroma rows). In one example, during interlaced scanning, rows (1751) and (1753) are scanned first, and then rows (1752) and (1754) are scanned.

[0221] Constrained Directional Enhancement Filter (CDEF) can be used in video coding. In-loop CDEF can be used to filter out coding artifacts while preserving image details. In some examples, such as in HEVC, the Sample Adaptive Offset (SAO) algorithm achieves similar goals by defining signal offsets for different categories of pixels. Unlike SAO, CDEF is a nonlinear spatial filter. In some examples, the design of the filter is constrained to be easy to vectorize (e.g., it can be implemented by single instruction, multiple data (SIMD) operations), which is not the case for other nonlinear filters, such as median filters and bilateral filters.

[0222] In one example, the CDEF design is derived from the following observations. In some examples, the amount of ringing artifacts in the encoded image can be approximately proportional to the quantization step size. The amount of detail is a property of the input image. However, the minimum detail retained in the quantized image can be proportional to the quantization step size. For a given quantization step size, the magnitude of the ringing artifact may be smaller than the magnitude of the detail.

[0223] CDEF can identify the direction of each block. CDEF can then filter adaptively along the identified direction, and to a lesser extent along the direction rotated 45 degrees from the identified direction. The filter strength can be explicitly signaled, allowing a high degree of control over the blur. An efficient encoder search can be designed for the filter strength. CDEF can be based on two loop filters, and the combined filter can be used for video encoding, such as in the AV1 codec.

[0224] The directional search can operate on reconstructed pixels (or samples), for example, after a deblocking filter. Since the reconstructed pixels can be used in the decoder, the direction may not need to be signaled. The search can operate on blocks of appropriate size (e.g., 8×8 blocks) that are small enough to adequately handle non-straight edges and large enough to reliably estimate the direction when applied to a quantized image. Having a constant direction over an 8×8 area can make vectorization of the filter easier. For each block, the direction that best matches the pattern in the block can be determined by minimizing a difference metric, such as the Sum of Squared Difference (SSD), RMS error, etc., between the quantized block and each fully oriented block (1830). In one example, a fully oriented block (e.g., one of (1830)) is a block where all pixels along a line in one direction have the same value. Fig.18 An example of directional search of an 8×8 block (1810) according to an embodiment of the present application is shown. Fig.18In the example shown, the 45 degree direction (1823) is selected from the set of directions (1820) because the 45 degree direction (1823) can minimize the error. For example, the error of the 45 degree direction is 12, and it is the smallest in the error range from 12 to 87 represented by the row (1840).

[0225] A nonlinear low-pass directional filter is described below. Identifying a direction can help align filter taps along the identified direction to reduce ringing artifacts while preserving directional edges or patterns. However, in some examples, directional filtering alone may not sufficiently reduce ringing artifacts. It is desirable to use additional filter taps on pixels that are not along the main direction (e.g., the identified direction). To reduce the risk of blurring, additional filter taps may be handled more conservatively. Therefore, CDEF may define main taps and secondary taps. In one example, a complete two-dimensional (2D) CDEF filter is represented as:

[0226]

[0227] In equation (14), D is the damping parameter, S (p) and S (S) are the strengths of the main tap and the secondary tap, respectively. The function round(·) rounds the relationship away from zero, and and is the filter weight, and f(d, S, D) is a constraint function that operates on the difference d (e.g., d=x(m, n)-x(i, j)) between the filtered pixel (e.g., x(i, j)) and each adjacent pixel (e.g., x(m, n)). When the difference is small, f(d, S, D) can be equal to the difference d (e.g., f(d, S, D)=d), and thus the filter can behave as a linear filter. When the difference is large, f(d, S, D) can be equal to 0 (e.g., f(d, S, D)=0), which effectively ignores the filter taps.

[0228] In deblocked video coding, such as in AV1, a set of in-loop repair schemes can be used to generally denoise and enhance the quality of edges in addition to the deblocking operation. The set of in-loop repair schemes can be switched within a frame (or image) for each appropriately sized tile. Some examples of in-loop repair schemes are described based on separable symmetric Wiener filters and dual self-guided filters with subspace projection. Because content statistics can vary greatly within a frame, the tools can be integrated within a switchable framework, where different tools can be triggered in different areas of the frame.

[0229] A separable symmetric Wiener filter may be described below. One repair tool may be a Wiener filter. Each pixel (or sample) in a degraded frame may be reconstructed as a non-causal filtered version of the pixel within a w×w window around the pixel, where w=2r+1 and is odd for integer r. The two-dimensional filter taps may be used with w 2 ×1 elements in column vectorized form, and direct linear minimum mean square error (LMMSE) optimization can lead to F = H -1 M gives the filter parameters, where H is equal to E[XX T ] and is the autocovariance of x, w in a w×w window around the pixel 2 The column vectorized version of the sample, and M is equal to E[YX T ], which is the cross-correlation of x with the scalar source sample y to be estimated. The encoder can estimate H and M based on the implementation in the deblocked frame and source, and send the resulting filter F to the decoder. However, in some examples, when transmitting w 2 A considerable bit rate cost may be incurred when tapping. In addition, non-separable filtering makes decoding extremely complicated. Therefore, a number of additional restrictions can be imposed on the properties of F. For example, F is constrained to be separable so that filtering can be implemented as separable horizontal and vertical w-tap convolutions. In one example, each of the horizontal filter and the vertical filter is constrained to be symmetric. In addition, in one example, it can be assumed that the sum of the horizontal filter and the vertical filter coefficients is equal to 1.

[0230] The dual self-guided filter with subspace projection can be described below. In some examples, guided filtering can be used for image filtering, where a local linear model can be used to calculate the filtered output y from the unfiltered sample x. The local linear model can be written as:

[0231] y = F x + G Equation (15)

[0232] Where F and G can be determined based on the statistics of the degraded image and the guide image (also called the guidance image) in the neighborhood of the filtered pixel. If the guide image is the same as the degraded image, the resulting self-guided filtering can achieve the effect of edge-preserving smoothing. According to some aspects of the embodiments of the present application, the specific form of the self-guided filtering can depend on two parameters: the radius r and the noise parameter e, and can be listed as follows:

[0233] 1. In the (2r+1)×(2r+1) window around each pixel, obtain the mean μ and variance σ2 of the pixel. Obtaining the mean μ and variance σ2 of the pixel can be effectively achieved by box filtering based on integral imaging.

[0234] 2. Calculate the parameters f and g for each pixel according to equation (16).

[0235] f = σ 2 / (σ 2 + e); g = (1 - f)μ Equation (16)

[0236] 3. Calculate F and G for each pixel as the average of parameters f and g in a 3×3 window around the pixel for use.

[0237] The filtering can be controlled by the radius r and the noise parameter e, where a larger radius r may imply a higher spatial variance and a higher noise parameter e may imply a higher range variance.

[0238] Fig.19 An example of subspace projection according to an embodiment of the present application is shown. Fig.19 In the example shown, subspace projection can use the cheap restorations X1 and X2 to produce a final restoration X that is closer to the source Y. f Even if the cheap restorations X1 and X2 are not close to the source Y, if the cheap restorations X1 and X2 are moved in the right direction, appropriate multipliers {α, β} can make the cheap restorations X1 and X2 closer to the source Y. In one example, the final restoration X1 is obtained based on the following equation (17): f .

[0239] X f =X+α(X1-X)+β(X2-X) Equation (17)

[0240] A loop filtering method called a cross-component sample offset (CCSO) filter or CCSO can be used to reduce the distortion of reconstructed samples (also called reconstructed samples). In the CCSO filtering process, a nonlinear mapping can be used to determine an output offset based on the processed input reconstructed samples of the first color component. In the CCSO filtering process, the output offset can be added to the reconstructed samples of the second color component.

[0241] like Fig. 20 As shown, the input reconstructed samples may be from the first color component located in the filter support region. Fig. 20An example of a filter support region in a CCSO filter according to an embodiment of an embodiment of the present application is shown. The filter support region may include four reconstruction samples: p0, p1, p2, and p3. The four input reconstruction samples may be in a cross shape in the vertical and horizontal directions. In one example, the center sample in the first color component (represented by c) and the sample to be filtered in the second color component (represented by f) are located at the same position. When processing the input reconstruction samples, the following steps may be applied:

[0242] Step 1: Calculate the incremental values ​​(eg, differences) between the four reconstructed samples p0, p1, p2, and p3 and the center sample c, and denote them as m0, m1, m2, and m3, respectively. For example, the incremental value between p0 and c is m0.

[0243] Step 2: The incremental values ​​m0 to m3 may be further quantized. The quantized values ​​may be represented as d0, d1, d2, and d3. In one example, the quantized values ​​are -1, 0, or 1 based on the following quantization process:

[0244] di = -1, if mi < -N; Equation (18)

[0245] di = 0, if -N <= mi <= N; Equation (19)

[0246] di = 1, if mi > N Equation (20)

[0247] Where N is the quantization step size, example values ​​of N are 4, 8, 12, 16, etc., di and mi refer to the respective quantization value and increment value, where i is 0, 1, 2 or 3.

[0248] The quantized values ​​d0 to d3 can be used to identify combinations of nonlinear mappings. Fig. 20 In the example shown, the CCSO filter has four filter inputs d0 to d3, and each filter input can have one of three quantized values ​​(eg, -1, 0, and 1), so the total number of combinations is 81 (eg, 34). FIG. 21A to FIG. 21C An example of 81 combinations according to an embodiment of the present application is shown. The last column may represent the output offset value of each combination. The output offset value may be an integer, such as 0, 1, -1, 3, -3, 5, -5, -7, etc.

[0249] The final filtering process of the CCSO filter can be applied as follows:

[0250] f′=clip(f+s), Equation (21)

[0251] Among them, f is the reconstructed sample to be filtered, and s is the output offset value. FIG. 21A to FIG. 21CIn one example shown in equation (21), the filtered sample value f' of the reconstructed sample f to be filtered may be further clipped to a range associated with the bit depth.

[0252] A Local Sample Offset (LSO) method or LSO filtering process may be used in video coding. In LSO, a filtering method similar to CCSO may be applied. However, the output offset value may be applied to the same color component as the color component where the input reconstructed samples used in the filtering process are located. Thus, in LSO, the input reconstructed samples used in the filtering process (e.g., p0 to p3 and c) and the reconstructed samples to be filtered (e.g., f) are located in the same component, e.g., a luminance component, a chrominance component, or any suitable component. LSO may have a similar or identical filter shape as CCSO (e.g., Fig. 20 shown).

[0253] A low-complexity three-tap CCSO design can be used for video coding. Fig. 22 An example implementation of CCSO according to an embodiment of the present application is shown. Eight different filter shapes can be defined. Each filter shape can define the position of three reconstructed samples (also called three taps) in the first component (also called the first color component). The three reconstructed samples may include a center sample (denoted as c). In one example, the reconstructed sample in the second color component to be filtered is located at the same position as the center sample c. For clarity, in Fig. 22 The reconstructed samples in the second color component to be filtered are not shown.

[0254] The SAO filter can be used for video coding. In one example, such as in HEVC, the SAO filter or SAO filtering process can be applied to the reconstructed signal after the deblocking filter by using an offset value, such as in a slice header. For luma samples, the encoder can determine whether the SAO filter is applied to the current slice. If the SAO filter is enabled, the current image can be recursively split into four sub-regions, and one of six SAO types (e.g., SAO types 1 to 6) can be selected for each sub-region, as shown in Table 4. The SAO filter can classify reconstructed pixels into multiple categories and reduce distortion by adding an offset to pixels of each category in the current sub-region. Edge attributes can be used for pixel classification in SAO types 1 to 4, and pixel intensity can be used for pixel classification in SAO types 5 to 6.

[0255] Table 4: SAO type specifications

[0256] SAO type SAO type used Number of categories 0 none 0 1 One-dimensional 0 degree graphic edge offset 4 2 One-dimensional 90 degree graphic edge offset 4 3 One-dimensional 135 degree graphic edge offset 4 4 One-dimensional 45 degree graphic edge offset 4 5 Center Band Band Offset 16 6 Sideband offset 16

[0257] Band Offset (BO) can be used to classify pixels (e.g., all pixels) of a sub-region into multiple bands, where each band may include pixels in the same intensity interval. The intensity range can be equally divided into multiple intervals (e.g., 32 intervals) from a minimum intensity value (e.g., zero) to a maximum intensity value (e.g., 255 for 8-bit pixels), and each interval may have an offset. Subsequently, multiple intervals or bands (e.g., 32 bands) can be divided into two groups. One group may include 16 center bands and the other group may include 16 remaining bands. In one example, only the offset in one group is transmitted. For pixel classification operations in BO, the five most significant bits of each pixel can be used directly as a band index.

[0258] like Fig.23 As shown, edge offset (EO) can use four one-dimensional 3-pixel patterns for pixel classification considering edge direction information. Fig.23 An example of four one-dimensional 3-pixel patterns for pixel classification in EO is shown. The four one-dimensional 3-pixel patterns correspond to a one-dimensional 0 degree pattern (2310), a one-dimensional 90 degree pattern (2320), a one-dimensional 135 degree pattern (2330), and a one-dimensional 45 degree pattern (2340) from left to right. For each sub-region of an image (e.g., a current image), one of the four patterns can be selected by comparing each pixel with its two neighboring pixels to classify the pixel into multiple categories. The selection can be sent in the code stream as incidental information. Table 5 shows the pixel classification rules for EO.

[0259] Table 5: EO pixel classification rules

[0260]

[0261] In one example, it is desired that the SAO on the decoder side operates independently of the LCU so that the line buffer can be saved. In order to operate the SAO LCU independently, in one example, when the 90 degree classification pattern, the 135 degree classification pattern, and the 45 degree classification pattern are selected, the pixels of the top and bottom rows in each LCU are not SAO processed, and when the 0 degree pattern, the 135 degree pattern, and the 45 degree pattern are selected, the pixels of the leftmost column and the rightmost column in each LCU are not SAO processed.

[0262] Table 6 below describes the syntax by which a CTU may be signaled if the parameters are not merged from neighboring CTUs.

[0263] Table 6: SAO syntax example

[0264] sao_offset_vlc(rx, ry, cIdx){ Descriptors sao_type_idx[cIdx][rx][ry] ue(v) if (sao_type_idx[cIdx][rx][ry] == 5) { sao_band_position[cIdx][rx][ry] ue(v) For (i=0; i<4; i++) sao_offset[cIdx][rx][ry][i] se(v) }if (sao_type_idx[cIdx][rx][ry]!=0) For (i=0; i<4; i++) sao_offset[cIdx][rx][ry][i] ue(v) }

[0265] Some filters or filter shapes, such as those used in the CCSO filtering process, require access to neighboring samples that may be relatively far away from a center sample co-located with the reconstructed sample to be filtered (e.g., Fig. 22 5, 7, and 8 shown, where it is necessary to access neighboring samples located four rows above the center sample). In some examples, accessing one or more neighboring samples relatively far from the center sample may be too expensive to implement the CCSO filter in hardware. Figure 24 to Figure 28 As shown, a sample offset filter (e.g., CCSO or LSO) that uses samples that are relatively close to the center sample can be used. In some examples, accessing rows that are below the center sample and relatively far from the center sample does not increase the implementation cost as significantly as accessing rows that are above the center sample and relatively far from the center sample. Therefore, in some examples, a restriction is applied to access a limited number of rows above the center sample, and a filter such as Fig.29 The sample offset filter shown in .

[0266] According to various aspects of the embodiments of the present application, a sample offset filter (or a sample offset filtering process) may refer to a filter or filtering process in which the reconstructed samples of a first component (also referred to as a first color component) are used as input to a sample offset filter (or a sample offset filtering process), and the output of the sample offset filter (or a sample offset filtering process) may be applied to the reconstructed samples (to be filtered) of a second component (or a second color component). The reconstructed samples of the second component may be filtered based on the output of the sample offset filter, as shown in equation (21). The sample offset filter may be a loop filter, through which the output (also referred to as an output value) is applied as an offset to the reconstructed samples to filter out coding artifacts while retaining the details of the second component. The sample offset filter may include one of the following: LSO, CCSO, and an adaptive sample offset filter (ASO).

[0267] Components such as the first component and the second component may be any suitable components in a color space (e.g., YCrCB, RGB, etc.) used in video encoding, such as a luminance component, a chrominance component, a Y component, a Cb component, a Cr component, an R component, a G component, a B component, etc. The second component may also be referred to as a current component because the reconstructed samples to be filtered will be processed by an encoder and / or a decoder.

[0268] In one embodiment, the first component (or first color component) is different from the second component (or second color component), and the sample offset filter is referred to as a CCSO filter. More specifically, CCSO may refer to a filter or filtering process that uses the reconstructed samples of the first component as input to a CCSO filter (or CCSO filtering process), and the output of the CCSO filter (or CCSO filtering process) may be applied to the reconstructed samples (to be filtered) of the second component (or second color component). The reconstructed samples of the second component may be filtered based on the output of the CCSO filter, as shown in equation (21). In one example, for CCSO, the first component is a luminance component and the second component is a chrominance component. In one example, for CCSO, the first component and the second component are two different chrominance components.

[0269] In one embodiment, the first component in the sample offset filter is the second component in the sample offset filter, and the sample offset filter may be referred to as an LSO filter. The LSO filter may be a filter or filtering process that uses the reconstructed samples (e.g., Y, Cb, or Cr) of the first component (or first color component) as input, and the output may be applied to the same first component, for example, to filter the reconstructed samples of the first component.

[0270] When the sample offset filter is applied to the reconstructed sample (to be filtered) of the current component (or current color component) in the current image, the filtering shape of the sample offset filter can be one of multiple filtering shapes (or multiple filters). Each of the multiple filtering shapes can include the first reconstructed sample of the first component (or first color component) in the current image. According to various aspects of the embodiments of the present application, for each of the multiple filtering shapes, the topmost sample in the first reconstructed sample is located within N rows (lines) from the center sample of the first component (also referred to as the center reconstructed sample), and the center sample is located at the same position as the reconstructed sample (to be filtered) of the current component. The reconstructed sample to be filtered is in the current component.

[0271] In some embodiments, the center sample of the first component is located at the same position as the reconstructed sample (to be filtered) of the current component. In one example, the first component is a luminance component and the current component is a chrominance component. Fig.17A , the center sample is the luma sample (1701(1)), the reconstructed sample to be filtered is the chroma sample 1705, and the center sample (e.g., (1701(1))) and the reconstructed sample to be filtered (e.g., (1705(1))) are located at the same position.

[0272] In some embodiments, the center sample of the first component is located at a slightly different position than the center sample of the reconstructed sample (to be filtered) of the current component. Fig.17A, the center sample is a luma sample (1701(1)), the reconstructed sample to be filtered is a chroma sample (1703(1)), and the center sample (e.g., (1701(1))) and the reconstructed sample to be filtered (e.g., (1703(1))) are located at slightly different positions (e.g., the difference along the vertical direction is less than one line). For example, referring to Fig.17A , the vertical offset in the current image is 0.5 lines. The center sample (eg, (1703(1))) and the reconstructed sample to be filtered (eg, (1703(1))) are located at the same position.

[0273] In some embodiments, for each of a plurality of filtering shapes, the topmost sample of the first reconstructed sample is located within N rows (or lines) of the reconstructed sample to be filtered. N can be any suitable positive integer, for example in the range of 1 to 1024. For example, N can be limited to a relatively small integer to reduce the hardware implementation cost of a sample offset filter (e.g., CCSO, LSO). In some examples, when N is limited to a relatively small integer, only adjacent samples within a limited number of rows or lines of the center sample are used. In one example, N is less than or equal to 3.

[0274] In one example, the number of first reconstructed samples in each of the plurality of filter shapes is less than five.

[0275] In one example, each of the plurality of filter shapes has the same number M of first reconstruction samples of the first component. M is a positive integer. A filter or filter shape having M first reconstruction samples may be referred to as an M-tap filter or a filter having M taps.

[0276] In one example, N is 1. In one example, when N is 1, the number of first reconstructed samples in each of the plurality of filter shapes is three.

[0277] In one embodiment, each of the plurality of filter shapes comprises a center sample located at the same position as the reconstructed sample (to be filtered) of the current component. The first reconstructed sample comprises the center sample. The topmost sample of the first reconstructed sample is located within N rows (or lines) from the center sample. As described above, N may be less than or equal to 3. In one example, N is 1.

[0278] Subsequently, the output value of the sample offset filter may be determined based on the first reconstructed sample of the filtered shape. In addition, the filtered sample value of the reconstructed sample of the current component may be determined based on the reconstructed sample and the output value of the sample offset filter, as shown in equation (21).

[0279] In one embodiment, each of the plurality of filter shapes includes a center sample co-located with a reconstructed sample of the current component, and the number of first reconstructed samples in each of the plurality of filter shapes is 3. The remaining first reconstructed samples (different from the center sample) in each of the plurality of filter shapes may be positioned symmetrically or asymmetrically relative to the center sample. In addition, in one example, the first reconstructed sample in each of the plurality of filter shapes includes the center sample and two samples symmetrical relative to the center sample. In one example, N is 1.

[0280] In one example, if Fig.24 As shown, the multiple filter shapes (or multiple filters) used in the sample offset filter (eg, CCSO or LSO) include one or more of the following 7 different 3-tap filter shapes. Fig.24 An example of multiple filter shapes (or multiple filters) used in a sample offset filter (e.g., CCSO or LSO) according to an embodiment of an embodiment of the present application is shown. The multiple filter shapes include filter shapes (or filters) 1 to 7. Each of the multiple filter shapes may include three first reconstructed samples, and the first reconstructed sample includes a center sample (denoted as c) and two samples symmetrical about the center sample. N is 1. For example, for filter shapes 1, 2, 4, 6, and 7, the topmost sample of the first reconstructed sample (e.g., each of samples (2401) to (2405)) is located within 1 row (e.g., 1 row above) from the reconstructed sample to be filtered (located at the same position as the center sample c). For example, for filter shapes 3 and 5, the topmost sample of the first reconstructed sample is located within 1 row (e.g., the same row) from the reconstructed sample to be filtered (located at the same position as the center sample c).

[0281] More specifically, refer to Fig.24, the position of the center sample c in the first component of the current image is (i, j), where i and j are integers. The multiple filter shapes include seven filter shapes 1 to 7. The positions of the two samples in the first shape of the seven filter shapes are (i-1, j) and (i+1, j). The positions of the two samples in the second shape of the seven filter shapes are (i-1, j-1) and (i+1, j+1). The positions of the two samples in the third shape of the seven filter shapes are (i, j-1) and (i, j+1). The positions of the two samples in the fourth shape of the seven filter shapes are (i+1, j-1) and (i-1, j+1). The positions of the two samples in the fifth shape of the seven filter shapes are (i, j-4) and (i, j+4). The positions of the two samples in the sixth shape of the seven filter shapes are (i-1, j-7) and (i+1, j+7). The positions of the two samples in the seventh of the seven filter shapes are (i+1, j-7) and (i-1, j+7). Fig.24 Subgroups of the filter shapes in may be used in other embodiments. In one example, the plurality of filter shapes (or filters) used in a sample offset filter (eg, CCSO or LSO) include Fig.24 One or more filter shapes from filter shapes 1 to 7 in .

[0282] In one example, if Fig.25 As shown, in a sample offset filter (e.g., CCSO or LSO)

[0283] The multiple filter shapes (or multiple filters) used in include one or more of the following 6 different 3-tap filter shapes. Fig.25 An example of multiple filter shapes (or multiple filters) used in a sample offset filter (e.g., CCSO or LSO) according to an embodiment of the present application is shown. The multiple filter shapes include filter shapes (or filters) 1 to 6. Each of the multiple filter shapes may include three first reconstruction samples, and the first reconstruction sample includes a center sample

[0284] (denoted as c) and two samples symmetric about the center sample. N is 1. For example, for filter shapes 1, 2, and 4, the topmost samples of the first reconstructed samples (e.g., samples (2501) to (2503)

[0285] Each sample in (c) is located within 1 row (e.g., 1 row above) from the reconstructed sample to be filtered (located at the same position as the center sample c). For example, for filtering shapes 3, 5, and 6, the topmost sample of the first reconstructed sample is located within 1 row (e.g., the same row) from the reconstructed sample to be filtered (located at the same position as the center sample c).

[0286] More specifically, refer to Fig.25 , the position of the center sample c in the first component of the current image is (i, j), where i and j are integers. The multiple filter shapes include seven filter shapes 1 to 6. The positions of the two samples in the first shape of the six filter shapes are (i-1, j) and (i+1, j). The positions of the two samples in the second shape of the six filter shapes are (i-1, j-1) and (i+1, j+1). The positions of the two samples in the third shape of the six filter shapes are (i, j-1) and (i, j+1). The positions of the two samples in the fourth shape of the six filter shapes are (i+1, j-1) and (i-1, j+1). The positions of the two samples in the fifth shape of the six filter shapes are (i, j-4) and (i, j+4). The positions of the two samples in the sixth shape of the six filter shapes are (i, j-7) and (i, j+7). Subgroups of filter shapes can be used in other embodiments. In one example, the plurality of filter shapes (or filters) used in a sample shift filter (eg, CCSO or LSO) include Fig.25 One or more filter shapes among filter shapes 1 to 6 in .

[0287] In one example, if Fig.26 As shown, the multiple filter shapes (or multiple filters) used in the sample offset filter (eg, CCSO or LSO) include one or more of the following 7 different 3-tap filter shapes. Fig.26 An example of multiple filter shapes (or multiple filters) used in a sample offset filter (e.g., CCSO or LSO) according to an embodiment of an embodiment of the present application is shown. The multiple filter shapes include filter shapes (or filters) 1 to 7. Each of the multiple filter shapes may include three first reconstructed samples, the first reconstructed sample including a center sample (denoted as c) and two samples symmetrical about the center sample. N is 1. For example, for filter shapes 1, 2, and 4 to 7, the topmost sample of the first reconstructed sample (e.g., each of samples (2601) to (2606)) is located within 1 row (e.g., 1 row above) from the reconstructed sample to be filtered (located at the same position as the center sample c). For example, for filter shape 3, the topmost sample in the first reconstructed sample is located within 1 row (e.g., the same row) from the reconstructed sample to be filtered (located at the same position as the center sample c).

[0288] More specifically, refer to Fig.26, the position of the center sample c in the first component of the current image is (i, j), where i and j are integers. The multiple filter shapes include seven filter shapes 1 to 7. The positions of the two samples in the first shape of the seven filter shapes are (i-1, j) and (i+1, j). The positions of the two samples in the second shape of the seven filter shapes are (i-1, j-1) and (i+1, j+1). The positions of the two samples in the third shape of the seven filter shapes are (i, j-1) and (i, j+1). The positions of the two samples in the fourth shape of the seven filter shapes are (i+1, j-1) and (i-1, j+1). The positions of the two samples in the fifth shape of the seven filter shapes are (i-1, j-3) and (i+1, j+3). The positions of the two samples in the sixth shape of the seven filter shapes are (i+1, j-5) and (i-1, j+5). The positions of the two samples in the seventh of the seven filter shapes are (i-1, j-7) and (i+1, j+7). Subgroups of filter shapes may be used in other embodiments. In one example, the plurality of filter shapes (or filters) used in a sample offset filter (e.g., CCSO or LSO) include Fig.26 One or more filter shapes from filter shapes 1 to 7 in .

[0289] In one example, if Fig. 27 As shown, the multiple filter shapes (or multiple filters) used in the sample offset filter (eg, CCSO or LSO) include one or more of the following 7 different 3-tap filter shapes. Fig. 27 An example of multiple filter shapes (or multiple filters) used in a sample offset filter (e.g., CCSO or LSO) according to an embodiment of an embodiment of the present application is shown. The multiple filter shapes include filter shapes (or filters) 1 to 7. Each of the multiple filter shapes may include three first reconstructed samples, and the first reconstructed sample includes a center sample (denoted as c) and two samples symmetrical about the center sample. N is 1. For example, for filter shapes 1, 2, and 4 to 7, the topmost sample of the first reconstructed sample (e.g., each of samples (2701) to (2706)) is located within 1 row (e.g., 1 row above) from the reconstructed sample to be filtered (located at the same position as the center sample c). For example, for filter shape 3, the topmost sample in the first reconstructed sample is located within 1 row (e.g., the same row) from the reconstructed sample to be filtered (located at the same position as the center sample c).

[0290] More specifically, refer to Fig. 27, the position of the center sample c in the first component of the current image is (i, j), where i and j are integers. The multiple filter shapes include seven filter shapes 1 to 7. The positions of the two samples in the first shape of the seven filter shapes are (i-1, j) and (i+1, j). The positions of the two samples in the second shape of the seven filter shapes are (i-1, j-1) and (i+1, j+1). The positions of the two samples in the third shape of the seven filter shapes are (i, j-1) and (i, j+1). The positions of the two samples in the fourth shape of the seven filter shapes are (i+1, j-1) and (i-1, j+1). The positions of the two samples in the fifth shape of the seven filter shapes are (i+1, j-3) and (i-1, j+3). The positions of the two samples in the sixth shape of the seven filter shapes are (i-1, j-5) and (i+1, j+5). The positions of the two samples in the seventh of the seven filter shapes are (i+1, j-7) and (i-1, j+7). Subgroups of filter shapes may be used in other embodiments. In one example, the plurality of filter shapes (or filters) used in a sample offset filter (e.g., CCSO or LSO) include Fig. 27 One or more filter shapes from filter shapes 1 to 7 in .

[0291] In one example, if Fig.28 As shown, the multiple filter shapes (or multiple filters) used in the sample offset filter (e.g., CCSO or LSO) include one or more of the following 7 different 3-tap filter shapes (filter shapes or filters 1 to 7). Fig.28 An example of multiple filter shapes (or multiple filters) used in a sample offset filter (e.g., CCSO or LSO) according to an embodiment of an embodiment of the present application is shown. Each of the multiple filter shapes 1 to 7 may include three first reconstructed samples, and the first reconstructed sample includes a center sample (denoted as c) and two samples symmetrical about the center sample. N is 1. For example, for filter shapes 1, 2, and 4, the topmost sample of the first reconstructed sample (e.g., each of samples (2801) to (2803)) is located within 1 row (e.g., 1 row above) from the reconstructed sample to be filtered (located at the same position as the center sample c). For example, for filter shapes 3 and 5 to 7, the topmost sample of the first reconstructed sample is located within 1 row (e.g., the same row) from the reconstructed sample to be filtered (located at the same position as the center sample c).

[0292] More specifically, refer to Fig.28, the position of the center sample c in the first component of the current image is (i, j), where i and j are integers. The positions of the two samples in the first of the seven filter shapes are (i-1, j) and (i+1, j). The positions of the two samples in the second of the seven filter shapes are (i-1, j-1) and (i+1, j+1). The positions of the two samples in the third of the seven filter shapes are (i, j-1) and (i, j+1). The positions of the two samples in the fourth of the seven filter shapes are (i+1, j-1) and (i-1, j+1). The positions of the two samples in the fifth of the seven filter shapes are (i, j-3) and (i, j+3). The positions of the two samples in the sixth of the seven filter shapes are (i, j-5) and (i, j+5). The positions of the two samples in the seventh of the seven filter shapes are (i, j-7) and (i, j+7). In one example, the plurality of filter shapes (or filters) used in a sample shift filter (eg, CCSO or LSO) include Fig.28 One or more of the filter shapes 1 to 7 in the filter. According to one embodiment, the plurality of filter shapes include Fig.28 The filter shapes in filters 1 to 6 in .

[0293] The filter or filter shape used in the sample offset filter (e.g., CCSO or LSO) may be asymmetric. In one example, the maximum number of top rows (e.g., rows located above the center sample of the first component co-located with the reconstructed sample to be filtered) and the maximum number of bottom rows (e.g., rows located below the center sample co-located with the reconstructed sample to be filtered) used as input to the sample offset filter (e.g., CCSO or LSO) are different. Fig.29 Examples of filter shapes (or filters) 1 to 4 and 6 to 8 according to embodiments of the present application are shown. Each of the filter shapes (or filters) 1 to 4 and 6 to 8 is a 3-tap filter including a center sample c of the first component and two other samples. Filter shapes 1 to 4 and 6 are symmetrical, wherein the other two samples are positioned symmetrically relative to the center sample c. Filter shapes 7 and 8 are asymmetrical, wherein the other two samples are positioned asymmetrically relative to the center sample c. In an example, when an asymmetric filter (e.g., CCSO or LSO) is used in a sample offset filter (e.g., Fig.29When the filter 7 or 8 in the first component is used, the maximum number of rows (or lines) of top samples relative to the current sample to be filtered (e.g., the current reconstructed sample in the current component) is limited to N. In one example, the current sample is co-located with the center sample c of the first component. As described above, N can be any suitable positive integer. Example values ​​of N can include, but are not limited to, 1 to 1024. In one example, N is less than or equal to 3. In one example, N is 1.

[0294] According to some aspects of embodiments of the present application, the filter (e.g., the position of the filter taps in the filter) or the filter shape used in a sample offset filter (e.g., CCSO or LSO) may depend on the relative position of the reconstructed sample to be filtered (also referred to as the current sample) within the FU (e.g., a 128×128 chroma block).

[0295] In one embodiment, the encoding information of the reconstructed sample of the current component in the filtering unit can be decoded according to the encoded video code stream. The encoding information can indicate the sample offset filter to be applied to the reconstructed sample to be filtered. The filtering shape of the sample offset filter may include the first reconstructed sample of the first component in the current image. FU is in the current image. As described above, the filtering shape can be determined based on the position (or relative position) of the reconstructed sample in FU. The reconstructed sample can be located at the same position as the center sample of the filtering shape. Subsequently, the output value of the sample offset filter can be determined based on the first reconstructed sample of the filtering shape. Then, the filtered sample value of the reconstructed sample of the current component can be determined based on the reconstructed sample and the output value of the sample offset filter.

[0296] In one embodiment, the current component is in a filtering unit in a current image. The filtering shape of the sample offset filter may be determined based on the position of the reconstructed sample in the filtering unit.

[0297] In one embodiment, when a sample offset filter (e.g., CCSO or LSO) is applied, if the sample offset filter has one or more filter taps that exceed the top K rows relative to the current sample to be filtered (or the center sample at the same position as the current sample), the sample offset filter is not allowed. In one example, the sample offset filter is replaced by another sample offset filter that has filter taps within the top K rows relative to the current sample to be filtered. K can be a positive integer. In one example, K is equal to N.

[0298] In one embodiment, when a sample offset filter (e.g., CCSO or LSO) is applied, if the sample offset filter has one or more filter taps and it exceeds the top L rows relative to the current FU (or CTU) to be filtered, the one or more filter taps can be moved to the nearest sample, which is located in the same column but is located within the top L rows relative to the current FU (or CTU) to be filtered or otherwise filled. The top L rows can be equivalent to the samples in the current FU (or CTU). In one example, the top L rows can be equivalent to the top boundary of the current FU (or CTU). For example, if the sample offset filter has one or more filter taps and it exceeds the top L rows relative to the top boundary of the current FU (or CTU) to be filtered, the one or more filter taps can be moved to the nearest sample, which is located in the same column but is located within the top L rows relative to the top boundary of the current FU (or CTU) to be filtered or otherwise filled. L can be an integer. In one example, L is equal to N.

[0299] Fig.30 An example is shown in which one or more filter taps (eg, filter taps 7 and 8 ) are moved according to an embodiment of the present application. Fig.30 Seven filters (or filter shapes) 1 to 4 and 6 to 8 are shown. While filter taps 1 to 4 and 6 are located within the top 4 rows of the current FU (or CTU), such as the top boundary of the current FU, two filter taps 7 and 8 (represented by (3001) to (3002)) are located outside the top 4 rows of the current FU (or CTU). Then, filter taps 7 and 8 are moved to new positions represented by (3003) to (3004). The new positions of filter taps 7 and 8 can be located within the top 4 rows of the current FU, such as within the top 1 row of the current FU (or CTU).

[0300] For example Fig. 20 , Fig. 22 as well as Figures 24 to 30 The example of the filter shown includes a center sample c co-located with the reconstructed sample (or current sample) to be filtered. In general, a sample shift filter may include the center sample or exclude the center sample.

[0301] Fig.31A flowchart outlining a process (3100) according to an embodiment of the present application is shown. The process (3100) can be used to filter samples in a current image, such as samples in a FU of the current image. In various embodiments, the process (3100) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (330) and (340), a processing circuit that performs the function of a video encoder (403), a processing circuit that performs the function of a video decoder (410), a processing circuit that performs the function of a video decoder (510), a processing circuit that performs the function of a video encoder (603), etc. In some embodiments, the process (3100) can be implemented in software instructions, so that when the processing circuit executes the software instructions, the processing circuit performs the process (3100). The process starts at step (S3101) and proceeds to step (S3110).

[0302] In step (3110), the encoding information of the reconstructed sample of the current component in the current image can be decoded according to the encoded video code stream. The encoding information can indicate a sample offset filter to be applied to the reconstructed sample. The filter shape of the sample offset filter can be one of a plurality of filter shapes. Each of the plurality of filter shapes can include a first reconstructed sample of a first component in the current image. In one embodiment, for each of the plurality of filter shapes, the topmost sample in the first reconstructed sample can be located within N rows of a center reconstructed sample (also referred to as a center sample) in the first component that is located at the same position as the reconstructed sample of the current component. N can be less than or equal to 3.

[0303] In one example, the number of first reconstructed samples in each of the plurality of filter shapes is less than five.

[0304] In one example, each of the plurality of filter shapes includes a central reconstruction sample co-located with a reconstruction sample of the current component, and the number of first reconstruction samples in each of the plurality of filter shapes is 3. The first reconstruction sample of each of the plurality of filter shapes may include a central reconstruction sample and two samples symmetrical with respect to the central reconstruction sample.

[0305] In one example, N is 1.

[0306] In one example, the position of the central reconstruction sample in the first component of the current image is (i, j), i and j are integers, and the multiple filter shapes include six filter shapes. The positions of the two samples in the first of the six filter shapes are (i-1, j) and (i+1, j). The positions of the two samples in the second of the six filter shapes are (i-1, j-1) and (i+1, j+1). The positions of the two samples in the third of the six filter shapes are (i, j-1) and (i, j+1). The positions of the two samples in the fourth of the six filter shapes are (i+1, j-1) and (i-1, j+1). The positions of the two samples in the fifth of the six filter shapes are (i, j-4) and (i, j+4). The positions of the two samples in the sixth of the six filter shapes are (i, j-7) and (i, j+7).

[0307] In one example, the position of the central reconstructed sample in the first component of the current image is (i, j), i and j are integers, and the multiple filter shapes include six different filter shapes. The positions of the two samples in the first filter shape are (i-1, j) and (i+1, j). The positions of the two samples in the second filter shape are (i-1, j-1) and (i+1, j+1). The positions of the two samples in the third filter shape are (i, j-1) and (i, j+1). The positions of the two samples in the fourth filter shape are (i+1, j-1) and (i-1, j+1). The positions of the two samples in the fifth filter shape are (i, j-3) and (i, j+3). The positions of the two samples in the sixth filter shape are (i, j-5) and (i, j+5).

[0308] The sample offset filter may be a CCSO filter or a LSO filter.

[0309] In one example, the first component is the second component. In one example, the first component is different from the second component. The first component may be a luminance component and the current component may be a chrominance component.

[0310] In one example, the sample offset filter is an LSO filter, the first component is a current component, and the reconstructed sample of the current component is a central reconstructed sample.

[0311] In step (S3120), an output value of a sample offset filter may be determined based on a first reconstructed sample in a filter shape.

[0312] In step (S3130), a filtered sample value of the reconstructed sample of the current component may be determined based on the reconstructed sample and the output value of the sample offset filter. In various examples, the sample offset filter is a loop filter, through which the output value is applied as an offset to the reconstructed sample to filter out coding artifacts while retaining the details of the current component in the current image. The sample offset filter may include one of the following: LSO, CCSO, and Adaptive Sample Offset filter (ASO). The process (3100) proceeds to step (S3199) and terminates.

[0313] The process (3100) may be adjusted appropriately. Steps in the process (3100) may be modified and / or omitted. Additional steps may be added. Any suitable implementation order may be used. In one example, the current component is in a filter unit in the current image. The filter shape of the sample offset filter may be determined based on the position of the reconstructed sample in the filter unit. The sample offset filter is a CCSO filter and the filter unit is a chroma block having 128×128 chroma samples.

[0314] Fig.32 A flowchart outlining a process (3200) according to an embodiment of the present application is shown. The process (3200) can be used to filter samples in a current image, such as samples in a FU of the current image. In various embodiments, the process (3200) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (330) and (340), a processing circuit that performs the function of a video encoder (403), a processing circuit that performs the function of a video decoder (410), a processing circuit that performs the function of a video decoder (510), a processing circuit that performs the function of a video encoder (603), etc. In some embodiments, the process (3200) can be implemented in software instructions, so that when the processing circuit executes the software instructions, the processing circuit performs the process (3200). The process starts at step (S3201) and proceeds to step (S3210).

[0315] In step (S3210), encoding information of a reconstructed sample of a current component in a filtering unit may be decoded according to an encoded video bitstream. The encoding information may indicate a sample offset filter to be applied to the reconstructed sample. The filtering shape of the sample offset filter may include a first reconstructed sample of a first component in a current image, and the filtering unit is located in the current image.

[0316] In one example, the filtering unit is a chroma block with 128x128 chroma samples.

[0317] In one example, the sample offset filter is a Cross-Component Sample Offset (CCSO) filter.

[0318] In one example, the sample offset filter is a Local Sample Offset (LSO) filter.

[0319] In step (S3220), the filtering shape (eg, the position of each filter tap) may be determined based on the position of the reconstructed sample in the filtering unit, as shown in FIG. Fig.30 described.

[0320] In step (S3230), an output value of the sample offset filter may be determined based on the first reconstructed sample of the filtered shape.

[0321] At step (S3240), filtered sample values ​​of the reconstructed samples of the current component may be determined based on the reconstructed samples and the output value of the sample offset filter. The process (3200) proceeds to step (S3299) and terminates.

[0322] The process (3200) may be adapted as appropriate. Steps in the process (3200) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0323] The embodiments of the present application can be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder and decoder can be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In an example, one or more processors can execute a program stored in a non-transitory computer-readable medium. The embodiments of the present application can be applied to luminance blocks and chrominance blocks.

[0324] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.33 A computer system (3300) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0325] Computer software may be encoded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through decoding, microcode, etc.

[0326] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, IoT devices, etc.

[0327] Fig.33 The components of the computer system (3300) shown are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing embodiments of the present application. The configuration of components should not be interpreted as having any dependency or requirement related to any one or combination of components shown in the exemplary embodiment of the computer system (3300).

[0328] The computer system (3300) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not depicted). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0329] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard (3301), mouse (3302), touch pad (3303), touch screen (3310), data gloves (not shown), joystick (3305), microphone (3306), scanner (3307), camera (3308).

[0330] The computer system (3300) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more human user senses, for example, through tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (3310), a data glove (not shown), or a joystick (3305), but may also be a tactile feedback device that is not an input device), an audio output device (e.g., a speaker (3309), headphones (not depicted)), a visual output device (e.g., a screen (3310) including a CRT screen, an LCD screen, a plasma screen, an OLED screen, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual outputs or outputs exceeding three dimensions through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0331] The computer system (3300) may also include human-accessible storage devices and their associated media: for example, optical media including CD / DVD ROM / RW (3320) with CD / DVD etc. media (3321), thumb drives (3322), removable hard drives or solid-state drives (3323), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), etc.

[0332] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0333] The computer system (3300) may also include an interface (3354) to one or more communication networks (3355). The network may be, for example, a wireless network, a wired network, an optical network. The network may further be a local network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter (e.g., a USB port of the computer system (3300)) connected to some general data port or peripheral bus (3349); as described below, other network interfaces are typically integrated into the kernel of the computer system (3300) by connecting to the system bus (e.g., connected to an Ethernet interface in a PC computer system or connected to a cellular network interface in a smartphone computer system). The computer system (3300) can use any of these networks to communicate with other entities. Such communications may be one-way receive only (e.g., broadcast television), one-way send only (e.g., a CANbus connected to certain CANbus devices), or bidirectional, such as connecting to other computer systems using a LAN or WAN digital network. As described above, certain protocols and protocol stacks may be used on each of those networks and network interfaces.

[0334] The above-mentioned human-machine interface device, human-machine accessible storage device and network interface may be attached to the kernel (3340) of the computer system (3300).

[0335] The core (3340) may include one or more central processing units (CPUs) (3341), graphics processing units (GPUs) (3342), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (3343), hardware accelerators (3344) for certain tasks, graphics adapters (3350), and the like. These devices, as well as read-only memory (ROM) (3345), random access memory (3346), internal mass storage (3347) such as internal non-user accessible hard drives, SSDs, and the like, may be connected via a system bus (3348). In some computer systems, the system bus (3348) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be connected directly to the core's system bus (1748) or to the core's system bus (3348) via a peripheral bus (3349). In an example, a display (3310) may be connected to a graphics adapter (3350). The architecture of the peripheral bus includes PCI, USB, and the like.

[0336] The CPU (3341), GPU (3342), FPGA (3343) and accelerator (3344) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (3345) or RAM (3346). Transition data can also be stored in RAM (3346), while permanent data can be stored, for example, in internal mass storage (3347). Fast storage and retrieval to any storage device can be performed by using a cache, which can be closely associated with the following: one or more CPUs (3341), GPUs (3342), mass storage (3347), ROM (3345), RAM (3346), etc.

[0337] The computer readable medium may have thereon computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of the embodiments of the present application, or the medium and computer code may be of a type known and available to those skilled in the art of computer software.

[0338] As a non-limiting example, a computer system having an architecture (3300), particularly a kernel (3340), can provide functionality due to one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with a user-accessible mass storage as described above, as well as memories of certain non-temporary kernels (3340), such as a kernel internal mass storage (3347) or ROM (3345). Software for implementing various embodiments of the present application can be stored in such devices and executed by the kernel (3340). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the kernel (3340), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (3346) and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality due to logic hardwired or otherwise embodied in a circuit (e.g., an accelerator (3344)) that may replace software or operate in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, a reference to a portion of software may include logic and vice versa. Where appropriate, a reference to a portion of a computer-readable medium may include a circuit (e.g., an integrated circuit (IC)) storing software for execution, a circuit embodying logic for execution, or both. Embodiments of the present application include any suitable combination of hardware and software.

[0339] Appendix A: Abbreviations

[0340] JEM: Joint Exploration Model

[0341] VVC: Versatile Video Coding

[0342] BMS: Benchmark Set

[0343] MV: Motion Vector

[0344] MVP: Motion Vector Prediction

[0345] HEVC: High Efficiency Video Coding

[0346] SEI: Supplemental Enhancement Information

[0347] VUI: Video Availability Information

[0348] GOP: Group of Pictures

[0349] TU: Transformation Unit,

[0350] PU: prediction unit

[0351] CTU: Coding Tree Unit

[0352] CTB: Coding Tree Block

[0353] PBS: Prediction Block

[0354] HRD: Hypothetical Reference Decoder

[0355] SNR: Signal to Noise Ratio

[0356] CPU: Central Processing Unit

[0357] GPU: Graphics Processing Unit

[0358] CRT: cathode ray tube

[0359] LCD: Liquid Crystal Display

[0360] OLED: Organic Light Emitting Diode

[0361] CD: compact disc

[0362] DVD: Digital Video Disc

[0363] ROM: Read Only Memory

[0364] RAM: Random Access Memory

[0365] ASIC: Application-Specific Integrated Circuit

[0366] PLD: Programmable Logic Device

[0367] LAN: Local Area Network

[0368] GSM: Global System for Mobile Communications

[0369] LTE: Long Term Evolution

[0370] CANBus: Controller Area Network Bus

[0371] USB: Universal Serial Bus

[0372] PCI: Peripheral Component Interconnect

[0373] FPGA: Field Programmable Gate Array

[0374] SSD: Solid State Drive

[0375] IC: Integrated Circuit

[0376] HDR: High Dynamic Range

[0377] SDR: Standard Dynamic Range

[0378] JVET: Joint Video Exploration Team

[0379] MPM: Most Probable Mode

[0380] WAIP: Wide Angle Intra Prediction

[0381] CU: Coding Unit

[0382] PU: prediction unit

[0383] TU: Transformation Unit

[0384] CTU: Coding Tree Unit

[0385] PDPC: Position Dependent Prediction Combination

[0386] ISP: Intra-frame sub-splitting

[0387] SPS: Sequence Parameter Settings

[0388] PPS: Picture Parameter Set

[0389] APS: Adaptive Parameter Set

[0390] VPS: Video Parameters Collection

[0391] DPS: Decoding Parameter Set

[0392] ALF: Adaptive Loop Filter

[0393] SAO: Sample Adaptive Offset

[0394] CC-ALF: Cross-Component Adaptive Loop Filter

[0395] CDEF: Constrained Directional Enhancement Filter

[0396] CCSO: Cross-Component Sample Offset Filter

[0397] LSO: Local Sample Offset Filter

[0398] LR: Loop Repair Filter

[0399] AV1: Alliance for Open Media Video 1

[0400] AV2: Alliance for Open Media Video 2

[0401] Although the embodiments of the present application have described multiple exemplary embodiments, there are modifications, permutations, and various replacement equivalents that fall within the scope of the embodiments of the present application. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the embodiments of the present application and therefore fall within the spirit and scope of the embodiments of the present application.

Claims

1. A video decoding method, characterized in that: The method comprises: Decoding, according to the encoded video bitstream, encoding information of a reconstructed sample of a current component in a current image, the encoding information indicating a sample offset filter to be applied to the reconstructed sample, the filter shape of the sample offset filter including a first reconstructed sample of a first component in the current image; determining an output value of the sample offset filter based on a first reconstructed sample in the filter shape indicated by the encoding information; and Based on the reconstructed samples and an output value of the sample offset filter, filtered sample values ​​of the reconstructed samples of the current component are determined.

2. The method according to claim 1, characterized in that For the filter shape, the topmost sample of the first reconstructed sample is located within N rows above the central reconstructed sample in the first component, the reconstructed sample of the current component and the central reconstructed sample of the first component are located at the same position, and N is less than or equal to 3.

3. The method according to claim 2, characterized in that The filter shape includes: the central reconstructed sample located at the same position as the reconstructed sample of the current component, and two samples symmetrical with respect to the central reconstructed sample.

4. The method according to claim 1, characterized in that: The sample offset filter is any one of the following types of filters: a local sample offset filter (LSO), a cross-component sample offset filter (CCSO), and an adaptive sample offset filter (ASO).

5. The method according to claim 4, characterized in that In the case where the sample offset filter is a cross-component sample offset filter CCSO, the first component is a luma component and the current component is a chroma component.

6. The method according to claim 4, characterized in that In the case where the sample offset filter is a local sample offset filter LSO, the first component is the current component, and the reconstructed sample is the central reconstructed sample.

7. The method according to claim 4, characterized in that When the sample offset filter is a cross-component sample offset CCSO filter, the filtering unit is a chroma block having 128x128 chroma samples.

8. The method according to claim 2, characterized in that: The N is 1.

9. The method according to claim 8, characterized in that The position of the central reconstructed sample in the first component of the current picture is (i, j), where i and j are integers, Multiple filter shapes include six filter shapes, The positions of the two samples in the first of the six filter shapes are (i-1, j) and (i+1, j), The positions of the two samples in the second of the six filter shapes are (i-1, j-1) and (i+1, j+1), The positions of the two samples in the third of the six filter shapes are (i, j-1) and (i, j+1), The positions of the two samples in the fourth of the six filter shapes are (i+1, j-1) and (i-1, j+1), The positions of the two samples in the fifth of the six filter shapes are (i, ja) and (i, j+a), and The positions of the two samples in the sixth of the six filter shapes are (i, jb) and (i, j+b), and a is a first integer and b is a second integer different from the first integer.

10. A device for video decoding, characterized in that: include: A processing circuit, wherein the processing circuit is configured to perform the video decoding method according to any one of claims 1 to 9.

11. A video encoding method, characterized in that: The method comprises: determining a filter shape to be applied to reconstructed samples of a current component in a current picture from a plurality of filter shapes of a sample offset filter, the filter shape of the sample offset filter comprising a first reconstructed sample of a first component in the current picture; A video code stream is generated based on samples of the current picture and the multiple filter shapes, the video code stream indicating encoding information of the filter shape of the sample offset filter applied to the reconstructed samples of the current component, the first reconstructed sample in the filter shape indicated by the encoding information is used to determine the output value of the sample offset filter; and the output value is used to determine the filtered sample value of the reconstructed sample of the current component.

12. A device for video encoding, characterized in that: include: A processing circuit, wherein the processing circuit is configured to execute the video encoding method according to claim 11.

13. A non-volatile computer-readable storage medium, characterized in that: The storage medium stores instructions, which, when executed by a computer for video decoding, enable the computer to execute the video decoding method described in any one of claims 1 to 9 to decode the video stream, or enable the computer to execute the video encoding method described in claim 11 to generate a video stream.

14. A method for processing a video code stream, characterized in that: The video code stream can be decoded based on the video decoding method described in any one of claims 1 to 9, or the video code stream can be generated based on the video encoding method described in claim 11.

Images