Enhanced intra prediction with bilateral filter

Abstract

A video decoding method includes receiving a bitstream including encoded information of one or more pictures. The encoded information indicates intra prediction of the current block. The method further includes enabling reference sample filtering with a bilateral filter when the current block satisfies one or more conditions associated with the reference sample filtering; when the reference sample filtering with the bilateral filter is started, applying the reference sample filtering with the bilateral filter to the reference sample of the current block to obtain a filtered reference sample; and reconstructing one or more samples of the current block according to intra prediction using the filtered reference samples.

Description

[0001] References This application claims priority to U.S. Patent Application No. 18 / 912,307, filed October 10, 2024, entitled "Enhanced Intra Prediction with Bilateral Filter," and U.S. Provisional Application No. 63 / 544,118, filed October 13, 2023, entitled "Enhanced Intra Prediction with Bilateral Filter." The disclosure of the earlier applications is incorporated herein by reference in its entirety. Technical Field

[0002] This application describes aspects generally related to video coding. Background Technology

[0003] The background description provided herein is for the purpose of presenting the general content of this application. The extent of the work performed by the currently named inventors as described in this background section and in various aspects of this specification does not imply that it was qualified as prior art at the time of filing, nor is it expressly or implied that it was recognized as prior art of this application.

[0004] Image / video compression helps transmit image / video data across different devices, storage systems, and networks with minimal quality degradation. In some examples, video codecs can compress video based on spatial and temporal redundancy. In one example, a video codec can use a technique called intra-frame prediction, which compresses images based on spatial redundancy. For example, intra-frame prediction can use reference data from the current image being reconstructed to predict samples. In another example, a video codec can use a technique called inter-frame prediction, which compresses images based on temporal redundancy. For example, inter-frame prediction can predict samples in the current image from an image previously reconstructed using motion compensation. Motion compensation can be indicated by motion vectors (MV). Summary of the Invention

[0005] This disclosure includes aspects such as bitstreams, methods and apparatus for video encoding / decoding. In some examples, an apparatus for video encoding / decoding includes processing circuitry.

[0006] Some aspects of this disclosure provide a video decoding method. The method includes: receiving a bitstream comprising encoded information of one or more images. The encoded information indicates intra-frame prediction for a current block. The method further includes: enabling reference sample filtering when the current block satisfies one or more conditions associated with reference sample filtering with a bilateral filter; when enabling reference sample filtering with a bilateral filter, applying the reference sample filtering with a bilateral filter to reference samples of the current block to obtain filtered reference samples; and reconstructing one or more samples of the current block based on intra-frame prediction using the filtered reference samples.

[0007] Some aspects of this disclosure provide a video coding method. The method includes: determining to encode a current block using intra-frame prediction; enabling reference sample filtering when the current block satisfies one or more conditions associated with reference sample filtering with a bilateral filter; when reference sample filtering with a bilateral filter is enabled, applying the reference sample filtering with a bilateral filter to reference samples of the current block to obtain filtered reference samples; and encoding the current block as encoded information in a bitstream using intra-frame prediction based on the filtered reference samples.

[0008] Some aspects of this disclosure also provide an apparatus for video encoding / decoding.

[0009] Some aspects of this disclosure also provide a method for processing visual media data. In this method, a bitstream of visual media data is processed according to format rules. For example, the bitstream may be a bitstream decoded / encoded using any decoding and / or encoding method described in this application. The format rules may specify one or more constraints on the bitstream and / or one or more processes to be performed by the decoder and / or encoder.

[0010] This disclosure also provides a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the described methods for video decoding / encoding. Attached Figure Description

[0011] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which: Figure 1 This is a schematic illustration of a block diagram example of a communication system (100).

[0012] Figure 2 This is a schematic diagram of a block diagram example of a decoder.

[0013] Figure 3 This is a schematic diagram of an example block diagram of an encoder.

[0014] Figure 4 Illustrations of intra-frame prediction modes are shown in some examples.

[0015] Figure 5 Illustrations of intra-frame prediction modes are shown in some examples.

[0016] Figure 6 Illustrations of several reference lines are shown in some examples.

[0017] Figure 7 A flowchart outlining some aspects of the decoding process according to this application is shown.

[0018] Figure 8 A flowchart outlining some aspects of the coding process according to this application is shown.

[0019] Figure 9 It is a schematic diagram based on one aspect of a computer system. Detailed Implementation

[0020] Figure 1 Block diagrams of some examples of video processing systems (100) are shown. The video processing system (100) is an application example of the disclosed subject matter, namely a video encoder and video decoder located in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0021] The video processing system (100) includes an acquisition subsystem (113) that may include a video source (101) such as a digital camera, which creates, for example, an uncompressed video image stream (102). In an embodiment, the video image stream (102) includes samples captured by a digital camera. The video image stream (102) is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data (104) (or encoded video bitstream). The video image stream (102) may be processed by an electronic device (120) including a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (102), the encoded video data (104) (or the encoded video bitstream 104) is depicted as a thin line to emphasize the lower data volume of the encoded video data (104) (or the encoded video bitstream 104), which can be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as Figure 1Client subsystems (106) and (108) can access a streaming server (105) to retrieve copies (107) and (109) of encoded video data (104). Client subsystem (106) may include, for example, a video decoder (110) in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and produces an output video picture stream (111) that can be displayed on a display (112) (e.g., a screen) or another presentation device (not indicated). In some streaming systems, the encoded video data (104), video data (107), and video data (109) (e.g., a video stream) may be encoded according to certain video coding / compression standards. Embodiments of these standards include ITU-T Recommendation H.265. In embodiments, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of the VVC standard.

[0022] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0023] Figure 2 A block diagram example of a video decoder (210) is shown. The video decoder (210) may be disposed in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., receiving circuitry). The video decoder (210) may be used in place of... Figure 1 The video decoder (110) in the embodiment.

[0024] The receiver (231) may receive, for example, one or more encoded video sequences included in the bitstream that will be decoded by the video decoder (210). In one aspect, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (231) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (231) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter referred to as "parser (220)"). In some applications, the buffer memory (215) is part of the video decoder (210). In other cases, the buffer memory (215) may be located externally to the video decoder (210) (not shown). In other cases, a buffer memory (not shown) may be located externally to the video decoder (210) to prevent network jitter, for example, and another buffer memory (215) may be configured internally to handle broadcast timing, for example. When the receiver (231) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer memory (215) may not be necessary, or it may be made smaller. Of course, for use on packet networks such as the Internet, the buffer memory (215) may also be required; it may be relatively large and advantageously have an adaptive size, and may be at least partially implemented in the operating system or a similar element (not shown) external to the video decoder (210).

[0025] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (210) and potential information for controlling a display device (212) (e.g., a display screen), which is not part of the electronic device (230) but may be coupled to it, such as... Figure 2As shown in the figure. The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (220) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels in the encoded video sequence based on at least one parameter corresponding to a group. The subgroup may include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (220) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0026] The parser (220) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to create symbols (221).

[0027] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (221) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (220). For brevity, the flow of such subgroup control information between the parser (220) and the various units described below is not described.

[0028] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0029] The first unit is the scaler / inverse transform unit (251). The scaler / inverse transform unit (251) receives quantization transform coefficients as symbols (221) and control information from the parser (220), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (251) can output a block containing sample values, which can be input into the aggregator (255).

[0030] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to an intra-coded block. An intra-coded block is a block that does not use predictive information from a previously reconstructed image, but can use predictive information from a previously reconstructed portion of the current image. Such predictive information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) uses reconstructed information extracted from the current picture buffer (258) to generate a surrounding block of the same size and shape as the reconstructed block. For example, the current picture buffer (258) buffers a partially reconstructed current image and / or a fully reconstructed current image. In some cases, the aggregator (255) adds the predictive information generated by the intra-picture prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251) based on each sample.

[0031] In other cases, the output samples of the scaler / inverse transform unit (251) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (253) may access the reference image memory (257) to extract samples for prediction. After motion compensation of the extracted samples according to the symbols (221) belonging to the block, these samples may be added by the aggregator (255) to the output of the scaler / inverse transform unit (251) (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit (253) may obtain the predicted samples from the address in the reference image memory (257) under motion vector control, and the motion vector is available to the motion compensation prediction unit (253) in the form of the symbols (221), which may include, for example, X components, Y components and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (257) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0032] The output samples of the aggregator (255) can be employed by various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (221) from the parser (220) in the loop filter unit (256). Video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded image or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0033] The output of the loop filter unit (256) can be a sample stream, which can be output to a display device (212) and stored in a reference image memory (257) for subsequent inter-frame image prediction.

[0034] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (220)) is identified as the reference image, the current image buffer (258) can become part of the reference image memory (257), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0035] The video decoder (210) can perform decoding operations according to a predetermined video compression technique or standard, such as ITU-T H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0036] On one hand, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0037] Figure 3 A block diagram example of a video encoder (303) is shown. The video encoder (303) is disposed in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., transmission circuitry). The video encoder (303) can be used in place of Figure 1 The video encoder (103) in the embodiment.

[0038] The video encoder (303) can be obtained from the video source (301) (not Figure 3 In one embodiment, the electronic device (320) receives video samples, and the video source (301) can capture video images to be encoded by the video encoder (303). In another embodiment, the video source (301) is part of the electronic device (320).

[0039] A video source (301) can provide a sequence of source video samples to be encoded by a video encoder (303) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (301) can be a storage device storing previously prepared video. In a video conferencing system, the video source (301) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc. used. The following focuses on describing the samples.

[0040] According to one aspect, the video encoder (303) can encode and compress the images of the source video sequence into an encoded video sequence (343) in real time or under any other required time constraints. Implementing an appropriate encoding rate is a function of the controller (350). In some aspects, the controller (350) controls and is functionally coupled to other functional units as described below. For clarity, coupling is not indicated in the figures. Parameters set by the controller (350) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be used with other suitable functions related to the video encoder (303) optimized for a particular system design.

[0041] In some aspects, the video encoder (303) operates within an encoding loop. As a simplified description, in an embodiment, the encoding loop may include a source encoder (330) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (333) embedded within the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data. The reconstructed sample stream (sample data) is input to a reference image memory (334). Since decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image memory (334) are also bit-accurately corresponding between the local and remote encoders. In other words, the reference image samples "seen" by the encoder's prediction section are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs, for example, when synchronization cannot be maintained due to channel errors) is also used in some related techniques.

[0042] The operation of the “local” decoder (333) can be combined with, for example, the above-mentioned... Figure 2 The video decoder (210) described in detail is the same as the "remote" decoder. However, a further brief reference is provided. Figure 2 When symbols are available and the entropy encoder (345) and parser (220) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (210), including the buffer (215) and parser (220), may not be fully implemented in the local decoder (333).

[0043] In one respect, any decoder technique other than parsing / entropy decoding present in the decoder also exists in the corresponding encoder in the same or substantially the same functional form. For this reason, the disclosed subject focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. In certain areas, more detailed descriptions are provided below.

[0044] During operation, in some embodiments, the source encoder (330) may perform motion-compensated predictive coding, referencing one or more previously encoded images from the video sequence designated as "reference images," which predictively encode the input image. In this manner, the encoding engine (332) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

[0045] The local video decoder (333) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (330). The operation of the encoding engine (332) can advantageously be a lossy process. When encoded video data can be decoded by the video decoder (333), Figure 3 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (333) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image memory (334). In this way, the video encoder (303) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0046] The predictor (335) can perform a prediction search against the encoding engine (332). That is, for a new image to be encoded, the predictor (335) can search in the reference image memory (334) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (335) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (335), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (334).

[0047] The controller (350) can manage the encoding operations of the source encoder (330), including, for example, setting parameters and subgroup parameters for encoding video data.

[0048] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (345). The entropy encoder (345) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into an encoded video sequence.

[0049] The transmitter (340) can buffer the encoded video sequence created by the entropy encoder (345) in preparation for transmission via a communication channel (360), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (340) can combine the encoded video data from the video encoder (303) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0050] The controller (350) manages the operation of the video encoder (303). During encoding, the controller (350) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types.

[0051] Intra-frame pictures (I-pictures) are pictures that can be encoded and decoded without using any other pictures in the sequence as prediction sources. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures.

[0052] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses motion vectors and reference indices to predict sample values ​​for each block.

[0053] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0054] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined based on the coding assignment of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0055] The video encoder (303) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (303) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0056] In one aspect, the transmitter (340) may transmit additional data while transmitting encoded video. The source encoder (330) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, SEI messages, VUI parameter set fragments, etc.

[0057] The acquired video can serve as multiple source images (video images) presented in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0058] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to this bidirectional prediction technique, two reference images are used, such as a first reference image and a second reference image that precede the current image in the video in decoding order (but may be past and future in display order, respectively). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted using a combination of the first and second reference blocks.

[0059] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0060] According to some aspects of this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video image sequence are segmented into coding tree units (CTUs) for compression. The CTUs in the images have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU comprises three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Furthermore, each CTU can be divided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one embodiment, each CU is analyzed to determine the prediction type used for the CU, such as inter-frame prediction or intra-frame prediction. Depending on temporal and / or spatial predictability, CUs are split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In one aspect, the prediction operation in encoding (encoding / decoding) is performed on a per-prediction-block basis. Taking the luma prediction block as an example, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0061] It should be noted that any suitable technology can be used to implement the video encoder (103) and video encoder (303), as well as the video decoder (110) and video decoder (210). In one aspect, one or more integrated circuits can be used to implement the video encoder (103) and video encoder (303), as well as the video decoder (110) and video decoder (210). In another aspect, one or more processors executing software instructions can be used to implement the video encoder (103) and video encoder (303), as well as the video decoder (110) and video decoder (210).

[0062] Some aspects of this application provide techniques for enhancing intra-frame prediction with bilateral filtering.

[0063] Figure 4 Illustrations of intra-prediction modes (e.g., HEVC) in some examples are shown. For instance, HEVC uses a total of 35 intra-prediction modes (e.g., from mode 0 to mode 34). Of these 35 intra-prediction modes, some are directional modes, and others are non-directional modes. In some examples, mode 0 and mode 1 are non-directional modes; for example, mode 0 is a planar mode, and mode 1 is a DC mode. Furthermore, modes 2 through 34 can be directional modes; for example, mode 10 is a horizontal mode, mode 26 is a vertical mode, and modes 2, 18, and 34 are diagonal modes, and so on. The values ​​of samples in a coding block are determined based on neighboring reference samples in the same image and the intra-prediction mode of the coding block. In one example, in DC mode, the mean is calculated by averaging reference samples in the same image, which can be used for flat surfaces. In another example, in planar mode, the value of each sample in the coding block is calculated under the assumption that the horizontal and vertical smoothing gradients of the amplitude plane are derived from boundary samples of neighboring blocks. In some examples, reference samples include neighboring samples in the row immediately above the coded block and / or neighboring samples in the column immediately to the left of the coded block.

[0064] In some examples, intra-prediction modes are represented by signals based on a list of most probable modes (MPMs) and remaining modes. For example, for a coded block, the MPM list is determined. In one example, the MPM list includes three of 35 intra-prediction modes. Then, when a specific intra-prediction mode of the coded block is one of the three modes in the MPM list, an index indicating one of those three modes is used for signaling. When a specific intra-prediction mode of the coded block is not one of the three modes in the MPM list, an index indicating one of the remaining modes (32 modes) is used for signaling. In some examples, the MPM list may include other suitable numbers of modes, such as 6, 10, etc.

[0065] It should be noted that other appropriate numbers of intra-frame prediction modes can be used.

[0066] Figure 5 The diagram illustrates some examples of intra-prediction modes (e.g., VVC). In some examples, VVC can use a total of 95 intra-prediction modes, such as from mode... Modes 14 through 80. Of these 95 intra-frame prediction modes, mode 0 is a planar mode, mode 1 is a DC mode, mode 18 is a horizontal mode, mode 50 is a vertical mode, and modes 2, 34, and 66 are diagonal modes. In some examples, mode... 1 to mode Modes 14 and 67 through 80 are referred to as wide-angle intra prediction (WAIP) modes.

[0067] In some examples, to encode the intra-mode (also known as intra-prediction mode) of a coding block (e.g., a luma block or chroma block of a coding unit), a most probable mode (MPM) list of size 3 is constructed based on the intra-modes of neighboring blocks of the coding block. This MPM list may be referred to as the MPM list or the master MPM list. If the intra-mode of a coding block is not in the MPM list, a signal is used to indicate whether the intra-mode belongs to a selected mode in the MPM list.

[0068] In some relevant examples, the reference samples used for intra-frame prediction of the coded block are located immediately adjacent to the coded block. The portion of the coded block consisting of these reference samples is called the reference line.

[0069] According to one aspect of this application, a technique known as multi-line intra-prediction can use more reference lines for intra-prediction. In some examples, the encoder determines and uses signals to indicate which reference line to use to generate the intra-predictor. In one example, a signal is used to indicate the reference line index before the intra-prediction mode, and the most probable mode is allowed only when a signal is used to indicate a non-zero reference line index.

[0070] Figure 6 Several reference lines are shown in some examples. Figure 6The diagram shows four reference lines for the coded block (610). The first reference line (620), also called reference line 0, comprises a row of reference samples immediately above the coded block (610) and a column of reference samples immediately to the left of the coded block (610). The second reference line (630), also called reference line 1, comprises a row of reference samples immediately above the first reference line (620) and a column of reference samples immediately to the left of the first reference line (620). The third reference line (640), also called reference line 2, comprises a row of reference samples immediately above the second reference line (630) and a column of reference samples immediately to the left of the second reference line (630). The fourth reference line (650), also called reference line 3, comprises a row of reference samples immediately above the third reference line (640) and a column of reference samples immediately to the left of the third reference line (640). Furthermore, each reference line consists of six segments (e.g., segments A to F) and a reference sample T in the upper left corner. In some examples, segments A and F are filled using the most recent samples from segments B and E, respectively.

[0071] In intra-frame prediction in video codecs (e.g., VVC and ECM), reference samples from neighboring reconstructed regions can be filtered using some predefined filters (e.g., [1, 2, 1] / 4) to generate the final predictor, or they can be directly copied as a predictor based on the intra-frame prediction mode, block size, etc., without filtering. The predefined filters have fixed filtering coefficients that are predefined.

[0072] In some examples, reference sample filtering is applied to neighboring reference samples of the current block. Reference sample filtering is also called pre-filtering because it is applied to neighboring reference samples before intra-frame prediction, while post-filtering is applied to the predicted samples after intra-frame prediction. In a relevant example, a predefined filter [1, 2, 1] / 4 can be used.

[0073] For example, the filtered sample value p(x, y) is derived from equations (1) to (5), where refUnfilt represents the unfiltered neighboring reference sample, and (x, y) represents the x and y coordinates of the corresponding sample. For example, the coordinates of the top-left sample position in the current block are (0, 0). In this example, the reference region includes: x = 1, y= 1 to refH The reference line for 1, and x=0 to refW 1, y= The reference column is 1. In this example, refH can be determined as twice the height of the transform block or coded block, and refW can be determined as twice the width of the transform block or coded block.

[0074] .

[0075] It should be noted that when filtering is applied to the neighboring reference samples, the filtered neighboring reference samples can be used as reference samples in the prediction sample derivation step to generate the final predictor; while when filtering is not applied to the neighboring reference samples, the unfiltered neighboring reference samples can be used as reference samples in the prediction sample derivation step to generate the final predictor.

[0076] According to one aspect of this application, a bilateral filter can be used in reference sample filtering. A bilateral filter is a non-linear, edge-preserving, noise-reducing smoothing filter for images. A bilateral filter can use a weighted average of the intensities of nearby pixels instead of the intensity of each pixel. The weights can be based on a Gaussian distribution. For example, the weights depend not only on the Euclidean distance between pixel samples but also on radiative differences (e.g., range differences, such as color intensity, depth distance, etc.). In a bilateral filter, the filtering weights decrease not only with the distance between samples but also with the increase in intensity differences. Therefore, the over-smoothing of edges can be improved. In one example, the weights can be defined according to equation (6): .

[0077] in, and These are the distances in the vertical and horizontal directions, respectively. This represents the intensity difference between samples. In some examples, the bilateral filter for edge-preserving denoising uses low-pass Gaussian filters for both the domain filter and the range filter. The domain low-pass Gaussian filter assigns higher weights to pixels spatially closer to the center pixel. The range low-pass Gaussian filter assigns higher weights to pixels similar to the center pixel. Combining the range filter and the domain filter, the bilateral filter at the edge pixels forms a Gaussian filter that extends along the edge direction and is significantly reduced in the gradient direction. Thus, the bilateral filter can smooth noise while preserving edge structure.

[0078] Several aspects of this application provide techniques for further enhancing intra-frame prediction with bilateral filters. These techniques can be used individually or in combination in any order. Furthermore, these techniques can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). It should be noted that these techniques can be used with a variety of video encoders, including but not limited to those mentioned in this application.

[0079] In some relevant examples, for intra-frame prediction, samples from neighboring reconstructed regions are used as reference samples to generate the predictor for the current block. The reference samples can be directly copied into the predictor without any filtering, or they can be filtered using a predefined filter with fixed filtering coefficients (e.g., a [1, 2, 1] / 4 filter). However, the final predictor may still contain reconstruction noise or may exceed smooth edges.

[0080] According to one aspect of this application, a bilateral filter is used for filtering reference samples in intra-frame prediction. The bilateral filter can be used to replace other reference sample filters, such as [1, 2, 1] / 4 filters, or to work serially or in parallel with other reference sample filters. When working serially with another reference sample filter, the bilateral filter can be introduced as an additional filtering step to generate the final predictor. When working in parallel with other reference sample filters, the bilateral filter and the other reference sample filters are executed independently.

[0081] It should be noted that the reference sample filtering technique in this application is applicable to each signal color component, all signal color components, or any combination of signal color components.

[0082] According to some aspects of this application, a conditional bilateral filtering stage (bilateral filter) for intra-frame prediction is used to generate the final predictor for the current block. For example, when it is determined that the current block will be encoded / decoded via intra-frame prediction, the encoder / decoder may enable reference sample filtering with a bilateral filter if the current block satisfies one or more conditions associated with reference sample filtering. Then, when reference sample filtering with a bilateral filter is enabled, the reference sample filtering with a bilateral filter can be applied to the reference samples of the current block to obtain filtered reference samples. The filtered reference samples are then used for encoding / decoding the current block via intra-frame prediction.

[0083] Various conditions can be used to determine the reference sample filtering configuration, which is used to process reference samples for intra-frame prediction.

[0084] In the first conditional example, for purely vertical and / or horizontal modes, the reference sample can be directly copied without any filtering, while all other modes are filtered directly through the reference sample with a bilateral filter or filtered based on certain additional conditions.

[0085] In the second conditional example, the condition for enabling reference sample filtering with a bilateral filter for the reference sample is when the intra-frame mode directly uses the reference sample falling at an integer position, such as intra-frame mode 0 (PLANAR_IDX) in VVC, intra-frame mode 14. In-frame mode 12. In-frame mode 10. In-frame mode 6. Intra-frame mode 2, Intra-frame mode 34, Intra-frame mode 66, Intra-frame mode 72, Intra-frame mode 76, Intra-frame mode 78 or Intra-frame mode 80.

[0086] In the third conditional example, the condition for enabling filtering for the reference sample (e.g., reference sample filtering with a bilateral filter) is that no multi-reference line intra-frame prediction is applied. For example, the reference sample is located in the row above the current block or in the column to the left of the current block.

[0087] In the fourth conditional example, the condition for enabling filtering for the reference sample (e.g., reference sample filtering with a bilateral filter) is that no intra-fractional sub-partition prediction is applied. For example, the current block has no further partitions.

[0088] In the fifth conditional example, the condition for enabling filtering for a reference sample (e.g., reference sample filtering with a bilateral filter) is that the block size is greater than a certain number of samples.

[0089] In the sixth conditional example, the condition for enabling filtering for the reference sample (e.g., reference sample filtering with a bilateral filter) is that the prediction block is located within the luma component. For example, no chroma component is associated with the prediction block, or separate trees are used for the luma and chroma components, and the prediction block is associated with the luma tree.

[0090] In the seventh conditional example, the condition for enabling filtering for the reference sample (e.g., reference sample filtering with a bilateral filter) is a combination of the conditions from the first to the sixth examples.

[0091] In the eighth conditional example, the condition for enabling filtering for a reference sample (e.g., reference sample filtering with a bilateral filter) is to use an explicit signal to indicate the use of the bilateral filter.

[0092] According to a first aspect of this application, a predictor for intra-frame prediction is generated using only bilateral filtering in reference sample filtering with bilateral filters. Whether to use a bilateral filter depends on some internal information of the encoder or decoder, such as one or more conditions or any combination of conditions in the first to eighth conditional examples.

[0093] According to a second aspect of this application, in reference sample filtering with a bilateral filter, two filters (including a bilateral filter and a predefined filter) are used to generate the predictor. The bilateral filter can be applied before or after the predefined filter. In one example, the reference sample filtering includes a first stage of a serially connected bilateral filter and a second stage of a predefined filter. The bilateral filter can be placed before or after the predefined filter. Whether to use reference sample filtering with a bilateral filter also depends on some internal information of the encoder or decoder, such as one or more conditions or any combination of conditions in the first to eighth conditional examples. In some examples, the predefined filter is a [1, 2, 1] / 4 filter.

[0094] According to a third aspect of this application, in reference sample filtering, a predictor is generated using two filters (including a bilateral filter and a predefined filter), with the bilateral filter and the predefined filter executed independently. In one example, reference sample filtering includes a bilateral filter and a predefined filter connected in parallel. In one example, one or more conditions from the first to the eighth conditional examples, or any combination of conditions, can be used to determine whether to use reference sample filtering or a bypass path (without using any filtering). When it is determined that reference sample filtering is enabled, which filter (e.g., a bilateral filter or a predefined filter) is used depends on some internal or signal representation information. In one example, when the size of the current block is less than a threshold (e.g., 64), a predefined filter (e.g., a [1, 2, 1] / 4 filter) is used to generate the predictor. Otherwise, a bilateral filter is applied. In another example, when the block content variance (e.g., luminance variance, chrominance variance, etc.) is less than a threshold, a predefined filter (e.g., a [1, 2, 1] / 4 filter) is used to generate the predictor. Otherwise, a bilateral filter is applied.

[0095] According to a fourth aspect of this application, a predictor is generated using two filters (including a bilateral filter and a predefined filter). Whether to use the two filters can be determined using conditions (e.g., one or more conditions in the first to eighth condition examples, or any combination of conditions). When it is determined that no filtering is to be applied to the reference sample, the bilateral filter is used to filter the reference sample. Otherwise, the predefined filter is used.

[0096] According to a fifth aspect of this application, a predictor is generated using two filters: a bilateral filter and a predefined filter. The bilateral filter is applied before all subsequent processing. Based on conditions (e.g., one or more conditions in the first to eighth conditional examples, or any combination of conditions), the output of the bilateral filter can either be filtered by the predefined filter or directly copied as the predictor.

[0097] According to a sixth aspect of this application, a predictor is generated using two filters (including a bilateral filter and a predefined filter). The bilateral filter is applied before the final predictor is generated. Therefore, based on one or more conditions (e.g., one or more conditions in the first to eighth condition examples, or any combination of conditions), a reference sample is first filtered by the predefined filter or used directly. The output of the predefined filter or the unfiltered reference sample is then filtered by the bilateral filter.

[0098] According to the seventh aspect of this application, when non-traditional intra-prediction modes (including but not limited to decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient-location-dependent intra-prediction) are enabled, the techniques in the first to sixth aspects can be used.

[0099] According to the eighth aspect of this application, a signal can be used to represent a syntax element, the syntax element indicating whether a bilateral filter is used, and the syntax element can be a high-level syntax (HLS) (e.g., a sequence parameter set (SPS), a video parameter set (VPS), a picture parameter set (PPS), an adaptation parameter set (APS), a picture header, a slice header) and / or can be a block-level syntax element (e.g., CTU, CU, PU, ​​TU).

[0100] Figure 7 A flowchart outlining a process (700) according to one aspect of this application is shown. The process (700) can be used with a video decoder. In various aspects, the process (700) is executed by processing circuitry, such as processing circuitry that performs the functions of video decoder (110), processing circuitry that performs the functions of video decoder (210), etc. In some aspects, the process (700) is implemented in software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes the process (700). The process begins at (S701) and proceeds to (S710).

[0101] At (S710), a bit stream is received, which includes encoded information of one or more images, the encoded information indicating intra-frame prediction of the current block in the current image.

[0102] At (S720), the reference sample filtering is enabled when the current block satisfies one or more conditions associated with the reference sample filtering with a bilateral filter.

[0103] At (S730), when reference sample filtering with a bilateral filter is enabled, the reference sample with the bilateral filter is applied to the reference sample of the current block to obtain the filtered reference sample.

[0104] At (S740), one or more samples of the current block are reconstructed by intra-frame prediction based on the filtered reference samples.

[0105] In some examples, when the current block does not meet one or more conditions, one or more samples of the current block are reconstructed by intra-frame prediction based on unfiltered reference samples.

[0106] According to some aspects of this application, one or more conditions include at least one of the following: the prediction mode of intra-frame prediction is not a vertical mode; the prediction mode of intra-frame prediction is not a horizontal mode; the prediction mode uses reference samples at integer positions; intra-frame prediction with multiple reference lines is not applied; sub-partition prediction is not applied; the block size of the current block is greater than a threshold; the current block is a prediction block within the luma component; and / or, a flag associated with the current block is decoded from the bitstream, and the flag indicates the use of a bilateral filter.

[0107] In some examples, reference sample filtering includes a single filtering stage of a bilateral filter.

[0108] In some examples, reference sample filtering uses a bilateral filter and a predefined filter with fixed filter coefficients, which are connected in series.

[0109] In some examples, the reference sample filtering uses a bilateral filter and a predefined filter with fixed filter coefficients, which are connected in parallel. In one example, the predefined filter is applied in the reference sample filtering when the block size of the current block is less than a threshold; and the bilateral filter is applied in the reference sample filtering when the block size of the current block is equal to or greater than the threshold. In another example, the predefined filter is applied in the reference sample filtering when the block content variance of the current block is less than a threshold; and the bilateral filter is applied in the reference sample filtering when the block content variance of the current block is equal to or greater than the threshold.

[0110] In some examples, reference sample filtering includes a bilateral filter and / or a predefined filter with fixed filter coefficients. A bilateral filter is applied to the reference sample to obtain a filtered reference sample when the current block does not meet one or more conditions; and a predefined filter is applied to the reference sample to obtain a filtered reference sample when the current block meets one or more conditions.

[0111] In some examples, reference sample filtering includes a bilateral filter and / or a predefined filter with fixed filter coefficients. A bilateral filter is applied to the reference sample to obtain a first filtered reference sample. When the current block does not meet one or more conditions, one or more current samples of the current block are reconstructed based on intra-frame prediction using the first filtered reference sample; and when the current block meets one or more conditions, one or more current samples of the current block are reconstructed based on intra-frame prediction using a second filtered reference sample, wherein the second filtered reference sample is obtained by applying a predefined filter to the first filtered reference sample.

[0112] In some examples, reference sample filtering includes a bilateral filter and / or a predefined filter with fixed filter coefficients. When the current block meets one or more conditions, the predefined filter is applied to the reference sample to obtain an intermediate filtered reference sample. When the current block does not meet one or more conditions, the reference sample is copied as an intermediate filtered reference sample. A bilateral filter is applied to the intermediate filtered reference sample to obtain a filtered reference sample, and the current block is reconstructed based on intra-frame prediction using the filtered reference sample.

[0113] In some examples, it is checked whether at least one of the following is enabled: decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction. When at least one of decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction is enabled, it is determined whether the current block satisfies one or more conditions associated with reference sample filtering including bilateral filters.

[0114] In some examples, a flag is decoded from the bitstream, and the flag is one of the following: Sequence Parameter Set (SPS) level, Video Parameter Set (VPS) level, Picture Parameter Set (PPS) level, Adaptive Parameter Set (APS) level, Picture Header level, Slice Header level, and / or Block level. When the flag indicates that reference sample filtering with a bilateral filter is used, reference sample filtering with a bilateral filter is applied to the reference sample to obtain the filtered reference sample.

[0115] Then, the process proceeds to (S799) and terminates.

[0116] The process (700) may be adjusted as appropriate. One (or more) steps in the process (700) may be modified and / or omitted. One (or more) additional steps may be added. Any suitable implementation order may be used.

[0117] Figure 8A flowchart outlining a process (800) according to one aspect of this application is shown. The process (800) can be used with a video decoder. In each aspect, the process (800) is executed by a processing circuit, such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), etc. In one aspect, the process (800) is implemented in software instructions, so that when the processing circuit executes the software instructions, the processing circuit executes the process (800). The process begins at (S801) and proceeds to (S810).

[0118] At (S810), it is determined that intra-frame prediction will be used to encode the current block.

[0119] At (S820), the reference sample filtering is enabled when the current block satisfies one or more conditions associated with the reference sample filtering with a bilateral filter.

[0120] At (S830), when reference sample filtering with a bilateral filter is enabled, the reference sample with the bilateral filter is applied to the reference sample of the current block to obtain the filtered reference sample.

[0121] At (S840), based on the filtered reference sample, the current block is encoded into the encoded information in the bitstream through intra-frame prediction.

[0122] In some examples, when the current block does not meet one or more conditions, the current block is encoded by intra-frame prediction based on unfiltered reference samples.

[0123] According to some aspects of this application, one or more conditions include at least one of the following: the prediction mode of intra-frame prediction is not a vertical mode; the prediction mode of intra-frame prediction is not a horizontal mode; the prediction mode uses reference samples at integer positions; intra-frame prediction with multiple reference lines is not applied; sub-partition prediction is not applied; the block size of the current block is greater than a threshold; and / or, the current block is a prediction block within the luma component.

[0124] In some examples, reference sample filtering includes a single filtering stage of a bilateral filter.

[0125] In some examples, reference sample filtering uses a bilateral filter and a predefined filter, which are connected in series.

[0126] In some examples, the reference sample filtering uses a bilateral filter and a predefined filter, which are connected in parallel. In one example, the predefined filter is applied in the reference sample filtering when the block size of the current block is less than a threshold; and the bilateral filter is applied in the reference sample filtering when the block size of the current block is equal to or greater than the threshold. In another example, the predefined filter is applied in the reference sample filtering when the block content variance of the current block is less than a threshold; and the bilateral filter is applied in the reference sample filtering when the block content variance of the current block is equal to or greater than the threshold.

[0127] In some examples, reference sample filtering includes bilateral filters and / or predefined filters. When the current block does not meet one or more conditions, a bilateral filter is applied to the reference sample to obtain a filtered reference sample; conversely, when the current block meets one or more conditions, a predefined filter is applied to the reference sample to obtain a filtered reference sample.

[0128] In some examples, reference sample filtering includes bilateral filters and / or predefined filters. A bilateral filter is applied to the reference sample to obtain a first filtered reference sample; when the current block does not meet one or more conditions, the current block is encoded using intra-frame prediction based on the first filtered reference sample; and when the current block meets one or more conditions, the current block is encoded using intra-frame prediction based on a second filtered reference sample, wherein the second filtered reference sample is obtained by applying a predefined filter to the first filtered reference sample.

[0129] In some examples, reference sample filtering includes bilateral filters and / or predefined filters. When the current block meets one or more conditions, a predefined filter is applied to the reference sample to obtain an intermediate filtered reference sample; when the current block does not meet one or more conditions, the reference sample is copied as an intermediate filtered reference sample; a bilateral filter is applied to the intermediate filtered reference sample to obtain a filtered reference sample.

[0130] In some examples, it is checked whether at least one of the following is enabled: decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction. When at least one of decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction is enabled, it is determined whether the current block satisfies one or more conditions associated with reference sample filtering including bilateral filters.

[0131] In some examples, when a reference sample filter with a bilateral filter is applied, a flag is encoded into the bitstream indicating the use of the reference sample filter with a bilateral filter, and the flag may be one of the following: Sequence Parameter Set (SPS) level, Video Parameter Set (VPS) level, Picture Parameter Set (PPS) level, Adaptive Parameter Set (APS) level, Picture Header level, Slice Header level, and / or Block level.

[0132] Then, the process proceeds to (S899) and terminates.

[0133] The process (800) may be adjusted as appropriate. One (or more) steps in the process (800) may be modified and / or omitted. One (or more) additional steps may be added. Any suitable implementation order may be used.

[0134] According to one aspect of this application, a method for processing visual media data is provided. In this method, a bitstream of visual media data is processed according to format rules. For example, the bitstream may be a bitstream decoded / encoded using any decoding and / or encoding method described in this application. The format rules may specify one or more constraints on the bitstream, and / or one or more processes to be performed by the decoder and / or encoder.

[0135] For example, the bitstream contains encoded information of one or more images, the encoded information indicating intra-frame prediction of the current block in the current image. The format rules specify that: when the current block satisfies one or more conditions associated with reference sample filtering with a bilateral filter, reference sample filtering with a bilateral filter is enabled; when reference sample filtering with a bilateral filter is enabled, the reference sample filtering with a bilateral filter is applied to the reference samples of the current block to obtain filtered reference samples; and, using the filtered reference samples, one or more samples of the current block are reconstructed based on intra-frame prediction.

[0136] The above-described technology can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media. For example, Figure 9 A computer system (900) suitable for implementing certain aspects of the disclosed subject matter is shown.

[0137] Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or through interpretation, microcode execution, or other means.

[0138] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0139] Figure 9 The components of the computer system (900) shown are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing various aspects of this application. The configuration of the components should also not be construed as having any dependency or requirement relating to any component or combination of components shown in the exemplary aspects of the computer system (900).

[0140] The computer system (900) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not indicated). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, images captured from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0141] Human-machine interface input devices may include one or more of the following (only one of each is shown): keyboard (901), mouse (902), touchpad (903), touch screen (910), data glove (not shown), joystick (905), microphone (906), scanner (907), and camera (908).

[0142] The computer system (900) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (910), data gloves (not shown), or joysticks (905), but may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers (909), headphones (not shown)), and visual output devices (e.g., screens (910) including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touch screen input functionality, each screen may or may not have tactile feedback functionality, some of which are capable of outputting two-dimensional or more three-dimensional visual outputs through devices such as stereoscopic image output, virtual reality glasses (not shown), holographic displays and smoke boxes (not shown), and printers (not shown).

[0143] The computer system (900) may also include human-machine-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (920) having media such as CD / DVD (921), finger drives (922), removable hard disk drives or solid-state drives (923), conventional magnetic media such as magnetic tapes and floppy disks (not indicated), devices based on dedicated ROM / ASIC / PLD such as security dongles (not indicated), etc.

[0144] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0145] The computer system (900) may also include an interface (954) leading to one or more communication networks (955). The network may be, for example, a wireless network, a wired network, or an optical network. The network may further be a local area network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a latency-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter (e.g., a USB port of the computer system (900)) attached to some general-purpose data port or peripheral bus (949); other network interfaces are typically integrated into the core of the computer system (900) by being attached to a system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system), as described below. The computer system (900) can use any of these networks to communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., a CANBus connected to certain CANBus devices), or bidirectional, such as connecting to other computer systems using a local area network (LAN) or wide area network (WAN) digital network. As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

[0146] The aforementioned human-machine interface device, human-machine accessible storage device, and network interface can be attached to the kernel (940) of the computer system (900).

[0147] The core (940) may include one or more central processing units (CPU) (941), graphics processing units (GPUs) (942), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (943), hardware accelerators (944) for certain tasks, graphics adapters (950), etc. These devices, as well as read-only memory (ROM) (945), random access memory (946), and internal mass storage (947) such as internal non-user-accessible hard disk drives, SSDs, etc., may be connected via a system bus (948). In some computer systems, the system bus (948) may be accessed in the form of one or more physical plugs to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (948) or attached to the core's system bus (948) via a peripheral bus (949). In one example, a touchscreen (910) may be connected to a graphics adapter (950). Peripheral bus architectures include PCI, USB, etc.

[0148] The CPU (941), GPU (942), FPGA (943), and accelerator (944) can execute certain instructions that can be combined to form the aforementioned computer code. This computer code can be stored in ROM (945) or RAM (946). Transient data can also be stored in RAM (946), while permanent data can be stored, for example, in internal mass storage (947). Fast storage and retrieval to any storage device can be achieved using a cache, which can be closely associated with one or more CPUs (941), GPUs (942), mass storage (947), ROM (945), RAM (946), etc.

[0149] Computer-readable media may have computer code thereon that performs various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this application, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0150] As an example, and not a limitation, a computer system (900) having an architecture, particularly a kernel (940), can provide functionality by one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and some non-transitory memory of the kernel (940), such as internal mass storage (947) or ROM (945). Software implementing various aspects of this application can be stored in such devices and executed by the kernel (940). Depending on specific needs, the computer-readable media may include one or more storage devices or chips. The software can cause the kernel (940), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (946) and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may be made functional by hard-wired or otherwise embodied logic in circuitry (e.g., accelerator (944)), which may replace or operate with the software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (e.g., integrated circuit (IC)), circuitry embodying logic for execution, or both. This application includes any suitable combination of hardware and software.

[0151] The use of “at least one of…” or “one of…” in this application is intended to include any one or a combination of the listed elements. For example, referring to at least one of A, B, and C, at least one of A, B, and C, at least one of A, B, and / or C, and at least one of A to C is intended to include only A, only B, only C, or any combination thereof. Referring to one of A and B, and referring to one of A and B, is intended to include either A or B or (A and B). Where applicable, the use of “one of…” does not exclude any combination of the listed elements, for example, when the elements are not mutually exclusive.

[0152] While several examples of various aspects have been described in this application, there are modifications, substitutions, and various equivalents that fall within the scope of this application. Therefore, it should be understood that those skilled in the art will be able to devise many systems and methods that, while not explicitly shown or described herein, embody the principles of this application and thus fall within its spirit and scope.

[0153] The above disclosure also includes the features described below. These features can be combined in various ways, and are not limited to the following combinations.

[0154] (1) A video decoding method includes: receiving a bit stream including encoded information of one or more images, the encoded information indicating intra-frame prediction of a current block in the current image; enabling reference sample filtering with a bilateral filter when the current block satisfies one or more conditions associated with reference sample filtering; applying reference sample filtering with a bilateral filter to reference samples of the current block when the reference sample filtering with a bilateral filter is enabled to obtain filtered reference samples; and reconstructing one or more samples of the current block by intra-frame prediction based on the filtered reference samples.

[0155] (2) The method according to feature (1), wherein when the current block does not meet one or more conditions, the method further includes: using unfiltered reference samples to reconstruct one or more samples of the current block based on intra-frame prediction.

[0156] (3) The method according to any one of features (1) and (2), wherein one or more conditions include at least one of the following: the prediction mode of the intra-frame prediction is not a vertical mode; the prediction mode of the intra-frame prediction is not a horizontal mode; the prediction mode uses reference samples at integer positions; intra-frame prediction with multiple reference lines is not applied; sub-partition prediction is not applied; the block size of the current block is greater than a threshold; the current block is a prediction block within the luma component; and / or, a flag associated with the current block is decoded from the bitstream, and the flag indicates the use of a bilateral filter.

[0157] (4) The method according to any one of features (1) to (3), wherein the reference sample filtering includes a single filtering stage of a bilateral filter.

[0158] (5) The method according to any one of features (1) to (4), wherein the reference sample filtering uses a bilateral filter and a predefined filter, and the bilateral filter and the predefined filter are serially connected.

[0159] (6) The method according to any one of features (1) to (5), wherein the reference sample filtering uses a bilateral filter and a predefined filter, the bilateral filter and the predefined filter being connected in parallel.

[0160] (7) The method according to any one of features (1) to (6), wherein the application step includes: applying a predefined filter when the block size of the current block is less than a threshold; and applying a bilateral filter in the reference sample filtering when the block size of the current block is equal to or greater than the threshold.

[0161] (8) The method according to any one of features (1) to (7), wherein the application step includes: applying a predefined filter when the block content variance of the current block is less than a threshold; and applying a bilateral filter when the block content variance of the current block is equal to or greater than a threshold.

[0162] (9) The method according to any one of features (1) to (8), wherein the reference sample filtering includes a bilateral filter and / or a predefined filter, and the method includes: applying a bilateral filter to the reference sample to obtain a filtered reference sample when the current block does not meet one or more conditions; and applying a predefined filter to the reference sample to obtain a filtered reference sample when the current block meets one or more conditions.

[0163] (10) The method according to any one of features (1) to (9), wherein the reference sample filtering includes a bilateral filter and / or a predefined filter, and the method includes: applying a bilateral filter to the reference sample to obtain a first filtered reference sample; when the current block does not satisfy one or more conditions, using the first filtered reference sample to reconstruct one or more samples of the current block according to intra-frame prediction; and when the current block satisfies one or more conditions, using a second filtered reference sample to reconstruct one or more samples of the current block according to intra-frame prediction, wherein the second filtered reference sample is obtained by applying a predefined filter to the first filtered reference sample.

[0164] (11) The method according to any one of features (1) to (10), wherein the reference sample filtering includes a bilateral filter and / or a predefined filter, and the method includes: applying a predefined filter to the reference sample when the current block satisfies one or more conditions to obtain an intermediate filtered reference sample; copying the reference sample as an intermediate filtered reference sample when the current block does not satisfy one or more conditions; applying a bilateral filter to the intermediate filtered reference sample to obtain a filtered reference sample; and reconstructing the current block based on intra-frame prediction using the filtered reference sample.

[0165] (12) The method according to any one of features (1) to (11) further comprises: checking whether at least one of the following is enabled: decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction; and, when at least one of decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction is enabled, determining whether the current block satisfies one or more conditions associated with reference sample filtering, the reference sample filtering including a bilateral filter.

[0166] (13) The method according to any one of features (1) to (12) further comprises: decoding a flag from the bitstream, the flag being one of the following: Sequence Parameter Set (SPS) level, Video Parameter Set (VPS) level, Picture Parameter Set (PPS) level, Adaptive Parameter Set (APS) level, Picture Header level, Slice Header level and / or Block level; and applying a reference sample with a bilateral filter to the reference sample when the flag indicates that a reference sample with a bilateral filter is used to obtain a filtered reference sample.

[0167] (14) A video coding method comprising: determining to encode a current block using intra-frame prediction; enabling reference sample filtering when the current block satisfies one or more conditions associated with reference sample filtering with a bilateral filter; when the reference sample filtering with a bilateral filter is enabled, applying the reference sample filtering with a bilateral filter to reference samples of the current block to obtain filtered reference samples; and encoding the current block as encoded information in a bitstream using intra-frame prediction based on the filtered reference samples.

[0168] (15) The method according to feature (14), wherein when the current block does not meet one or more conditions, the current block is encoded using intra-frame prediction based on unfiltered reference samples.

[0169] (16) The method according to any one of features (14) and (15), wherein one or more conditions include at least one of the following: the prediction mode of the intra-frame prediction is not a vertical mode; the prediction mode of the intra-frame prediction is not a horizontal mode; the prediction mode uses reference samples at integer positions; intra-frame prediction with multiple reference lines is not applied; sub-partition prediction is not applied; the block size of the current block is greater than a threshold; and / or, the current block is a prediction block within the luminance component.

[0170] (17) The method according to any one of features (14) to (16), wherein the reference sample filtering includes a single filtering stage of a bilateral filter.

[0171] (18) The method according to any one of features (14) to (17), wherein the reference sample filtering uses a bilateral filter and a predefined filter, the bilateral filter and the predefined filter being serially connected.

[0172] (19) The method according to any one of features (14) to (18), wherein the reference sample filtering uses a bilateral filter and a predefined filter, the bilateral filter and the predefined filter being connected in parallel.

[0173] (20) The method according to any one of features (14) to (19), wherein when the block size of the current block is less than the threshold, a predefined filter is applied in the reference sample filtering; and when the block size of the current block is equal to or greater than the threshold, a bilateral filter is applied in the reference sample filtering.

[0174] (21) The method according to any one of features (14) to (20), wherein when the block content variance of the current block is less than a threshold, a predefined filter is applied in the reference sample filtering; and when the block content variance of the current block is equal to or greater than the threshold, a bilateral filter is applied in the reference sample filtering.

[0175] (22) The method according to any one of features (14) to (21), wherein the reference sample filtering includes a bilateral filter and / or a predefined filter; when the current block does not meet one or more conditions, a bilateral filter is applied to the reference sample to obtain a filtered reference sample; and when the current block meets one or more conditions, a predefined filter is applied to the reference sample to obtain a filtered reference sample.

[0176] (23) The method according to any one of features (14) to (22), wherein the reference sample filtering includes a bilateral filter and / or a predefined filter; the bilateral filter is applied to the reference sample to obtain a first filtered reference sample; when the current block does not meet one or more conditions, the current block is encoded by intra-frame prediction based on the first filtered reference sample; and when the current block meets one or more conditions, the current block is encoded by intra-frame prediction based on a second filtered reference sample, wherein the second filtered reference sample is obtained by applying a predefined filter to the first filtered reference sample.

[0177] (24) The method according to any one of features (14) to (23), wherein the reference sample filtering includes a bilateral filter and / or a predefined filter; when the current block satisfies one or more conditions, the predefined filter is applied to the reference sample to obtain an intermediate filtered reference sample; when the current block does not satisfy one or more conditions, the reference sample is copied as an intermediate filtered reference sample; and a bilateral filter is applied to the intermediate filtered reference sample to obtain a filtered reference sample.

[0178] (25) The method according to any one of features (14) to (24) further comprises: checking whether at least one of the following is enabled: decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction. When at least one of decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction is enabled, determining whether the current block satisfies one or more conditions associated with a reference sample filter having a bilateral filter.

[0179] (26) The method according to any one of features (14) to (25) further comprises: when applying reference sample filtering with a bilateral filter, encoding a flag into the bitstream, the flag indicating the use of reference sample filtering with a bilateral filter, and the flag may be at the sequence parameter set (SPS) level, video parameter set (VPS) level, picture parameter set (PPS) level, adaptive parameter set (APS) level, picture header level, slice header level and / or block level.

[0180] (27) A method for processing visual media data, the method comprising: processing a bitstream of visual media data according to a format rule; the bitstream including encoded information of one or more images, the encoded information indicating intra-frame prediction of a current block in the current image; and the format rule specifying that: reference sample filtering is enabled when the current block satisfies one or more conditions associated with reference sample filtering with a bilateral filter; when reference sample filtering with a bilateral filter is enabled, the reference sample filtering with a bilateral filter is applied to reference samples of the current block to obtain filtered reference samples; and using the filtered reference samples, reconstructing one or more samples of the current block according to intra-frame prediction.

[0181] (28) An apparatus for video decoding, the apparatus comprising a processing circuit configured to perform the method of any one of features (1) to (13).

[0182] (29) An apparatus for video decoding, the apparatus comprising processing circuitry configured to perform the method described in any one of features (14) to (26).

[0183] (30) A non-transitory computer-readable storage medium storing instructions which, when executed by at least one processor, cause at least one processor to perform any of the methods of features (1) to (27).

Claims

1. A video decoding method, comprising: Receive a bitstream, the bitstream including encoded information of one or more images, the encoded information indicating intra-frame prediction of the current block; The reference sample filtering is enabled when the current block satisfies one or more conditions associated with the reference sample filtering with a bilateral filter; When the reference sample filtering with the bilateral filter is enabled, the reference sample filtering with the bilateral filter is applied to the reference sample of the current block to obtain the filtered reference sample; as well as Using the filtered reference samples, one or more samples of the current block are reconstructed based on the intra-frame prediction.

2. The method according to claim 1, wherein, When the current block does not satisfy one or more of the conditions, the method further includes: Using the unfiltered reference samples, one or more samples of the current block are reconstructed based on the intra-frame prediction.

3. The method according to claim 1, wherein, The one or more conditions include at least one of the following: The prediction mode of the intra-frame prediction is not the vertical mode. The prediction mode of the intra-frame prediction is not the horizontal mode. The prediction pattern uses reference samples at integer positions; Intra-frame prediction without multi-reference lines was not applied; Sub-partition prediction was not applied; The current block size is greater than the threshold; The current block is a prediction block within the luminance component; and / or Decode the flag associated with the current block from the bitstream, and the flag indicates the use of the bilateral filter.

4. The method according to claim 1, wherein, The reference sample filtering includes a single filtering stage of the bilateral filter.

5. The method according to claim 1, wherein, The reference sample filtering uses the bilateral filter and the predefined filter, which are connected in series.

6. The method according to claim 1, wherein, The reference sample filtering uses the bilateral filter and the predefined filter, which are connected in parallel.

7. The method according to claim 6, wherein, The applications include: When the size of the current block is less than a threshold, the predefined filter is applied; and When the block size of the current block is equal to or greater than the threshold, the bilateral filter is applied in the reference sample filtering.

8. The method according to claim 6, wherein, The applications include: When the block content variance of the current block is less than a threshold, the predefined filter is applied; and The bilateral filter is applied when the variance of the current block content is equal to or greater than the threshold.

9. The method according to claim 1, wherein, The reference sample filtering includes the bilateral filter and / or a predefined filter, and the method includes: When the current block does not meet one or more of the conditions, the bilateral filter is applied to the reference sample to obtain the filtered reference sample; and When the current block satisfies one or more of the conditions, the predefined filter is applied to the reference sample to obtain the filtered reference sample.

10. The method according to claim 1, wherein, The reference sample filtering includes the bilateral filter and / or a predefined filter, and the method includes: The bilateral filter is applied to the reference sample to obtain the first filtered reference sample; When the current block does not meet one or more of the conditions, one or more samples of the current block are reconstructed based on the intra-frame prediction using the first filtered reference samples; and When the current block satisfies one or more of the conditions, one or more samples of the current block are reconstructed based on the intra-frame prediction using the second filtered reference sample, wherein the second filtered reference sample is obtained by applying the predefined filter to the first filtered reference sample.

11. The method according to claim 1, wherein, The reference sample filtering includes the bilateral filter and / or a predefined filter, and the method includes: When the current block satisfies one or more of the conditions, the predefined filter is applied to the reference sample to obtain the intermediate filtered reference sample; When the current block does not meet one or more of the conditions, the reference sample is copied as the intermediate filtered reference sample; The bilateral filter is applied to the intermediate filtered reference sample to obtain the filtered reference sample; and The current block is reconstructed using the filtered reference sample and the intra-frame prediction.

12. The method according to claim 1, further comprising: Check if at least one of the following is enabled: decoder-side intra-mode derivation, template-based intra-mode derivation, and gradient location-dependent intra-prediction; as well as When at least one of the decoder-side intra-mode derivation, the template-based intra-mode derivation, and the gradient location-dependent intra-prediction is enabled, it is determined whether the current block satisfies one or more conditions associated with the reference sample filtering including the bilateral filter.

13. The method according to claim 1, further comprising: Decode the flags from the bitstream, the flags being one of the following: Sequence Parameter Set (SPS) level, Video Parameter Set (VPS) level, Picture Parameter Set (PPS) level, Adaptive Parameter Set (APS) level, Picture Header level, Slice Header level, and / or Block level; as well as When the flag indicates that the reference sample is filtered using the bilateral filter, the reference sample is filtered using the bilateral filter to obtain the filtered reference sample.

14. A video coding method, comprising: Determine to use intra-frame prediction to encode the current block; The reference sample filtering is enabled when the current block satisfies one or more conditions associated with the reference sample filtering with a bilateral filter; When the reference sample filtering with the bilateral filter is enabled, the reference sample filtering with the bilateral filter is applied to the reference sample of the current block to obtain the filtered reference sample; as well as Based on the filtered reference samples, the current block is encoded into encoded information in the bitstream using the intra-frame prediction.

15. A method for processing visual media data, the method comprising: Process the bitstream of visual media data according to format rules; The bitstream includes encoded information of one or more images, the encoded information indicating intra-frame prediction of the current block in the current image; as well as The format rules stipulate that: The reference sample filtering is enabled when the current block satisfies one or more conditions associated with the reference sample filtering with a bilateral filter; When the reference sample filtering with the bilateral filter is enabled, the reference sample filtering with the bilateral filter is applied to the reference sample of the current block to obtain the filtered reference sample; as well as Using the filtered reference samples, one or more samples of the current block are reconstructed based on the intra-frame prediction.

Images