Method and apparatus for selecting intra interpolation filter for multi-line intra prediction, and computer program

Abstract

To provide a method and apparatus for selecting an intra interpolation filter for multi-line intra prediction based on a reference line index for decoding a video sequence.SOLUTION: A method includes: identifying a set of reference lines associated with a coding unit; applying a first type of interpolation filter to reference samples included in a first reference line, of the set of reference lines, that is adjacent to the coding unit to generate a first set of prediction samples based on the first reference line being associated with a first reference line index; and applying a second type of interpolation filter to reference samples included in a second reference line, of the set of reference lines, that is non-adjacent to the coding unit to generate a second set of prediction samples based on the second reference line being associated with a second reference line index.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) Pursuant to 35 U.S.C. 119, this application claims priority to U.S. Provisional Application No. 62 / 729,395, filed September 10, 2018, and U.S. Patent Application No. 16 / 202,902, filed in the U.S. Patent and Trademark Office on November 28, 2018, the disclosures of which are incorporated herein by reference in their entireties.

[0002] This disclosure relates to a set of advanced video coding techniques. More specifically, this disclosure provides a modified intra-interpolation filter scheme for multi-line intra-prediction. [Background technology]

[0003] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC l / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1) and provided updates in 2014 (version 2), 2015 (version 3) and 2016 (version 4). Since then, ITU has been studying the potential need for standardization of future video coding techniques with compression capabilities significantly beyond the HEVC standard (including extensions).

[0004] In October 2017, the ITU released a Joint Proposal (CfP) for video compression with capabilities beyond HEVC. By February 15, 2018, a total of 22 CfP responses for Standard Dynamic Range (SDR), 12 CfP responses for High Dynamic Range (HDR), and 12 CfP responses for the 360 ​​video category had been submitted.

[0005] April 2018, 122 MPEG / 10 thAt the JVET (Joint Video Exploration Team - Joint Video Experts Team) meeting, all received CfP responses were evaluated. After careful evaluation, the JVET formally started the standardization of the next generation of video coding beyond HEVC, the so-called Versatile Video Coding (VVC).

[0006] HEVC has a total of 35 intra prediction modes, where mode 10 is the horizontal mode, mode 26 is the vertical mode, and modes 2, 18, and 34 are diagonal modes. The intra prediction modes are signaled by three most probable modes (MPMs) and the remaining 32 modes.

[0007] In the current development of VVC, there are a total of 87 intra-prediction modes, where mode 18 is the horizontal mode, mode 50 is the vertical mode, and modes 2, 34 and 66 are diagonal modes. Modes 1 to 10 and modes 67 to 76 are designated as wide-angle intra-prediction (WAIP) modes.

[0008] To code an intra mode, generate a Most Probable Mode (MPM) list of three modes based on the intra modes of neighboring blocks. This MPM list is called the MPM list or the primary MPM list. If the intra mode is not included in the MPM list, a flag is signaled to indicate whether the intra mode belongs to the selected mode.

[0009] In the development of VVC, a method of realizing a primary MPM list and a secondary MPM list is proposed. The modes in the secondary MPM list are not included in the primary MPM list. The number of modes in the MPM list may be 3, 4, 5, 6, 7, 8, etc., and the number of modes in the secondary MPM list may be 8, 16, 32, etc.

[0010] In VVC, for the luma component, before the generation process, neighboring samples used to generate intra prediction samples are filtered (i.e., intra smoothing process). The filtering is controlled by a given intra prediction mode and transform block size. If the intra prediction mode is DC or the transform block size is 4x4, the neighboring samples are not filtered. Also, if the distance between the given intra prediction mode and the vertical mode (or horizontal mode) is greater than a certain threshold, the filtering process is started. A [1,2,1] filter and a bilinear filter are used to filter the neighboring samples. For example, article 8.2.4.2.4 and table 8-4 of VVC draft 2 described the intra smoothing process proposed in VVC.

[0011] Multiline intra prediction was proposed to use additional reference lines for intra prediction. The encoder can determine and signal the reference lines used to generate the intra predictor. Prior to the intra prediction mode, it signals the reference line index and excludes planar mode / DC mode from the intra prediction modes in case of signaling a non-zero reference line index.

[0012] Wider angles beyond the range of prediction directions covered by traditional intra prediction modes are proposed and are called wide-angle intra prediction modes. These wide angles apply only to non-square blocks: If the block width is greater than the block height, it is at an angle of more than 45 degrees to the upper right (intra prediction mode 34 in HEVC); If the block height is greater than the block width, it is at an angle of more than 45 degrees to the lower left (intra prediction mode 2 in HEVC).

[0013] The original method is used to signal the replaced modes and remap them to the wide-angle mode index after analysis. The total number of intra prediction modes remains unchanged, i.e., 35 for VTM-1.0 and 67 for BMS-1.0, and the coding of intra modes remains unchanged.

[0014] A bilateral filter is a nonlinear, edge-preserving, noise-reducing smoothing filter used on images. It replaces the luminance of each pixel using a weighted average of the luminance values ​​from neighboring pixels. The weights can be based on a Gaussian distribution. The weights depend on the Euclidean distance of the pixels and further on the radial differences (e.g. differences in range of color intensity, depth distance, etc.). Bilateral filters help to preserve sharp edges. Given the original (unfiltered) reference samples I(x) of an intra block, the bilateral filter function is defined as follows: JPEG2024114867000002.jpg29151

[0015] 4-tap and 6-tap intra-interpolation filters are proposed to generate prediction samples used in directional intra-prediction. Two types of 4-tap interpolation filters are used: cubic and Gaussian. The cubic filter is suitable for preserving image edges, and the Gaussian filter is suitable for removing image noise. The cubic filter is realized when the intra-prediction mode is a diagonal mode (i.e., mode 34) or higher and the block width is 8 or less, and when the intra-prediction mode is a diagonal mode (i.e., mode 34) or lower and the block height is 8 or less.

[0016] The Gaussian filter is implemented when the intra prediction mode is a diagonal mode (i.e., mode 34) or higher and the block width is greater than 8, and when the intra prediction mode is a diagonal mode (i.e., mode 34) or lower and the block height is greater than 8. Summary of the Invention

[0017] A method for selecting an intra-line intra-prediction filter based on a reference line index for decoding a video sequence, comprising: identifying a set of reference lines associated with a coding unit; applying a first type of interpolation filter to reference samples included in a first reference line that is adjacent to the coding unit based on a first reference line index being associated with the first reference line index, to generate a first set of predicted samples; and applying a second type of interpolation filter to reference samples included in a second reference line that is not adjacent to the coding unit based on a second reference line that is associated with a second reference line index, to generate a second set of predicted samples.

[0018] 11. An apparatus for selecting an intra-interpolation filter for multi-line intra prediction based on a reference line index for decoding a video sequence, comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to instructions of the program code, the program code including: an identification code configured to cause the at least one processor to identify a set of reference lines associated with a coding unit; a first application code configured to cause the at least one processor to apply a first type of interpolation filter to reference samples included in a first reference line that is adjacent to the coding unit based on a first reference line in the set of reference lines being associated with a first reference line index to generate a first set of predicted samples; and a second application code configured to cause the at least one processor to apply a second type of interpolation filter to reference samples included in a second reference line that is not adjacent to the coding unit based on a second reference line in the set of reference lines being associated with a second reference line index to generate a second set of predicted samples.

[0019] A non-transitory computer-readable medium storing instructions including one or more instructions that, when executed by one or more processors of an apparatus for selecting an intra interpolation filter for multi-line intra prediction based on a reference line index for decoding a video sequence, cause the one or more processors to perform the following operations: identify a set of reference lines associated with a coding unit; based on a first reference line in the set of reference lines being associated with a first reference line index, apply a first type of interpolation filter to reference samples included in the first reference line that is adjacent to the coding unit to generate a first set of predicted samples; and based on a second reference line in the set of reference lines being associated with a second reference line index, apply a second type of interpolation filter to reference samples included in the second reference line that is not adjacent to the coding unit to generate a second set of predicted samples. [Brief description of the drawings]

[0020] Other features, characteristics and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which: [Figure 1] 11 is a flowchart of an example process for selecting an intra-interpolation filter for multi-line intra prediction based on a reference line index for decoding a video sequence. [Diagram 2] 1 is a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a diagram of the arrangement of a video encoder and decoder in a streaming environment. [Figure 4] FIG. 2 is a functional block diagram of a video decoder according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a functional block diagram of a video encoder according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The Gaussian filter is suitable for smooth image regions, while the cubic filter is suitable for regions with edges. The addition of additional reference lines used in multi-line intra prediction helps with regions with edges. However, applying both the same cubic and Gaussian filters to all reference lines is not a desirable design.

[0022] 1 is a flow chart of an example process 100 for selecting an intra-interpolation filter for multi-line intra-prediction based on a reference line index for decoding a video sequence. In some implementations, one or more processing blocks of FIG. 1 may be performed by a decoder. In some implementations, one or more processing blocks of FIG. 1 may be performed by another device or group of devices (e.g., an encoder) separate from or including the decoder.

[0023] As shown in FIG. 1, the process 100 may include identifying a set of reference lines associated with a coding unit (block 110).

[0024] In some implementations, the line index of the closest reference line is 0 (zero reference line), and the maximum number of reference lines signaled is represented as N. In the following, the intra-interpolation filter is an interpolation filter for generating a predicted value that points to a fractional position of a reference sample.

[0025] In some implementations, the selection of the intra-interpolation filter depends on the signaled reference line index. Types of intra-interpolation filters include edge-preserving filters and edge-smoothing filters and / or similar filters. Edge-smoothing filters include linear interpolation filters with positive or zero filter coefficients. Edge-preserving filters include linear interpolation filters or nonlinear filters with at least one or two negative filter coefficients, such as bilateral filters.

[0026] As further shown in FIG. 1, process 100 may include applying an interpolation filter of a first type to reference samples included in a first reference line that is adjacent to the encoding unit based on a first reference line in the set of reference lines being associated with a first reference line index to generate a first set of predicted samples (block 120), and applying an interpolation filter of a second type to reference samples included in a second reference line that is not adjacent to the encoding unit based on a second reference line in the set of reference lines being associated with a second reference line index to generate a second set of predicted samples (block 130).

[0027] According to an embodiment, both an edge smoothing filter and an edge preserving filter are applied to the zero reference line, and only an edge preserving filter is applied to the non-zero reference line, and the edge smoothing filter may include a bilinear filter or a Gaussian filter.

[0028] According to an embodiment, the edge-preserving filter may be a cubic interpolation filter, a DCT-based interpolation filter (DCT-IF), a 4 / 6-tap polynomial-based interpolation filter, a bilateral filter, a Hermite interpolation filter, and / or similar filters.

[0029] According to an embodiment, the edge-preserving filters used for different lines are different. For example, the taps of the edge-preserving filters may be different for the zero reference line and the non-zero reference line. Additionally or alternatively, an M-tap edge-preserving filter is used for the zero reference line and an N-tap edge-preserving filter is used for the non-zero reference line (e.g., M and N are positive integers and M is not equal to N). Additionally or alternatively, the filter coefficients of the edge-preserving filters used for different reference lines may be different.

[0030] In another embodiment, the coefficients of the edge-preserving and edge-smoothing filters are fixed and do not depend on the sample values.

[0031] In some implementations, the selection of the intra-interpolation filter depends on the reference line index and other coded information, or any information available at both the encoder and decoder, such as the intra-prediction mode and block size.

[0032] According to an embodiment, an edge-preserving filter is used based on a condition that is met. For example, the condition may be met if the intra prediction mode is equal to or greater than a diagonal mode (i.e., mode 34) and the width of the block is equal to or less than S. Additionally or alternatively, the condition may be met if the intra prediction mode is equal to or less than a diagonal mode (i.e., mode 34) and the height of the block is equal to or less than S. For example, S may indicate a threshold value of the width or height of the block. For different lines, S may be different. For example, according to an embodiment, for a zero reference line, S is 8, and for a non-zero reference line, S is 16 or 32.

[0033] According to an embodiment, an edge smoothing filter is used for wide angles at the zero reference line and an edge preserving filter is used for wide angles at the non-zero reference line. Additionally or alternatively, an edge preserving filter is used for wide angles at the zero reference line and an edge smoothing filter is used for wide angles at the non-zero reference line.

[0034] According to an embodiment, both edge-preserving and edge-smoothing filters are applied to the zero reference line and / or an edge-preserving or edge-smoothing filter is applied to at least one reference line, in which case there is a possibility of not applying two filters. According to an embodiment, if there are three reference lines, the line index may be {0,1,2} or {0,1,3}, and both edge-preserving and edge-smoothing filters are applied to line 0, only an edge-preserving or edge-smoothing filter is applied to line 1, and only an edge-smoothing or edge-preserving filter is applied to line 2 or line 3.

[0035] According to an embodiment, the edge-preserving filter and the edge-smoothing filter are applied to all reference lines, but the number of taps of the intra-interpolation filter may be different for each line. For example, in an embodiment, the edge-preserving filter used for the zero reference line is M taps, and the edge-preserving filter used for the non-zero reference line is N taps (e.g., M and N are positive integers, and M is not equal to N, e.g., M=6 and N=4). Additionally or alternatively, the edge-smoothing filter used for the zero reference line is M taps, and the edge-smoothing filter used for the non-zero reference line is N taps (e.g., M and N are positive integers, and M is not equal to N, e.g., M=6 and N=4).

[0036] According to an embodiment, for some reference lines, wide angles are prohibited or limited differently. For example, for non-zero reference lines, wide angles are prohibited. Additionally or alternatively, at least one reference line does not use wide angles. Additionally or alternatively, the wide angles used for different reference lines are different. For example, the number of conventional angle intra-prediction directions replaced by wide-angle intra-prediction directions depends on the reference line index.

[0037] According to an embodiment, the intra-smoothing filter may be different for each line. For example, a bilateral filter is used for intra-smoothing of non-zero reference lines. Additionally or alternatively, the intra-smoothing filter is inhibited for non-zero reference lines when signaling a reference line index. Additionally or alternatively, at least one reference line uses a bilateral filter for intra-smoothing.

[0038] According to an embodiment, the number of filter taps of the intra smoothing filter is different for each line. For example, the number of filter taps of the intra smoothing filter used for the zero reference line is M, and the number of filter taps of the intra smoothing filter used for the non-zero reference line is N (for example, M and N are positive integers, and M is not equal to N, for example, M=6 and N=4).

[0039] According to an embodiment, the intra prediction mode that applies intra smoothing is different for each line. For example, a threshold T defines which intra prediction mode applies intra smoothing when the intra prediction mode Mode satisfies the following condition: min(abs(Mode-Hor),abs(Mode-Ver)) <Tであり、 Here, "Hor" denotes the intra-prediction mode index of the horizontal mode, and "Ver" denotes the intra-prediction mode index of the vertical mode. The value of T depends on the reference line index and other coded information or any information known to both the encoder and the decoder, including, but not limited to, the block region size, the block width, the block height, the block aspect ratio, etc.

[0040] According to an embodiment, for non-zero reference lines, only MPM modes are allowed, including primary and secondary MPM lists. In one embodiment, if the reference line index of the current block is greater than 0, one bin is signaled, called "primary_mpm_flag", to indicate whether the intra prediction mode of the current block belongs to the primary or secondary MPM list. If "primary_mpm_flag" is "true", the primary MPM index is signaled. Otherwise, the secondary MPM index is signaled. No non-MPM modes are used for non-zero reference lines, and no flag is signaled to indicate whether to use secondary MPM or non-MPM.

[0041] According to an embodiment, planar and DC modes are excluded from the primary and secondary MPM lists.

[0042] According to an embodiment, some implementations use multiple contexts to code the first bin of the reference line index. For example, the selection of the context depends on the reference line index of the neighboring block. As a specific example, if both the reference line index of the left block and the reference line index of the block above are equal to 0, select context 0, and if both the reference line index of the left block and the reference line index of the block above are not equal to 0, select context 1. Otherwise, select context 2.

[0043] According to an embodiment, the selection of the context depends on the CBF (Coded Block Flag) of the neighboring block. The CBF is an indicator of whether the current block contains non-zero coefficients. If the CBF is equal to 0, there are no non-zero coefficients in the current block. In one example, if the CBF of the left block and the CBF of the block above are both equal to 0, select context 0. And if the CBF of the left block and the CBF of the block above are both not equal to 0, select context 1. Otherwise, select context 2.

[0044] According to an embodiment, the context for encoding the transform selection information according to the reference line index value includes, but is not limited to, an MTS flag, an MTS index, and an NSST index.

[0045] Although Figure 1 illustrates example blocks of process 100, in some implementations, process 100 may include other blocks, fewer blocks, different blocks, or blocks arranged differently as compared to those blocks illustrated in Figure 1. Additionally or alternatively, two or more blocks of process 100 may be performed in parallel.

[0046] FIG. 2 shows a schematic block diagram of a communication system (200) according to an embodiment of the present disclosure. The communication system (200) may include at least two terminals (210-220) interconnected via a network (250). In the case of one-way data transmission, a first terminal (210) can encode video data at a local location and transmit it to another terminal (220) via the network (250). The second terminal (220) can receive the encoded video data of the other terminal from the network (250), decode the encoded data, and display the restored video data. One-way data transmission is common in media service applications, etc.

[0047] 2 illustrates a second pair of terminals (230, 240) provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal (230, 240) may encode video data captured at a local location and transmit it to the other terminal over the network (250). Each terminal (230, 240) may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the reconstructed video data on a local display device.

[0048] In FIG. 2, the terminals (210-240) are illustrated as servers, personal computers, and smartphones, although the principles of the present disclosure are not so limited. The embodiments of the present disclosure apply to laptop computers, tablets, media players, and / or dedicated video conferencing equipment. The network (250) represents any number of networks that transmit encoded video data between the terminals (210-240), including, for example, wired and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of the network (250) may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0049] As an example application of the disclosed subject matter, FIG. 3 illustrates a video encoder and decoder arrangement in a streaming environment, although the disclosed subject matter can be equally utilized in other applications that support video, including, for example, video conferencing, digital TV, storage of compressed video on digital media such as CDs, DVDs, memory sticks, etc.

[0050] The streaming system can include a capture subsystem (313), which may include a video source (301) (e.g., a digital camera) for creating an uncompressed video sample stream (302). The video sample stream (302) is depicted as a thick line to emphasize the large amount of data compared to an encoded video bitstream. The sample stream (302) can be processed by an encoder (303) coupled to the camera (301). The encoder (303) can include hardware, software, or a combination thereof to realize or implement aspects of the disclosed subject matter, which are described in more detail below. The encoded video bitstream (304) is depicted as a thin line to emphasize the small amount of data compared to the sample stream. The encoded video bitstream (304) can be stored in a streaming server (305) for future use. One or more streaming clients (306, 308) can access the streaming server (305) to retrieve copies (307, 309) of the encoded video bitstream (304). The client (306) can include a video decoder (310) that decodes the incoming copy of the encoded video bitstream (307) and creates an outgoing video sample stream (311) that can be rendered on a display (312) or other rendering device (not shown). In some streaming systems, the video bitstreams (304, 307, 309) can be encoded according to a particular video encoding / compression standard. Examples of these standards include the ITU-T H.265 recommendation. A video encoding standard, informally referred to as Versatile Video Coding, is under development. The disclosed subject matter can be used in the context of VVC.

[0051] FIG. 4 may be a functional block diagram of a video decoder (310) according to an embodiment of the disclosure.

[0052] The receiver (410) may receive one or more codec video sequences to be decoded by the decoder (310). In the same or another embodiment, it receives one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. It may receive coded video sequences from a channel (412), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (410) may receive coded video data and other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to a respective using entity (not shown). The receiver (410) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (410) and the entropy decoder / parser (420) (hereafter referred to as the "parser"). If the receiver (410) receives data from a store-and-forward device or an isochronous real-time network with sufficient bandwidth and controllability, the buffer (415) may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer (415) may be required, and may be relatively large and advantageously have a self-adaptive size.

[0053] The video decoder (310) may include a parser (420) to reconstruct symbols (421) based on the entropy coded video sequence. These categories of symbols include information for managing the operation of the decoder (310) and latent information for controlling rendering devices that are not part of the decoder but may be coupled to the decoder, such as a display (312). As shown in FIG. 4, the control information for one or more rendering devices may be in the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may conform to a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context-dependent coding, etc. The parser (420) can extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The entropy decoder / parser may also extract, for example, transform coefficients, Quantizer Parameter (QP) values, motion vectors, etc. from the coded video sequence information.

[0054] The parser (420) may perform an entropy decoding / parsing operation on the video sequence received from the buffer (415) to produce symbols (421). The parser (420) may receive the encoded data and selectively decode particular symbols (421). The parser (420) may also determine whether a particular symbol (421) is to be provided to a motion compensated prediction unit (453), a scaler / inverse transform unit (451), an intra prediction unit (452), or a loop filter (456).

[0055] The reconstruction of the symbols (421) may involve several different units, depending on the type of coded video picture or part of the coded video picture (e.g., inter-picture vs. intra-picture, inter-block vs. intra-block) and other factors. The units involved, and the manner in which they are involved, may be controlled by subgroup control information parsed by the parser (420) from the coded video sequence. For simplicity, the flow of such subgroup control information between the parser (420) and the following units is not described.

[0056] In addition to the functional blocks already mentioned, the decoder (310) can be conceptually subdivided into a number of functional units, which are described below. In an actual implementation implemented under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0057] The first unit is a scalar / inverse transform unit (451), which receives quantized transform coefficients as symbol(s) (421) and control information from the parser (420), including the transform method to be used, block size, quantization coefficients, quantization scaling matrix, etc. The scalar / inverse transform unit (451) can output a block containing sample values, which can be input to an aggregator (455).

[0058] In some cases, the output samples of the scaler / inverse transform unit (451) may belong to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but use prediction information from a previously reconstructed part of the current picture. Such prediction information can be provided by an intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) uses surrounding already reconstructed information extracted from the current (partially reconstructed) picture (456) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (455) adds the prediction information generated by the intra-prediction unit (452) based on each sample to the output sample information provided by the scaler / inverse transform unit (451).

[0059] In other cases, the output samples of the scalar / inverse transform unit (451) may belong to an inter-coded and potential motion compensation block. In such cases, the motion compensation prediction unit (453) may access the reference picture memory (457) to extract samples for prediction. After performing motion compensation on the extracted samples based on the symbols (421) belonging to the block, these samples may be added by the aggregator (455) to the output of the scalar / inverse transform unit (called residual samples or residual signals in this case) to generate output sample information. The addresses in the reference picture memory from which the motion compensation unit extracts the prediction samples may be controlled by a motion vector, which may be used by the motion compensation unit in the form of a symbol (421), which may have, for example, X, Y, and reference picture components. Motion compensation may include interpolation on sample values ​​extracted from the reference picture memory, motion vector prediction mechanisms, etc., when sub-sample accurate motion vectors are used.

[0060] The output samples of the aggregator (455) may be processed by various loop filtering techniques in a loop filter unit (456). The video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and made available to the loop filter unit (456) as symbols (421) from the parser (420), but may also be responsive to meta-information obtained during decoding of a previous portion (in decoding order) of the encoded picture or encoded video sequence, or to previously reconstructed loop filtered sample values.

[0061] The output of the loop filter unit (456) may be a sample stream, which may be output to the rendering device (312) and stored in a reference picture memory (456) for use in future inter-picture prediction.

[0062] Once fully reconstructed, some coded pictures can be used as reference pictures for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current reference picture (656) can become part of the reference picture buffer (457), and a new current picture memory can be reallocated before beginning reconstruction of a subsequent coded picture.

[0063] The video decoder (310) may perform decoding operations according to a given video compression technique, e.g., as documented in the ITU-T H.265 Recommendation standard. The encoded video sequence may comply with the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence complies with the syntax of the video compression technique or standard (as specified in the video compression technique document or standard, and in particular in the document files therein). Compliance also requires that the complexity of the encoded video sequence is within the range limited by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in megasamples per second), the maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the specification of a Hypothetical Reference Decoder (HRD) and metadata for HRD buffer management signaled in the encoded video sequence.

[0064] In an embodiment, the receiver (410) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (310) to accurately decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0065] FIG. 5 may be a functional block diagram of a video encoder (303) according to an embodiment of the present disclosure.

[0066] The encoder (303) may receive video samples from a video source (301) (not part of the encoder), which may capture the video image(s) that are encoded by the encoder (303).

[0067] The video source (301) may provide a source video sequence in the form of a digital video sample stream to be encoded by the encoder (303), the digital video sample stream may have any suitable bit depth (e.g. 8-bit, 10-bit, 12-bit, etc.), any color space (e.g. BT.601 Y CrCb, RGB, etc.) and any suitable sampling configuration (e.g. Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (503) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of separate pictures that are given motion when viewed in sequence. The pictures themselves may be organized as a spatial pixel array, where each pixel may contain one or more samples, depending on the sampling configuration, color space, etc. used. Those skilled in the art can easily understand the relationship between pixels and samples. In the following description, the focus is on samples.

[0068] According to an embodiment, the encoder (303) may encode and compress pictures of a source video sequence into an encoded video sequence (543) in real time or under any other time constraint required by the application. Enforcing an appropriate encoding rate is one of the functions of the controller (550). The controller controls and is operatively coupled to other functional units described below. For the sake of simplicity, the coupling is not depicted. Parameters set by the controller may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily recognize other functions of the controller (550), which relate to a video encoder (303) optimized for a particular system design.

[0069] Some video encoders operate in a manner that those skilled in the art will readily recognize as a "coding loop." As a very simplified description, the encoding loop may include an encoding portion of an encoder (530) (hereafter referred to as the "source encoder") (responsible for creating symbols based on the input picture to be encoded and the reference picture(s)), and a (local) decoder (533) embedded in the encoder (303) that reconstructs the symbols to create sample data that is also created by a (remote) decoder (since the compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream is input to a reference picture memory (534). The decoding of the symbol stream produces results that are bit-accurate regardless of the location of the decoder (local or remote), so the contents of the reference picture buffer are also bit-accurate between the local and remote encoders. That is, the reference picture samples that the prediction part of the encoder "sees" are exactly the same as the sample values ​​that the decoder "sees" when using the prediction during decoding. The basic principle of reference picture synchrony (and the drift that occurs when synchrony cannot be maintained, e.g., due to channel errors) is well known to those skilled in the art.

[0070] The operation of the "local" decoder (533) may be the same as the operation of the "remote" decoder (310) described in detail above with reference to Figure 4. However, and referring briefly to Figure 5, the entropy decoding portion of the decoder (310), including the channel (412), receiver (410), buffer (415) and parser (420), may not be implemented entirely in the local decoder (533), provided that symbols are available and the entropy encoder (545) and parser (420) are capable of losslessly encoding / decoding the symbols into an encoded video sequence.

[0071] In this case, in addition to the analysis / entropy decoding present in the decoder, it is understood that any decoder techniques will necessarily be present in the corresponding encoder in essentially the same functional form. The description of the encoder technique can be simplified since the encoder technique and the fully described decoder technique are mutually inverse. More detailed descriptions are only necessary in certain areas and are provided below.

[0072] As part of its operation, the source encoder (530) may perform motion-compensated predictive encoding, which predictively encodes an input frame by referencing one or more previously encoded frames from the video sequence designated as "reference frames." In this manner, the encoding engine (532) may encode differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references for the input frame.

[0073] The local video decoder (533) may decode the encoded video data of frames that may be designated as reference frames based on the symbols created by the source encoder (530). The operation of the encoding engine (532) may advantageously be a lossy process. If the encoded video data may be decoded in a video decoder (not shown in FIG. 5), the reconstructed video sequence may be a copy of the source video sequence, usually with some errors. The local video decoder (533) may copy the decoding process that may be performed by the video decoder on the reference frames and store the reconstructed reference frames in a reference picture buffer (534). In this way, the encoder (303) may store a copy of the reconstructed reference frames locally, which has a common content (no transmission errors) with the reconstructed reference frames obtained by the remote video decoder.

[0074] The predictor (535) can perform a prediction search for the coding engine (532). That is, for a new frame to be coded, the predictor (535) may search in the reference picture memory (534) for sample data or specific metadata, such as the motion vectors, block shapes, etc., of the reference pictures that can be used as suitable prediction references for the new picture (as candidate reference pixel blocks). The predictor (535) can operate on a pixel block by pixel block basis to find suitable prediction references. In some cases, the input picture may have prediction references extracted from multiple reference pictures stored in the reference picture memory (534), e.g., as determined by the search results obtained by the predictor (535).

[0075] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters for encoding the video data and subgroup parameters.

[0076] The output of all the functional units mentioned above may be entropy coded in an entropy coder (545) which converts the symbols produced by each functional unit into an encoded video sequence by performing lossless compression on the symbols according to techniques well known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0077] The transmitter (540) may buffer the encoded video sequence created by the entropy encoder (545) to prepare it for transmission over a communication channel (560), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (540) may merge the encoded video data from the video encoder (530) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0078] The controller (550) can manage the operation of the encoder (303). During encoding, the controller (550) can assign each encoded picture a particular encoded picture type, which can affect the encoding technique that can be applied to the corresponding picture. For example, a picture can typically be assigned as one of the following frame types:

[0079] An intra picture (I-picture) may be a picture that is coded and decoded without using any other frame in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of these variations of I-pictures and their corresponding uses and characteristics.

[0080] A predicted picture (P-picture) may be a picture that is coded and decoded using intra- or inter-prediction, which may predict the sample values ​​of each block using at most one motion vector and reference index.

[0081] A bidirectionally predicted picture (B-picture) may be a picture that is coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predictive picture may use more than two reference images and associated metadata to reconstruct a single block.

[0082] A source picture may generally be spatially subdivided into a number of sample blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples, respectively) and coded block by block. These blocks may be predictively coded with reference to other (coded) blocks determined by the coding assignment applied to the block's corresponding picture. For example, blocks of I-pictures may be non-predictively coded or they may be predictively coded with reference to coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P-pictures may be predictively coded via spatial or temporal prediction with reference to one previously coded reference picture. Blocks of B-pictures may be predictively coded via spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0083] The video encoder (303) may perform encoding operations based on a given video encoding technique or standard, such as the ITU-T H.265 recommendation. During its operation, the video encoder (303) may perform various compression operations, including predictive encoding operations that rely on temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0084] In an embodiment, the transmitter (540) may transmit additional data along with the encoded video. The video encoder (530) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as, for example, redundant pictures, slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0085] Also, the proposed methods may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium to perform one or more of the proposed methods.

[0086] The techniques described above may be implemented as computer software by computer readable instructions and physically stored on one or more computer readable media. For example, FIG. 6 illustrates a computer system 600 suitable for implementing some embodiments of the disclosed subject matter.

[0087] Computer software can be coded using any suitable machine code or computer language, and can be subjected to mechanisms such as assembling, compiling, linking, etc. to produce code containing instructions that are executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or interpreted, by microcode, etc.

[0088] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0089] 6 are exemplary in nature and are not intended to impose limitations on the scope or functionality of the computer software used to implement embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 600.

[0090] The computer system 600 may include several human-machine interface input devices. Such human-machine interface input devices may be responsive to one or more human user inputs, such as, for example, tactile inputs (e.g., keystrokes, slides, data glove movements), audio inputs (e.g., voice, claps), visual inputs (e.g., gestures), olfactory inputs (not shown), etc. The human-machine interface devices may also be used to capture certain media that are not necessarily directly related to a conscious human input, such as, for example, audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capture devices), and videos (e.g., two-dimensional video, three-dimensional video including stereo video).

[0091] The input man-machine interface devices may include one or more of a keyboard 601, a mouse 602, a touchpad 603, a touch panel 610, a data glove, a joystick 605, a microphone 606, a scanner 607, and a photographing device (camera) 608 (only one of each is shown).

[0092] The computer system 600 may further include man-machine interface output devices. Such man-machine interface output devices may stimulate one or more of the human user's senses, for example, via haptic output, sound, light, and smell / taste. Such man-machine interface output devices include haptic output devices (e.g., haptic feedback via a touch panel 610, data gloves, or joystick 605, although there are also haptic feedback devices that are not used as input devices), audio output devices (e.g., speakers 609, headphones (not shown)), visual output devices (e.g., screens 610, including CRT screens, LCD screens, plasma screens, OLED screens, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), which may or may not have touch panel input and haptic feedback capabilities, some of which output two-dimensional visual output or three or more dimensional output, such as by means of stereoscopic image output), and printers (not shown).

[0093] Computer system 600 may further include human accessible storage devices and associated media, including, for example, optical media such as CD / DVD ROM / RW 620 with media 621 such as CD / DVDs, thumb drives 622, removable hard drives or solid state drives 623, conventional magnetic media such as magnetic tape and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices (e.g., dongles (not shown)), etc.

[0094] Those skilled in the art will appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed in this application does not include transmission media, carrier waves or other instantaneous signals.

[0095] The computer system 600 may further include interface(s) to one or more communication networks. The network may be, for example, a wireless network, a wired network, an optical network. The network may further be a local network, a wide area network, a metropolitan network, an in-vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include local area networks (e.g., Ethernet, wireless LAN), cellular networks (including Global System for Mobile Communications (GSM), 3rd Generation (3G), 4th Generation (4G), 5th Generation (5G), Long Term Evolution (LTE), etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial television), in-vehicle and industrial networks (including CAN Bus), etc. Some networks typically require an adapter to be connected to an external network interface of some general-purpose data port or peripheral bus 649 (e.g., a USB port of computer system 600), while other networks are typically integrated into the core of computer system 600 by being connected to the system bus by, for example, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system, as described below. Computer system 600 can communicate with other entities over any of these networks. Such communications may be one-way receive only (e.g., broadcast television), one-way transmit only (e.g., a CAN bus to a CAN bus device), or bidirectional (e.g., using a local or wide area digital network to reach other computer systems). Specific protocols and protocol stacks may be utilized for each of these networks and network interfaces described above.

[0096] The man-machine interface devices, human-accessible storage devices and network interfaces may be connected to a core 640 of the computer system 600 .

[0097] The core 640 includes one or more central processing units (CPUs) 641, graphic processing units (GPUs) 642, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 643, hardware accelerators for certain tasks 644, etc. These devices are connected via a system bus 648, along with a read only memory (ROM) 645, a random access memory 646, and an internal mass memory 647 (internal non-user accessible hard disk drive, solid state drive (SSD), etc.). In some computer systems, access to the system bus 648 in the form of one or more physical plugs allows expansion with additional CPUs, GPUs, etc. Peripheral devices are connected to the core's system bus 648 directly or via a peripheral bus 649. Peripheral bus architectures include peripheral component interconnect (PCI), USB, etc.

[0098] The CPU 641, the GPU 642, the FPGA 643 and the accelerator 644 can execute some instructions, which, when combined, can constitute the above computer code. The computer code is stored in the ROM 645 or the RAM 646. Temporary data can be stored in the RAM 646, and permanent data can be stored, for example, in the internal mass memory 647. A cache memory can be used to realize fast storage and retrieval in any of the memory devices, and the cache memory can be closely associated with one or more of the CPU 641, the GPU 642, the mass storage memory 647, the ROM 645, the RAM 646, etc.

[0099] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the type well known and available to those skilled in the computer software arts.

[0100] By way of example and not limitation, a computer system having architecture 600, and in particular core 640, may provide functionality by having processor(s) (including CPU, GPU, FPGA, accelerator, etc.) execute software embodied in one or more tangible computer-readable media. Such computer-readable media may be the media related to mass storage devices accessible by users as introduced above, as well as storage devices of a non-transitory nature of core 640, such as core internal mass memory 647 or ROM 645. Software for implementing embodiments of the present disclosure may be stored in such devices and executed by core 640. Depending on the particular needs, computer-readable media may include one or more storage devices or chips. The software may cause core 640, and in particular its processors (including CPU, GPU, FPGA, etc.) to perform certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM 646 and modifying such data structures based on the processes defined by the software. Also or alternatively, the computer system may provide functionality by logic hardwired or otherwise implemented in circuitry (e.g., accelerator 644) that may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may include logic and conversely, references to logic may include software. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) on which software is stored to be executed, circuitry embodying the logic to be executed, or both. The present disclosure includes any appropriate combination of hardware and software.

[0101] While the present disclosure has previously described several exemplary embodiments, there are modifications, substitutions, and various alternative equivalents that are within the scope of the present disclosure. It should therefore be understood that those skilled in the art will be able to devise many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. 1. A method performed by an encoder for video coding, comprising: identifying a reference line set associated with a coding unit; One of the following steps: applying a first intra-interpolation filter to reference samples included in a first reference line to generate a first set of predicted samples based on the first reference line in the reference line set, the first reference line being adjacent to the coding unit, the first reference line being associated with a first reference line index, and the first intra-interpolation filter being selectable from both an edge-preserving filter and an edge-smoothing filter; and applying a second intra-interpolation filter to reference samples included in a second reference line to generate a second set of predicted samples based on the second reference line in the reference line set, the second reference line being not adjacent to the coding unit, the second reference line being associated with a second reference line index, and the second intra-interpolation filter being selectable only from an edge-preserving filter and not an edge-smoothing filter; and encoding the coding unit using the generated first or second set of prediction samples; signaling a reference line index, the reference line index being the first or second reference line index; The method includes:

2. The method of claim 1 , wherein the edge smoothing filter has only positive or zero filter coefficients.

3. the edge smoothing filter includes a linear interpolation filter having only positive or zero filter coefficients; The method of claim 1 , wherein the edge-preserving filter comprises at least one of a linear interpolation filter, a non-linear filter, or a bilateral filter having at least one or two negative filter coefficients.

4. The method of claim 1 , wherein the first reference line has a first reference line index having a value of 0.

5. The method of claim 2 , wherein the edge-preserving filter comprises at least one of a cubic filter, a discrete cosine transform (DCT) based filter, a polynomial-based interpolation filter, a bilateral filter, and a Hermite interpolation filter.

6. The method of claim 1 , wherein the second reference line has a second reference line index having a non-zero value.

7. The method of claim 1 , wherein a first number of taps of the edge-smoothing filter is different from a second number of taps of the edge-preserving filter.

8. The method of claim 1 , wherein the edge smoothing filter comprises a bilinear filter or a Gaussian filter.

9. 2. The method of claim 1 , wherein the edge-preserving filter applied to the reference samples included in the first reference line is a first edge-preserving filter having a first set of filter coefficients, and the edge-preserving filter applied to the reference samples included in the second reference line is a second edge-preserving filter having a second set of filter coefficients different from the first set of filter coefficients.

10. The method of claim 1 , further comprising, after the step of signaling the reference line index, signaling an intra-prediction mode, wherein the signaled intra-prediction mode excludes a planar mode.

11. 1. A method performed by an encoder for video coding, comprising: identifying a reference line set associated with a coding unit; One of the following steps: applying a first intra-interpolation filter to reference samples included in a first reference line to generate a first set of predicted samples based on the first reference line in the reference line set, the first reference line being adjacent to the coding unit, the first reference line being associated with a first reference line index, and the first intra-interpolation filter being selectable from both an edge-preserving filter and an edge-smoothing filter; and applying a second intra-interpolation filter to reference samples included in a second reference line to generate a second set of predicted samples based on the second reference line in the reference line set, the second reference line being not adjacent to the coding unit, the second reference line being associated with a second reference line index, and the second intra-interpolation filter being selectable only from an edge-preserving filter and not an edge-smoothing filter; and encoding the coding unit using the generated first or second set of prediction samples; transmitting or storing an encoded video bitstream including the encoded coding unit and a reference line index that is the first or second reference line index; The method includes:

12. 1. A method performed by a decoder for video coding, comprising: identifying a reference line set associated with a coding unit; decoding, from the encoded video bitstream, a reference line index associated with the coding unit, the reference line index being a first or a second reference line index; One of the following steps: applying a first intra-interpolation filter to reference samples included in a first reference line to generate a first set of predicted samples based on a first reference line in the reference line set associated with a first reference line index, the first reference line being adjacent to the coding unit, and the first intra-interpolation filter being selectable from both an edge-preserving filter and an edge-smoothing filter; and applying a second intra-interpolation filter to reference samples included in a second reference line to generate a second set of predicted samples based on a second reference line in the reference line set associated with a second reference line index, the second reference line not adjacent to the coding unit, and the second intra-interpolation filter is selectable only from an edge-preserving filter and not an edge-smoothing filter; and The method includes:

13. 1. An apparatus for video encoding, comprising: at least one memory configured to store program code; at least one processor coupled to the at least one memory; The program code, when executed by the at least one processor, causes the at least one processor to perform the method of any one of claims 1 to 11. Device.

14. 1. An apparatus for video decoding, comprising: at least one memory configured to store program code; at least one processor coupled to the at least one memory; The program code, when executed by the at least one processor, causes the at least one processor to perform the method of claim 12. Device.

15. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 11.

16. A computer program product for causing a computer to carry out the method according to claim 12.

Images