Method, apparatus, and computer program for selecting an intra-interpolation filter for multi-line intra-prediction

Abstract

To provide a method and apparatus for selecting an intra interpolation filter for multi-line intra prediction based on a reference line index for decoding a video sequence.SOLUTION: A method includes: identifying a set of reference lines associated with a coding unit; applying a first type of interpolation filter to reference samples included in a first reference line, of the set of reference lines, that is adjacent to the coding unit to generate a first set of prediction samples based on the first reference line being associated with a first reference line index; and applying a second type of interpolation filter to reference samples included in a second reference line, of the set of reference lines, that is non-adjacent to the coding unit to generate a second set of prediction samples based on the second reference line being associated with a second reference line index.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) Pursuant to 35 U.S.C. 119, this application claims priority to U.S. Provisional Application No. 62 / 729,395, filed September 10, 2018, and U.S. Patent Application No. 16 / 202,902, filed November 28, 2018, with the U.S. Patent and Trademark Office, the disclosures of which are incorporated herein by reference in their entireties.

[0002] This disclosure relates to a set of advanced video coding techniques. More specifically, this disclosure provides a modified intra-interpolation filter scheme for multi-line intra-prediction. [Background technology]

[0003] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC l / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), and provided updates in 2014 (version 2), 2015 (version 3), and 2016 (version 4). Since then, the ITU has been studying the potential need for standardization of future video coding technologies with compression capabilities significantly exceeding the HEVC standard (including extensions).

[0004] In October 2017, the ITU published a Joint Proposal (CfP) for video compression with capabilities beyond HEVC. By February 15, 2018, a total of 22 CfP responses had been submitted for standard dynamic range (SDR), 12 for high dynamic range (HDR), and 12 for the 360 ​​video category.

[0005] April 2018, 122 MPEG / 10 thAt a meeting of the Joint Video Exploration Team (JVET), all received CfP responses were evaluated. After careful evaluation, the JVET officially launched the standardization of the next generation of video coding beyond HEVC, known as Versatile Video Coding (VVC).

[0006] HEVC has a total of 35 intra prediction modes, with mode 10 being the horizontal mode, mode 26 being the vertical mode, and modes 2, 18, and 34 being the diagonal modes. The intra prediction modes are signaled by three most probable modes (MPMs) and the remaining 32 modes.

[0007] In the current development of VVC, there are a total of 87 intra-prediction modes, with mode 18 being the horizontal mode, mode 50 being the vertical mode, and modes 2, 34, and 66 being the diagonal modes. Modes 1 to 10 and modes 67 to 76 are designated as wide-angle intra-prediction (WAIP) modes.

[0008] To encode an intra mode, a most probable mode (MPM) list of three modes is generated based on the intra modes of neighboring blocks. This MPM list is called the MPM list or primary MPM list. If the intra mode is not included in the MPM list, a flag is signaled to indicate whether the intra mode belongs to the selected mode.

[0009] In the development of VVC, a method for realizing a primary MPM list and a secondary MPM list is proposed. The modes in the secondary MPM list are not included in the primary MPM list. The number of modes in the MPM list may be 3, 4, 5, 6, 7, 8, etc., and the number of modes in the secondary MPM list may be 8, 16, 32, etc.

[0010] In VVC, for the luma component, neighboring samples used to generate intra-predicted samples are filtered (i.e., intra-smoothing processing) before the generation process. The filtering is controlled by a given intra-prediction mode and transform block size. If the intra-prediction mode is DC or the transform block size is 4x4, neighboring samples are not filtered. Also, if the distance between a given intra-prediction mode and the vertical mode (or horizontal mode) is greater than a predetermined threshold, the filtering process is started. A [1,2,1] filter and a bilinear filter are used to filter neighboring samples. For example, Article 8.2.4.2.4 and Table 8-4 of VVC Draft 2 describe the intra-smoothing processing proposed in VVC.

[0011] Multiline intra prediction was proposed to use additional reference lines for intra prediction. The encoder can determine and signal the reference lines used to generate the intra predictor. The encoder signals the reference line index before the intra prediction mode, and excludes planar mode / DC mode from the intra prediction modes if a non-zero reference line index is signaled.

[0012] Wider angles beyond the range of prediction directions covered by conventional intra prediction modes have been proposed and are called wide-angle intra prediction modes. These wide angles only apply to non-square blocks: If the block width is greater than the block height, the angle is greater than 45 degrees to the upper right (HEVC intra prediction mode 34); If the block height is greater than the block width, the angle is greater than 45 degrees to the lower left (HEVC intra prediction mode 2).

[0013] The original method is used to signal the replaced modes and remap them to wide-angle mode indices after parsing. The total number of intra prediction modes remains unchanged, i.e., 35 for VTM-1.0 and 67 for BMS-1.0, and the coding of intra modes remains unchanged.

[0014] A bilateral filter is a nonlinear, edge-preserving, noise-reducing smoothing filter used on images. It replaces the luminance of each pixel using a weighted average of luminance values ​​from neighboring pixels. The weights can be based on a Gaussian distribution. The weights depend on the Euclidean distance of the pixels and, in turn, on radiance differences (e.g., differences in range of color intensity, depth distance, etc.). Bilateral filters help preserve sharp edges. Given the original (unfiltered) reference samples I(x) of an intra block, the bilateral filter function is defined as follows: JPEG0007807179000001.jpg29151

[0015] 4-tap and 6-tap intra-interpolation filters have been proposed to generate prediction samples used in directional intra-prediction. Two types of 4-tap interpolation filters are used: cubic and Gaussian. The cubic filter is suitable for preserving image edges, while the Gaussian filter is suitable for removing image noise. The cubic interpolation filter is implemented when the intra-prediction mode is a diagonal mode (i.e., mode 34) or higher and the block width is 8 or less, and when the intra-prediction mode is a diagonal mode (i.e., mode 34) or lower and the block height is 8 or less.

[0016] The Gaussian filter is implemented when the intra prediction mode is a diagonal mode (i.e., mode 34) or higher and the block width is greater than 8, and when the intra prediction mode is a diagonal mode (i.e., mode 34) or lower and the block height is greater than 8. Summary of the Invention

[0017] A method for selecting an intra-interpolation filter for multi-line intra-prediction based on a reference line index for decoding a video sequence, comprising: identifying a set of reference lines associated with a coding unit; applying a first type of interpolation filter to reference samples included in a first reference line that is adjacent to the coding unit based on a first reference line index being associated with the first reference line in the set of reference lines; and applying a second type of interpolation filter to reference samples included in a second reference line that is not adjacent to the coding unit based on a second reference line index being associated with the second reference line in the set of reference lines, to generate a second set of predicted samples.

[0018] 1. An apparatus for selecting an intra-interpolation filter for multi-line intra prediction based on a reference line index for decoding a video sequence, the apparatus comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to instructions of the program code, the program code including: an identification code configured to cause the at least one processor to identify a set of reference lines associated with a coding unit; a first application code configured to cause the at least one processor to apply a first type of interpolation filter to reference samples included in a first reference line that is adjacent to the coding unit, based on the first reference line being associated with a first reference line index; and a second application code configured to cause the at least one processor to apply a second type of interpolation filter to reference samples included in a second reference line that is not adjacent to the coding unit, based on the second reference line being associated with a second reference line index.

[0019] 10. A non-transitory computer-readable medium storing instructions, the instructions including one or more instructions that, when executed by one or more processors of an apparatus for selecting an intra-interpolation filter for multi-line intra prediction based on a reference line index for decoding a video sequence, cause the one or more processors to perform the following operations: identify a set of reference lines associated with a coding unit; apply a first type of interpolation filter to reference samples included in a first reference line that is adjacent to the coding unit based on a first reference line index being associated with the first reference line; and apply a second type of interpolation filter to reference samples included in a second reference line that is not adjacent to the coding unit based on a second reference line index being associated with the second reference line; [Brief explanation of the drawings]

[0020] Other features, characteristics and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which: [Figure 1] 10 is a flowchart of an example process for selecting an intra-interpolation filter for multi-line intra prediction based on a reference line index for decoding a video sequence. [Figure 2] 1 is a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a diagram of a video encoder and decoder arrangement in a streaming environment. [Figure 4] FIG. 2 is a functional block diagram of a video decoder according to an embodiment of the present disclosure. [Figure 5] FIG. 2 is a functional block diagram of a video encoder according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] Gaussian filters are suitable for smooth image regions, while cubic filters are suitable for regions with edges. The addition of additional reference lines used in multi-line intra prediction helps with regions with edges. However, applying both the same cubic and Gaussian filters to all reference lines is not a desirable design.

[0022] 1 is a flowchart of an example process 100 for selecting an intra-interpolation filter for multi-line intra-prediction based on a reference line index for decoding a video sequence. In some implementations, one or more processing blocks of FIG. 1 may be performed by a decoder. In some implementations, one or more processing blocks of FIG. 1 may be performed by another device or group of devices (e.g., an encoder) separate from or including the decoder.

[0023] As shown in FIG. 1, the process 100 may include identifying a set of reference lines associated with a coding unit (block 110).

[0024] In some implementations, the line index of the closest reference line is 0 (zero reference line), and the maximum number of reference lines signaled is represented as N. Hereinafter, an intra-interpolation filter, as described in, is an interpolation filter for generating a predicted value that points to a fractional position of a reference sample.

[0025] In some implementations, the selection of an intra-interpolation filter depends on the signaled reference line index. Types of intra-interpolation filters include edge-preserving filters and edge-smoothing filters and / or similar filters. Edge-smoothing filters include linear interpolation filters with positive or zero filter coefficients. Edge-preserving filters include linear interpolation filters or nonlinear filters with at least one or two negative filter coefficients, such as bilateral filters.

[0026] As further shown in FIG. 1, process 100 may include applying an interpolation filter of a first type to reference samples included in a first reference line that is adjacent to the coding unit based on the first reference line in the set of reference lines being associated with a first reference line index to generate a first set of predicted samples (block 120), and applying an interpolation filter of a second type to reference samples included in a second reference line that is not adjacent to the coding unit based on the second reference line in the set of reference lines being associated with a second reference line index to generate a second set of predicted samples (block 130).

[0027] According to an embodiment, both an edge smoothing filter and an edge preserving filter are applied to the zero reference line, and only an edge preserving filter is applied to the non-zero reference line, and the edge smoothing filter may include a bilinear filter or a Gaussian filter.

[0028] According to an embodiment, the edge-preserving filter may be a cubic interpolation filter, a DCT-based interpolation filter (DCT-IF), a 4 / 6-tap polynomial-based interpolation filter, a bilateral filter, a Hermite interpolation filter and / or similar filters.

[0029] According to an embodiment, the edge-preserving filters used for different lines are different. For example, the taps of the edge-preserving filter may be different for the zero reference line and the non-zero reference line. Additionally or alternatively, an M-tap edge-preserving filter is used for the zero reference line and an N-tap edge-preserving filter is used for the non-zero reference line (e.g., M and N are positive integers, and M is not equal to N). Additionally or alternatively, the filter coefficients of the edge-preserving filters used for different reference lines may be different.

[0030] In another embodiment, the coefficients of the edge-preserving and edge-smoothing filters are fixed and do not depend on the sample values.

[0031] In some implementations, the selection of the intra-interpolation filter depends on the reference line index and other coded information, or any information available at both the encoder and decoder, such as the intra-prediction mode and block size.

[0032] According to an embodiment, an edge-preserving filter is used based on a satisfied condition. For example, the condition may be satisfied if the intra prediction mode is equal to or greater than a diagonal mode (i.e., mode 34) and the width of the block is equal to or less than S. Additionally or alternatively, the condition may be satisfied if the intra prediction mode is equal to or less than a diagonal mode (i.e., mode 34) and the height of the block is equal to or less than S. For example, S may indicate a threshold value for the block width or block height. For different lines, S may be different. For example, according to an embodiment, for a zero reference line, S is 8, and for a non-zero reference line, S is 16 or 32.

[0033] According to an embodiment, an edge smoothing filter is used for wide angles at the zero reference line and an edge preserving filter is used for wide angles at the non-zero reference line. Additionally or alternatively, an edge preserving filter is used for wide angles at the zero reference line and an edge smoothing filter is used for wide angles at the non-zero reference line.

[0034] According to an embodiment, both an edge-preserving filter and an edge-smoothing filter are applied to the zero reference line, and / or an edge-preserving filter or an edge-smoothing filter is applied to at least one reference line. In this case, it is possible to apply no two filters. According to an embodiment, if there are three reference lines, the line indices may be {0,1,2} or {0,1,3}, and both an edge-preserving filter and an edge-smoothing filter are applied to line 0, only an edge-preserving filter or an edge-smoothing filter is applied to line 1, and only an edge-smoothing filter or an edge-preserving filter is applied to line 2 or line 3.

[0035] According to an embodiment, the edge-preserving filter and the edge-smoothing filter are applied to all reference lines, but the number of taps of the intra-interpolation filter may vary from line to line. For example, in an embodiment, the edge-preserving filter used for the zero reference line is M taps, and the edge-preserving filter used for the non-zero reference lines is N taps (e.g., M and N are positive integers, and M is not equal to N, e.g., M=6 and N=4). Additionally or alternatively, the edge-smoothing filter used for the zero reference line is M taps, and the edge-smoothing filter used for the non-zero reference lines is N taps (e.g., M and N are positive integers, and M is not equal to N, e.g., M=6 and N=4).

[0036] According to an embodiment, for some reference lines, wide angles are prohibited or limited differently. For example, for non-zero reference lines, wide angles are prohibited. Additionally or alternatively, at least one reference line does not use wide angles. Additionally or alternatively, the wide angles used for different reference lines are different. For example, the number of conventional angular intra-prediction directions replaced by wide-angle intra-prediction directions depends on the reference line index.

[0037] According to an embodiment, the intra-smoothing filter may be different for each line. For example, a bilateral filter is used for intra-smoothing of non-zero reference lines. Additionally or alternatively, when signaling a reference line index, the intra-smoothing filter is inhibited for non-zero reference lines. Additionally or alternatively, at least one reference line is intra-smoothed using a bilateral filter.

[0038] According to an embodiment, the number of filter taps of the intra-smoothing filter is different for each line. For example, the number of filter taps of the intra-smoothing filter used for the zero reference line is M, and the number of filter taps of the intra-smoothing filter used for the non-zero reference line is N (for example, M and N are positive integers, and M is not equal to N, for example, M=6 and N=4).

[0039] According to an embodiment, the intra prediction mode that applies intra smoothing varies from line to line. For example, a threshold T defines which intra prediction mode applies intra smoothing if the intra prediction mode Mode satisfies the following condition: min(abs(Mode-Hor),abs(Mode-Ver)) <Tであり、 where "Hor" indicates an intra-prediction mode index for horizontal mode, and "Ver" indicates an intra-prediction mode index for vertical mode. The value of T depends on the reference line index and other coded information, or any information known to both the encoder and decoder, including, but not limited to, block region size, block width, block height, block aspect ratio, etc.

[0040] According to an embodiment, for non-zero reference lines, only MPM modes, including a primary MPM list and a secondary MPM list, are allowed. In one embodiment, if the reference line index of the current block is greater than 0, one bin is signaled, called "primary_mpm_flag," to indicate whether the intra prediction mode of the current block belongs to the primary or secondary MPM list. If "primary_mpm_flag" is "true," the primary MPM index is signaled. Otherwise, the secondary MPM index is signaled. Non-MPM modes are not used for non-zero reference lines, and no flag is signaled to indicate whether to use secondary MPM or non-MPM.

[0041] According to an embodiment, planar and DC modes are excluded from the primary and secondary MPM lists.

[0042] According to embodiments, some implementations encode the first bin of the reference line index using multiple contexts. For example, the selection of the context depends on the reference line index of the neighboring block. As a specific example, if both the reference line index of the left block and the reference line index of the block above are equal to 0, select context 0; and if both the reference line index of the left block and the reference line index of the block above are not equal to 0, select context 1. Otherwise, select context 2.

[0043] According to an embodiment, the selection of the context depends on the CBF (Coded Block Flag) of the neighboring block. The CBF is an indicator of whether the current block contains non-zero coefficients. If the CBF is equal to 0, there are no non-zero coefficients in the current block. In one example, if the CBF of the left block and the CBF of the block above are both equal to 0, then select context 0. And if the CBF of the left block and the CBF of the block above are both not equal to 0, then select context 1. Otherwise, select context 2.

[0044] According to an embodiment, the context for encoding the transform selection information according to the reference line index value includes, but is not limited to, an MTS flag, an MTS index, and an NSST index.

[0045] Although Figure 1 illustrates example blocks of process 100, in some implementations, process 100 may include other, fewer, different, or differently arranged blocks compared to those blocks illustrated in Figure 1. Additionally, or alternatively, two or more blocks of process 100 may be performed in parallel.

[0046] FIG. 2 shows a schematic block diagram of a communication system (200) according to an embodiment of the present disclosure. The communication system (200) may include at least two terminals (210-220) interconnected via a network (250). In the case of one-way data transmission, a first terminal (210) can locally encode video data and transmit it to another terminal (220) via the network (250). The second terminal (220) can receive the encoded video data of the other terminal from the network (250), decode the encoded data, and display the decoded video data. One-way data transmission is common in media service applications, for example.

[0047] 2 illustrates a second pair of terminals (230, 240) provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal (230, 240) may encode video data captured at a local location and transmit it to the other terminal over the network (250). Each terminal (230, 240) may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the decoded video data on a local display device.

[0048] In FIG. 2 , the terminals (210-240) are illustrated as servers, personal computers, and smartphones, although the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure apply to laptop computers, tablets, media players, and / or dedicated videoconferencing equipment. The network (250) represents any number of networks transmitting encoded video data between the terminals (210-240), including, for example, wired and / or wireless communication networks. The communication network (250) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of the network (250) may not be important to the operation of the present disclosure, unless otherwise described herein.

[0049] As an example application of the disclosed subject matter, FIG. 3 illustrates a video encoder and decoder arrangement in a streaming environment; the disclosed subject matter can be equally utilized in other applications that support video, including, for example, video conferencing, digital TV, storage of compressed video on digital media such as CDs, DVDs, memory sticks, etc.

[0050] The streaming system can include a capture subsystem (313), which may include a video source (301) (e.g., a digital camera) for creating an uncompressed video sample stream (302). The video sample stream (302) is depicted as a bold line to emphasize its large amount of data compared to an encoded video bitstream. The sample stream (302) can be processed by an encoder (303) coupled to the camera (301). The encoder (303) can include hardware, software, or a combination thereof to realize or implement aspects of the disclosed subject matter, which are described in more detail below. The encoded video bitstream (304) is depicted as a thin line to emphasize its small amount of data compared to the sample stream. The encoded video bitstream (304) can be stored on a streaming server (305) for future use. One or more streaming clients (306, 308) can access the streaming server (305) to retrieve copies (307, 309) of the encoded video bitstream (304). The client (306) can include a video decoder (310) that decodes the incoming copy of the encoded video bitstream (307) and creates an outgoing video sample stream (311) that can be rendered on a display (312) or other rendering device (not shown). In some streaming systems, the video bitstreams (304, 307, 309) can be encoded according to specific video encoding / compression standards. Examples of these standards include the ITU-T H.265 recommendation. A video encoding standard informally referred to as Versatile Video Coding is under development. The disclosed subject matter can be used in the context of VVC.

[0051] FIG. 4 may be a functional block diagram of a video decoder (310) according to an embodiment of the present disclosure.

[0052] The receiver (410) can receive one or more codec video sequences to be decoded by the decoder (310). In the same or another embodiment, it receives one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences can be received from a channel (412), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (410) may receive coded video data and other data, such as coded audio data and / or auxiliary data streams, which can be forwarded to respective using entities (not shown). The receiver (410) can separate the coded video sequences from other data. To prevent network jitter, a buffer memory (415) can be coupled between the receiver (410) and the entropy decoder / parser (420) (hereinafter referred to as the "parser"). If the receiver 410 receives data from a store-and-forward device or an isochronous real-time network with sufficient bandwidth and controllability, the buffer 415 may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer 415 may be needed, and the buffer 415 may be relatively large and advantageously have a self-adaptive size.

[0053] The video decoder (310) may include a parser (420) to reconstruct symbols (421) based on the entropy-coded video sequence. These symbol categories include information for managing the operation of the decoder (310) and latent information for controlling rendering devices that are not part of the decoder but may be coupled to it, such as a display (312). As shown in FIG. 4, the control information for one or more rendering devices may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may conform to a video coding technique or standard and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context-sensitive coding, etc. The parser (420) can extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The entropy decoder / parser may also extract, for example, transform coefficients, quantizer parameter (QP) values, motion vectors, etc. from the coded video sequence information.

[0054] The parser (420) may perform entropy decoding / parsing operations on the video sequence received from the buffer (415) to create symbols (421). The parser (420) may receive the encoded data and selectively decode particular symbols (421). The parser (420) may also determine whether a particular symbol (421) is to be provided to the motion compensated prediction unit (453), the scaler / inverse transform unit (451), the intra prediction unit (452), or the loop filter (456).

[0055] The reconstruction of the symbols (421) may involve several different units, depending on the type of coded video picture or some type of coded video picture (e.g., inter-picture vs. intra-picture, inter-block vs. intra-block), and other factors. The units involved, and the manner in which they are involved, may be controlled by subgroup control information parsed by the parser (420) from the coded video sequence. For simplicity, the flow of such subgroup control information between the parser (420) and the following units is not described.

[0056] In addition to the functional blocks already mentioned, the decoder (310) can be conceptually subdivided into a number of functional units, as described below. In actual implementations implemented under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0057] The first unit is a scalar / inverse transform unit (451), which receives quantized transform coefficients as symbol(s) (421) and control information from the parser (420), including the transform method to be used, block size, quantization coefficients, quantization scaling matrix, etc. The scalar / inverse transform unit (451) can output a block containing sample values, which can be input to an aggregator (455).

[0058] In some cases, the output samples of the scaler / inverse transform unit (451) may belong to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but do use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be provided by an intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) uses surrounding already reconstructed information extracted from the current (partially reconstructed) picture (456) to generate blocks of the same size and shape as the block being reconstructed. In some cases, the aggregator (455) adds, based on each sample, the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451).

[0059] In other cases, the output samples of the scaler / inverse transform unit (451) may belong to an inter-coded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (453) may access a reference picture memory (457) to extract samples for prediction. After performing motion compensation on the extracted samples based on the symbols (421) belonging to the block, these samples can be added by an aggregator (455) to the output of the scaler / inverse transform unit (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory from which the motion compensation unit extracts prediction samples may be controlled by motion vectors, which may be used by the motion compensation unit in the form of symbols (421), which may have, for example, X, Y, and reference picture components. Motion compensation may include interpolation of sample values ​​extracted from the reference picture memory, motion vector prediction mechanisms, etc., when sub-sample accurate motion vectors are used.

[0060] The output samples of the aggregator (455) may be processed by various loop filtering techniques in a loop filter unit (456). The video compression techniques may include in-loop filtering techniques controlled by parameters included in the coded video bitstream and made available to the loop filter unit (456) as symbols (421) from the parser (420), but may also respond to meta-information obtained during decoding of a previous portion (in decoding order) of the coded picture or coded video sequence, or may respond to previously reconstructed loop-filtered sample values.

[0061] The output of the loop filter unit (456) may be a sample stream, which may be output to the rendering device (312) and stored in the reference picture memory (456) for use in future inter-picture prediction.

[0062] Once fully reconstructed, some coded pictures can be used as reference pictures for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current reference picture (656) can become part of the reference picture buffer (457), and a new current picture memory can be reallocated before beginning reconstruction of a subsequent coded picture.

[0063] The video decoder (310) may perform decoding operations according to a predetermined video compression technology, such as that documented in the ITU-T H.265 Recommendation. A coded video sequence may conform to the syntax specified by the video compression technology or standard being used, in the sense that the coded video sequence conforms to the syntax of the video compression technology or standard (as specified in the video compression technology document or standard, and particularly as specified in the documentation files therein). Compliance also requires that the complexity of the coded video sequence be within the range limited by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0064] In embodiments, the receiver (410) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of one or more encoded video sequences. The additional data may be used by the video decoder (310) to accurately decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0065] FIG. 5 may be a functional block diagram of a video encoder (303) according to an embodiment of the present disclosure.

[0066] The encoder (303) may receive video samples from a video source (301) (not part of the encoder), which may capture one or more video images to be encoded by the encoder (303).

[0067] The video source (301) may provide a source video sequence in the form of a digital video sample stream to be encoded by the encoder (303). The digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any suitable color space (e.g., BT.601 Y CrCb, RGB, etc.), and any suitable sampling configuration (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media services system, the video source (301) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (503) may be a camera that captures local image information as a video sequence. Video data may be provided as multiple individual pictures that, when viewed sequentially, impart motion. These pictures may themselves be organized as a spatial pixel array, where each pixel may contain one or more samples, depending on the sampling configuration, color space, etc. used. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion will focus on samples.

[0068] According to an embodiment, the encoder (303) may encode and compress pictures of a source video sequence into an encoded video sequence (543) in real time or under any other time constraint required by the application. Enforcing the appropriate encoding rate is one of the functions of the controller (550). The controller controls and is operatively coupled to other functional units described below. For simplicity, coupling is not depicted. Parameters set by the controller may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will readily recognize other functions of the controller (550), which relate to optimizing the video encoder (303) for a particular system design.

[0069] Some video encoders operate in a manner that those skilled in the art will readily recognize as a "coding loop." As a highly simplified description, the encoding loop may include an encoding portion of an encoder (530) (hereafter referred to as the "source encoder") (responsible for creating symbols based on the input picture to be coded and one or more reference pictures), and a (local) decoder (533) embedded in the encoder (303) that reconstructs the symbols to create sample data that is also created by a (remote) decoder (since the compression between the symbols and the coded video bitstream is lossless in the video compression techniques contemplated in the disclosed subject matter). This reconstructed sample stream is input into a reference picture memory (534). Because decoding the symbol stream produces results that are bit-accurate regardless of the location of the decoder (local or remote), the contents of the reference picture buffer are also bit-accurate between the local and remote encoders. That is, the reference picture samples that the prediction part of the encoder "sees" are exactly the same sample values ​​that the decoder "sees" when using the prediction during decoding. The basic principle of reference picture synchrony (and the drift that occurs when synchrony cannot be maintained, e.g., due to channel error) is well known to those skilled in the art.

[0070] The operation of the "local" decoder (533) may be the same as the operation of the "remote" decoder (310) described in detail above with reference to Figure 4. However, and referring briefly to Figure 5, the entropy decoding portion of the decoder (310), including the channel (412), receiver (410), buffer (415), and parser (420), may not be fully implemented in the local decoder (533), provided that symbols are available and the entropy coder (545) and parser (420) can losslessly encode / decode the symbols into an encoded video sequence.

[0071] In this case, in addition to the analysis / entropy decoding present in the decoder, any decoder techniques will necessarily be found to exist in the corresponding encoder in essentially the same functional form. The description of the encoder technique can be simplified because the encoder technique and the fully described decoder technique are the inverse of each other. Further details are only necessary in certain areas and are provided below.

[0072] As part of its operation, the source encoder (530) may perform motion-compensated predictive coding, which predictively codes an input frame by referencing one or more previously coded frames from the video sequence designated as "reference frames." In this manner, the encoding engine (532) may code differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references for the input frame.

[0073] The local video decoder (533) may decode the encoded video data of frames that may be designated as reference frames based on symbols created by the source encoder (530). The operation of the encoding engine (532) may advantageously be a lossy process. When the encoded video data is decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence may typically be a copy of the source video sequence, with some errors. The local video decoder (533) may copy the decoding process that may be performed by the video decoder on the reference frames and store the reconstructed reference frames in a reference picture buffer (534). In this way, the encoder (303) can locally store copies of reconstructed reference frames that have common content (i.e., are free of transmission errors) with the reconstructed reference frames obtained by the remote video decoder.

[0074] The predictor (535) can perform a prediction search for the coding engine (532). That is, for a new frame to be coded, the predictor (535) may search the reference picture memory (534) for sample data or specific metadata, such as the reference picture's motion vectors, block shape, etc., that can be used as suitable prediction references for the new picture (as candidate reference pixel blocks). The predictor (535) can operate pixel block by pixel block based on the sample blocks to find suitable prediction references. In some cases, for example, as determined by the search results obtained by the predictor (535), the input picture may have prediction references extracted from multiple reference pictures stored in the reference picture memory (534).

[0075] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters for encoding video data and subgroup parameters.

[0076] The output of all of the functional units mentioned above may be entropy coded in an entropy coder (545), which converts the symbols produced by each functional unit into an encoded video sequence by performing lossless compression on the symbols according to techniques well known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0077] The transmitter (540) may buffer the encoded video sequence created by the entropy encoder (545) to prepare it for transmission over a communication channel (560), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (540) may merge the encoded video data from the video encoder (530) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0078] The controller (550) can manage the operation of the encoder (303). During encoding, the controller (550) can assign each coded picture a particular coded picture type, which can affect the coding technique that can be applied to the corresponding picture. For example, pictures can typically be assigned as one of the following frame types:

[0079] An intra-picture (I-picture) may be a picture that is coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of these variations of I-pictures and their corresponding uses and characteristics.

[0080] A predicted picture (P picture) may be a picture that is coded and decoded using intra- or inter-prediction, which may predict the sample values ​​of each block using at most one motion vector and reference index.

[0081] A bidirectionally predicted picture (B-picture) may be a picture that is coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predicted picture may use more than two reference images and associated metadata to reconstruct a single block.

[0082] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. These blocks may be predictively coded with reference to other (coded) blocks as determined by the coding assignment applied to the block's corresponding picture. For example, blocks of an I-picture may be non-predictively coded, or they may be predictively coded with reference to coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0083] The video encoder (303) may perform encoding operations based on a predetermined video encoding technique or standard, such as the ITU-T H.265 recommendation. During its operation, the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0084] In embodiments, the transmitter (540) may transmit additional data along with the encoded video. The video encoder (530) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures, slices, Supplemental Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0085] Additionally, the proposed methods may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium to perform one or more of the proposed methods.

[0086] The techniques described above may be implemented as computer software by computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 6 illustrates a computer system 600 suitable for implementing some embodiments of the disclosed subject matter.

[0087] Computer software can be coded using any suitable machine code or computer language, and can be subjected to mechanisms such as assembling, compiling, linking, etc. to create code containing instructions that are executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or that are interpreted, executed by microcode, etc.

[0088] The instructions may be executed by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0089] 6 are exemplary in nature and are not intended to impose limitations on the scope or functionality of the computer software used to implement embodiments of the present disclosure. Neither the arrangement of components should be interpreted as having any dependency or requirement regarding any one or combination of components shown in the exemplary embodiment of computer system 600.

[0090] The computer system 600 may include several human-machine interface input devices. Such human-machine interface input devices may be responsive to one or more human user inputs, such as tactile input (e.g., keystrokes, slides, data glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), olfactory input (not shown), etc. The human-machine interface devices may also be used to capture certain media not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capture devices), and video (e.g., two-dimensional video, three-dimensional video including stereo video).

[0091] The input human-machine interface device may include one or more of a keyboard 601, a mouse 602, a touchpad 603, a touch panel 610, a data glove, a joystick 605, a microphone 606, a scanner 607, and a photographing device (camera) 608 (only one of each is shown).

[0092] The computer system 600 may also include human-machine interface output devices. Such human-machine interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human-machine interface output devices include haptic output devices (e.g., haptic feedback via a touchscreen 610, data gloves, or joystick 605; although haptic feedback devices that are not used as input devices also exist), audio output devices (e.g., speakers 609, headphones (not shown)), visual output devices (e.g., screens 610, including CRT screens, LCD screens, plasma screens, and OLED screens, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), which may or may not have touchscreen input and haptic feedback capabilities, some of which may provide two-dimensional visual output or three-dimensional or higher-dimensional output, such as by means of stereoscopic output), and printers (not shown).

[0093] Computer system 600 may further include human-accessible storage devices and associated media, including, for example, optical media such as CD / DVD ROM / RW 620 with media 621 such as CD / DVD, thumb drive 622, removable hard drive or solid state drive 623, conventional magnetic media such as magnetic tape and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices (e.g., dongles (not shown)), etc.

[0094] Those skilled in the art will appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transient signals.

[0095] The computer system 600 may further include interface(s) to one or more communication networks. The network may be, for example, a wireless network, a wired network, or an optical network. The network may also be a local network, a wide area network, a metropolitan network, an in-vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include local area networks (e.g., Ethernet, Wireless LAN), cellular networks (including Global System for Mobile Communications (GSM), Third Generation (3G), Fourth Generation (4G), Fifth Generation (5G), Long Term Evolution (LTE), etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial television), in-vehicle and industrial networks (including CAN Bus), etc. Some networks typically require an adapter to connect to an external network interface of a general-purpose data port or peripheral bus 649 (e.g., a USB port on computer system 600), while other networks are typically integrated into the core of computer system 600 by connecting to the system bus via, for example, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system, as described below. Computer system 600 can communicate with other entities over any of these networks. Such communication may be one-way receive only (e.g., broadcast television), one-way transmit only (e.g., a CAN bus to a CAN bus device), or bidirectional (e.g., using a local-area or wide-area digital network to reach another computer system). Specific protocols and protocol stacks may be utilized for each of these networks and network interfaces.

[0096] The human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to a core 640 of the computer system 600 .

[0097] The core 640 includes one or more central processing units (CPUs) 641, graphics processing units (GPUs) 642, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 643, hardware accelerators for certain tasks 644, etc. These devices, along with read-only memory (ROM) 645, random access memory 646, and internal mass memory 647 (such as an internal non-user-accessible hard disk drive or solid-state drive (SSD)), are connected via a system bus 648. In some computer systems, access to the system bus 648 in the form of one or more physical plugs allows expansion with additional CPUs, GPUs, etc. Peripheral devices connect to the core's system bus 648 either directly or via a peripheral bus 649. Peripheral bus architectures include Peripheral Component Interconnect (PCI), USB, etc.

[0098] The CPU 641, GPU 642, FPGA 643, and accelerator 644 can execute several instructions, which, when combined, constitute the above-mentioned computer code. The computer code is stored in ROM 645 or RAM 646. Temporary data may be stored in RAM 646, and permanent data may be stored, for example, in internal mass memory 647. Cache memory can be used to achieve fast storage and retrieval from any of the memory devices, and the cache memory may be closely associated with one or more of the CPU 641, GPU 642, mass storage memory 647, ROM 645, RAM 646, etc.

[0099] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the type well known and available to those skilled in the computer software arts.

[0100] By way of example and not limitation, a computer system having architecture 600, and in particular core 640, can provide functionality by having one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software embodied in one or more tangible computer-readable media. Such computer-readable media may include the user-accessible mass storage devices discussed above, as well as non-transitory storage devices of core 640, such as core internal mass memory 647 or ROM 645. Software for implementing embodiments of the present disclosure may be stored in such devices and executed by core 640. Depending on particular needs, computer-readable media may include one or more storage devices or chips. The software may cause core 640, and in particular its processors (including CPUs, GPUs, FPGAs, etc.), to perform certain processes or portions of certain processes described herein, including defining data structures stored in RAM 646 and modifying such data structures based on the software-defined processes. Additionally or alternatively, a computer system may provide functionality by hardwired or otherwise implemented logic in circuitry (e.g., accelerator 644) that can operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) on which software is stored to be executed, circuitry embodying the logic to be executed, or both. The present disclosure includes any appropriate combination of hardware and software.

[0101] While this disclosure has previously described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. It should thus be understood that those skilled in the art will be able to devise many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. 1. A method performed by an encoder for video coding, comprising: identifying a reference line set associated with a coding unit; One of the following steps: applying a first intra-interpolation filter to reference samples included in a first reference line to generate a first set of predicted samples based on the first reference line in the reference line set, the first reference line being adjacent to the coding unit, the first reference line being associated with a first reference line index, and the first intra-interpolation filter being selectable from both an edge-preserving filter and an edge-smoothing filter; and applying a second intra-interpolation filter to reference samples included in a second reference line to generate a second set of predicted samples based on the second reference line in the reference line set, the second reference line not adjacent to the coding unit, the second reference line being associated with a second reference line index, and the second intra-interpolation filter being selectable only from edge-preserving filters and not edge-smoothing filters; and performing one of: encoding the coding unit using the generated first or second set of prediction samples; signaling a reference line index, the reference line index being the first or second reference line index; Including, the edge-preserving filter applied to the reference samples included in the first reference line is a first edge-preserving filter having a first set of filter coefficients, and the edge-preserving filter applied to the reference samples included in the second reference line is a second edge-preserving filter having a second set of filter coefficients different from the first set of filter coefficients. method.

2. The method of claim 1 , wherein the edge smoothing filter has only positive or zero filter coefficients.

3. the edge smoothing filter includes a linear interpolation filter having only positive or zero filter coefficients; The method of claim 1 , wherein the edge-preserving filter comprises at least one of a linear interpolation filter, a nonlinear filter, or a bilateral filter having at least one or two negative filter coefficients.

4. The method of claim 1 , wherein the first reference line has a first reference line index having a value of 0.

5. The method of claim 2 , wherein the edge-preserving filter comprises at least one of a cubic filter, a discrete cosine transform (DCT)-based filter, a polynomial-based interpolation filter, a bilateral filter, and a Hermite interpolation filter.

6. The method of claim 1 , wherein the second reference line has a second reference line index with a non-zero value.

7. The method of claim 1 , wherein a first number of taps of the edge-smoothing filter is different from a second number of taps of the edge-preserving filter.

8. The method of claim 1 , wherein the edge smoothing filter comprises a bilinear filter or a Gaussian filter.

9. The method of claim 1 , further comprising, after the step of signaling the reference line index, signaling an intra-prediction mode, wherein the signaled intra-prediction mode excludes a planar mode.

10. 1. A method performed by an encoder for video coding, comprising: identifying a reference line set associated with a coding unit; One of the following steps: applying a first intra-interpolation filter to reference samples included in a first reference line to generate a first set of predicted samples based on the first reference line in the reference line set, the first reference line being adjacent to the coding unit, the first reference line being associated with a first reference line index, and the first intra-interpolation filter being selectable from both an edge-preserving filter and an edge-smoothing filter; and applying a second intra-interpolation filter to reference samples included in a second reference line to generate a second set of predicted samples based on the second reference line in the reference line set, the second reference line not adjacent to the coding unit, the second reference line being associated with a second reference line index, and the second intra-interpolation filter being selectable only from edge-preserving filters and not edge-smoothing filters; and performing one of: encoding the coding unit using the generated first or second set of prediction samples; transmitting or storing an encoded video bitstream including the first or second reference line index and the encoded coding unit; Including, the edge-preserving filter applied to the reference samples included in the first reference line is a first edge-preserving filter having a first set of filter coefficients, and the edge-preserving filter applied to the reference samples included in the second reference line is a second edge-preserving filter having a second set of filter coefficients different from the first set of filter coefficients. method.

11. 1. A decoder-implemented method for video coding, comprising: identifying a reference line set associated with a coding unit; decoding, from the coded video bitstream, a reference line index associated with the coding unit, the reference line index being a first or second reference line index; One of the following steps: applying a first intra-interpolation filter to reference samples included in a first reference line to generate a first set of predicted samples based on a first reference line in the reference line set associated with a first reference line index, the first reference line being adjacent to the coding unit, and the first intra-interpolation filter being selectable from both an edge-preserving filter and an edge-smoothing filter; and applying a second intra-interpolation filter to reference samples included in a second reference line to generate a second set of predicted samples based on a second reference line in the reference line set associated with a second reference line index, the second reference line not being adjacent to the coding unit, and the second intra-interpolation filter being selectable only from edge-preserving filters and not edge-smoothing filters; and performing one of: Including, the edge-preserving filter applied to the reference samples included in the first reference line is a first edge-preserving filter having a first set of filter coefficients, and the edge-preserving filter applied to the reference samples included in the second reference line is a second edge-preserving filter having a second set of filter coefficients different from the first set of filter coefficients. method.

12. 1. An apparatus for video encoding, comprising: at least one memory configured to store program code; at least one processor coupled to the at least one memory; The program code, when executed by the at least one processor, causes the at least one processor to perform the method of any one of claims 1 to 10. Device.

13. 1. An apparatus for video decoding, comprising: at least one memory configured to store program code; at least one processor coupled to the at least one memory; The program code, when executed by the at least one processor, causes the at least one processor to perform the method of claim 11. Device.

14. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 10.

15. A computer program for causing a computer to carry out the method according to claim 11.

Images