Harmonious design of offset-based refinement and multiple reference line selection
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TENCENT AMERICA LLC
- Filing Date
- 2024-08-14
- Publication Date
- 2026-07-31
Smart Images
Figure 0007898482000005 
Figure 0007898482000006 
Figure 0007898482000007
Abstract
Description
Technical Field
[0001]
[0001] Related Applications This application claims priority based on U.S. Provisional Application No. 63 / 217,061 filed on June 30, 2021, and U.S. Non - Provisional Application No. 17 / 573,823 filed on January 12, 2022, which are hereby incorporated by reference in their entirety.
[0002]
[0002] Technical Field This disclosure relates to video coding and / or decoding techniques, and more particularly, to offset - based refinement and improved design and signaling of a multiple - reference - line selection scheme.
Background Art
[0003]
[0003] Background of the Disclosure The description of the background made herein is for the purpose of generally presenting the context of the disclosure. The work done under the name of the current inventors is not admitted as prior art to this disclosure, either expressly or impliedly, to the extent that the work is described in a manner that might not otherwise qualify as prior art at the time of filing this application, solely in this background section.
[0004]
[0004] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can contain a series of pictures, each picture having spatial dimensions of, for example, 1920x1080 luminance samples and associated full or subsampled chrominance samples. The series of pictures can have a fixed or variable picture rate (alternatively called frame rate), for example, 60 pictures / second, i.e., frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video with 8 bits / pixel / color channel, 1920x1080 pixel resolution, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.
[0005]
[0005] One of the purposes of video coding and decoding is to reduce the redundancy of input images and / or video signals through compression. Compression can, in some cases, help reduce the aforementioned bandwidth and / or storage space requirements by more than two orders of magnitude. Both lossless and non-lossless compression, as well as combinations thereof, can be used. Lossless compression is a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal by the decoding process. Non-lossless compression is a coding / decoding process in which the original video information is not fully preserved during coding and is not fully recoverable during decoding. When using non-lossless compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal is useful for its intended use despite some information loss. In the case of video, non-lossless compression is widely used in many applications. The amount of distortion that is acceptable depends on the application. For example, users of certain consumer video streaming applications may be able to tolerate higher distortion than users of film or television broadcast applications. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerance distortion generally allows for coding algorithms that result in higher losses and higher compression ratios.
[0006]
[0006] Video encoders and decoders can utilize techniques from a wide range of categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.
[0007]
[0007] Video codec technology may include a technique known as intra coding. In intra coding, sample values are represented without referencing samples or other data from a previously reconstructed reference picture. In some video codecs, the picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the picture may be referred to as an intra picture. Intra pictures and their derivatives (e.g., independent decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in a coded video bitstream and video session, or as a still image. Samples of a block after intra prediction may be subject to conversion to the frequency domain, and the conversion coefficients thus produced can be quantized before entropy coding. Intra prediction represents a technique to minimize the sample values in the domain before conversion. In some cases, the smaller the converted DC value and the smaller the AC coefficient, the fewer bits are required for a given quantization step size to represent a block after entropy coding.
[0008]
[0008] Traditional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode blocks based on, for example, surrounding sample data and / or metadata obtained during encoding / decoding that are spatially adjacent and precede the block of data being coded or decoded in the decoding order. Such techniques are hereafter referred to as “intra-prediction” techniques. Note that in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from a reference picture.
[0009]
[0009] There may be many different forms of intra-prediction. If more than one such technique can be used in a given video coding technique, the technique used can be coded in intra-prediction mode. One or more intra-prediction modes can be provided in a particular codec. In some cases, a mode can have submodes and / or be associated with various parameters, and the mode / submode information and intra-coding parameters for a block of video can be coded individually or collectively included in a mode codeword. The codeword used for a given combination of mode, submode, and / or parameters can affect the coding efficiency gain by intra-prediction, and therefore can similarly affect the entropy coding technique used to convert the codeword into a bitstream.
[0010]
[0010] A certain mode of intra prediction was introduced in H.264, improved in H.265, and further improved with new coding techniques such as Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Generally, in intra prediction, a prediction block can be formed using available neighboring sample values. For example, available values from a particular set of neighboring samples along a particular direction and / or line can be copied into a predictor block.
[0011]
[0011] Referring to Figure 1A, the lower right shows a subset of nine predictor directions specified among the 33 possible intra-predictor directions in H.265 (corresponding to 33 of the 35 intra-modes specified in H.265, i.e., 33 angular modes). The point where the arrows converge (101) represents the predicted sample. The arrows indicate directions, and in order to predict the sample at 101, samples from the vicinity in that direction are used. For example, arrow (102) indicates that sample (101) is predicted from one or more vicinity samples pointing to the upper right at a 45-degree angle from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more vicinity samples pointing to the lower left of sample (101) at a 22.5-degree angle from the horizontal.
[0012]
[0012] Referring further to Figure 1A, a 4x4 sample square block (104) is shown in the upper left (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block is 4x4 sample size, S44 is in the lower right. Furthermore, an exemplary reference sample following a similar numbering scheme is shown. The reference sample is labeled with R and its Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, a predicted sample adjacent to the block being reconstructed is used.
[0013]
[0013] Intra-picture prediction in block 104 may begin by copying a reference sample value from a nearby sample according to the signaled prediction direction. For example, suppose the coded video bitstream includes signaling with respect to this block 104 that indicates the prediction direction of arrow (102) - i.e., the sample is predicted from one or more prediction samples that are angled 45 degrees to the upper right from the horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05, and sample S44 is predicted from reference sample R08.
[0014]
[0014] In some cases, the values of multiple reference samples may be combined, for example, by interpolation, in order to calculate the reference sample; especially if the direction is not evenly divisible by 45 degrees.
[0015]
[0015] The number of possible directions is increasing due to the continued development of video coding technology. For example, in H.264 (2003), nine different directions were available for intra-prediction. This increased to 33 in H.265 (2013), and as of the present disclosure, JEM / VVC / BMS can support up to 65 directions. Experimental studies have been conducted to help identify the most appropriate intra-prediction direction, and using certain techniques in entropy coding, the most appropriate direction can be encoded with fewer bits while accepting a certain bit penalty for the direction. Furthermore, the direction itself can often be predicted from the nearby directions used in the intra-prediction of the nearby block being decoded.
[0016]
[0016] Figure 1B shows a schematic diagram (180) of 65 intra prediction directions according to JEM, illustrating the increasing number of prediction directions in various coding techniques that have evolved over time.
[0017]
[0017] The way bits representing intra-prediction directions are mapped to prediction directions in a coded video bitstream may differ from video coding technique to video coding technique; for example, it can range from simple direct mapping of prediction directions to complex adaptive schemes including intra-prediction modes, codewords, most probable modes, and similar techniques. However, in any case, there may be certain intra-prediction directions in video content that are statistically less likely than certain other directions. Since the goal of video compression is to reduce redundancy, these less likely directions may be represented with more bits than more likely directions in a well-designed video coding technique.
[0018]
[0018] Inter-picture prediction or intra-prediction can be based on motion compensation. Motion compensation allows sample data from a previously reconstructed picture or a portion thereof (reference picture) to be used to predict the newly reconstructed picture or a portion of a picture (e.g., a block) after being spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third dimension being an indication of the reference picture in use (the latter being analogous to the time dimension).
[0019]
[0019] In some video compression techniques, the current MV applicable to a particular area of sample data can be predicted from other MVs, for example, those relating to other areas of sample data spatially adjacent to the area being reconstructed and preceding the current MV in the order of decoding. This can significantly reduce the amount of data required to code an MV by relying on the reduction of redundancy in the interrelated MVs, thereby increasing compression efficiency. MV prediction can work effectively because, for example, when coding an input video signal derived from a camera (known as natural video), there is a statistical likelihood that an area larger than the area to which a single MV is applicable will move in a similar direction in the video sequence, and therefore, in some cases, can be predicted using similar motion vectors derived from the MVs of adjacent areas. This results in an actual MV for a given area that is similar to or identical to the MV predicted from the surrounding MVs. Such an MV can be represented with fewer bits than would be used when the MV is coded directly rather than predicted from nearby MVs after entropy coding. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself can be lossless due to rounding errors, for example, when calculating the predictor from several surrounding MVs.
[0020]
[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec.H.265, “High Efficiency Video Coding”, December 2016). Of the many MV prediction mechanisms specified in H.265, the one described here is a technique hereafter referred to as “spatial merge”.
[0021]
[0021] Specifically, referring to Figure 2, the current block (201) contains samples discovered by the encoder during the motion search process so that it is predictable from previous blocks of the same size that have been spatially shifted. Instead of directly coding its MV, the MV can be derived from the most recent reference picture (in the order of decoding) using the MV associated with any of the five surrounding samples (202 to 206, respectively) designated as A0, A1 and B0, B1, B2, from the metadata associated with one or more reference pictures. In H.265, the MV prediction can use predictors from the same reference pictures used by the adjacent blocks. [Overview of the project]
[0022]
[0022] This disclosure describes various embodiments of methods, apparatus, and computer-readable storage media for video encoding and / or decoding.
[0023]
[0023] In one aspect, an embodiment of the present disclosure provides an offset refinement method for intra-prediction and multi-reference-line intra-prediction in video decoding. The method includes the step of a device receiving a coded video bitstream relating to a block. The device includes a memory for storing instructions and a processor for communicating with the memory. The method includes the step of the device determining, based on mode information of the block, whether an offset refinement for intra-prediction is applied to the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; and, in response to the determination that an offset refinement for intra-prediction is applied to the block, the device performing the offset refinement to generate an intra-predictor for intra-prediction of the block.
[0024]
[0024] According to another aspect, an embodiment of the present disclosure provides an apparatus for video encoding and / or decoding. The apparatus includes a memory for storing instructions and a processor communicating with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to execute the above method related to video encoding and / or decoding.
[0025]
[0025] According to another aspect, an embodiment of the present disclosure provides a non - transient computer - readable storage medium for storing instructions, which, when executed by a computer for video encoding and / or decoding, cause the computer to execute a method related to video encoding and / or decoding.
[0026]
[0026] The above and other aspects and their implementations are described in more detail in the drawings, the specification, and the claims.
Brief Description of the Drawings
[0027]
[0027] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A]
[0028] FIG. 1A shows a schematic diagram of an exemplary subset of intra - prediction direction modes. [Figure 1B]
[0029] FIG. 1B shows an example of an exemplary intra - prediction direction. [Figure 2]
[0030] FIG. 2 shows a schematic diagram of a current block and its surrounding spatial merge candidates for motion vector prediction in one example. [Figure 3]
[0031] FIG. 3 shows a schematic diagram of a simplified block diagram of a communication system (300) according to an exemplary embodiment. [Figure 4]
[0032] FIG. 4 shows a schematic diagram of a simplified block diagram of a communication system (400) according to an exemplary embodiment. [Figure 5]
[0033] Figure 5 shows a schematic diagram of a simplified block diagram of a video decoder according to an exemplary embodiment. [Figure 6]
[0034] Figure 6 shows a schematic diagram of a simplified block diagram of a video encoder according to an exemplary embodiment. [Figure 7]
[0035] Figure 7 shows a block diagram of a video encoder according to another exemplary embodiment. [Figure 8]
[0036] Figure 8 shows a block diagram of a video decoder according to another exemplary embodiment. [Figure 9]
[0037] Figure 9 shows a directional intra-prediction mode according to an exemplary embodiment of the present disclosure. [Figure 10]
[0038] Figure 10 shows a non-directional intra-prediction mode according to an exemplary embodiment of the present disclosure. [Figure 11]
[0039] Figure 11 shows a recursive intra-prediction mode according to an exemplary embodiment of the present disclosure. [Figure 12]
[0040] Figure 12 shows various reference line-based intra-prediction schemes according to exemplary embodiments of the present disclosure. [Figure 13]
[0041] Figure 13 shows an offset-based refinement for intra-prediction according to an exemplary embodiment of the present disclosure. [Figure 14A]
[0042] Figure 14A shows another example of offset-based refinement for intra-prediction according to an exemplary embodiment of the present disclosure. [Figure 14B]
[0043] Figure 14B shows another example of offset-based refinement for intra-prediction according to an exemplary embodiment of the present disclosure. [Figure 15]
[0044] Figure 15 shows a flowchart of the method according to an exemplary embodiment of the present disclosure. [Figure 16]
[0045] Figure 16 shows a schematic diagram of a computer system according to an embodiment of the present disclosure. [Modes for carrying out the invention]
[0028]
[0046] The present invention will be described in detail below with reference to the accompanying drawings, which constitute part of the invention and illustrate specific examples of embodiments. However, it should be noted that the invention can be carried out in a variety of different forms, and therefore the subject matter covered or claimed is intended to be construed as not being limited to any embodiment described below. It should also be noted that the invention can be carried out as a method, device, component, or system. Accordingly, embodiments of the invention can take the form of, for example, hardware, software, firmware, or any combination thereof.
[0029]
[0047] Throughout the specification and claims, terms may have nuances implied or suggested in context beyond their expressly stated meaning. The phrases “in one embodiment” or “in some embodiments” used herein do not necessarily refer to the same embodiment, and the phrases “in another embodiment” or “in other embodiments” used herein do not necessarily refer to different embodiments. Similarly, the phrases “in one implementation” or “in some implementations” used herein do not necessarily refer to the same implementation, and the phrases “in another implementation” or “in other implementations” used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include, in whole or in part, exemplary combinations of embodiments / implementations.
[0030]
[0048] In general, terms can be understood, at least partially, from their usage in context. For example, terms like "and," "or," or "and / or," when used in this context, can have various meanings that may depend at least partially on the context in which such terms are used. Typically, when "or" is used to relate a list such as A, B, or C, it is intended to mean "A, B, or C" in an exclusive sense, as well as "A, B, or C" in an inclusive sense. Furthermore, the terms "one or more" or "at least one" as used in this context may be used, at least partially context-dependent, to describe some feature, structure, or characteristic in a singular sense, or to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms like "a," "an," or "the" can also be understood, at least partially context-dependent, to convey either a singular or plural usage. Furthermore, the terms “based on” or “determined by” are not necessarily intended to convey an exclusive set of factors, but rather, at least partially context-dependent, may allow for the presence of additional factors that are not necessarily explicitly stated.
[0031]
[0049] Figure 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices capable of communicating with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) are capable of one-way transmission of data. For example, terminal device (310) can encode video data (e.g., a stream of video pictures captured by terminal device (310)) for transmission to other terminal devices (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive coded video data from the network (350), decode the coded video data to restore the video picture, and display the video picture according to the restored video data. One-way data transmission may be implemented in media serving applications, etc.
[0032]
[0050] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) for bidirectional transmission of coded video data, which may occur, for example, during a video conferencing application. With regard to bidirectional transmission of data, for example, each terminal device of terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other terminal device of terminal devices (330) and (340) via the network (350). Each terminal device of terminal devices (330) and (340) can also receive coded video data transmitted by the other terminal device of terminal devices (330) and (340), decode the coded video data to restore the video pictures, and display the video pictures on an accessible display device according to the restored video data.
[0033]
[0051] In the example in Figure 3, terminal devices (310), (320), (330), and (340) are implemented as a server, a personal computer, and a smartphone, but the applicability of the principles of this disclosure is not limited thereto. Embodiments of this disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and / or similar devices. Network (350) represents any number or type of network that carries coded video data between terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. Communication network (350) can exchange data over circuit-switched, packet-switched, and / or other types of channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this disclosure, the architecture and topology of the network (350) may not be important to the operation of this disclosure unless expressly described below.
[0034]
[0052] Figure 4 shows an example of the application of the disclosed subject matter, illustrating the arrangement of a video encoder and video decoder in a streaming environment. The disclosed subject matter can be equally applied to other video-enabled applications, including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, and the storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).
[0053] The streaming system may include a video source (401), a capture subsystem (413) which may include, for example, a digital camera, and which generates a stream of uncompressed video pictures or images (402). In one example, the stream of video pictures (402) includes samples recorded by the digital camera of the video source 401. The stream of video pictures (402), which is drawn as a thick line to emphasize a larger amount of data compared to encoded video data (404) (or encoded video bitstream), can be processed by an electronic device (420) which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof, and can operate or realize aspects of the disclosed subject matter as described in detail below. The encoded video data (404) (or encoded video bitstream (404)), which is drawn as a thin line to emphasize the smaller amount of data compared to the uncompressed video picture (402) stream, can be stored directly in a streaming server (405) or a downstream video device (not shown) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410) within, for example, an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an output stream (411) of a decompressed video picture that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described herein.In some streaming systems, encoded video data (404), (407), and (409) (e.g., video bitstream) can be encoded according to specific video coding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Multipurpose Video Coding (VVC). The information disclosed may be applicable to the context of VVC and other video coding standards.
[0035]
[0054] It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may include a video encoder (not shown).
[0036]
[0055] Figure 5 shows a block diagram of a video decoder (510) according to one embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of Figure 4.
[0037]
[0056] The receiver (531) is capable of receiving one or more coded video sequences that are to be decoded by the video decoder (510). In the same or different embodiments, it is possible to receive one coded video sequence at a time, provided that the decoding of each coded video sequence is independent of other coded video sequences. A coded video sequence may relate to multiple video frames or images. A coded video sequence can be received from a channel (501), which may be a hardware / software link to a storage device that stores coded video data or to a streaming source that transmits coded video data. The receiver (531) is capable of receiving coded video data together with other data, such as coded audio data and / or auxiliary data streams, which can be transmitted using their respective entities (not shown). The receiver (531) can isolate the coded video sequence from other data. To address network jitter, the buffer memory (515) may be located between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be separate from and outside the video decoder (510) (not shown). In yet another application, for example to address network jitter, the buffer memory (not shown) may exist outside the video decoder (510), and another additional buffer memory (515) may exist inside the video decoder (510) to handle playback timing, for example.If the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (515) may not be required or can be made smaller. For use in best-effort packet networks such as the Internet, a sufficiently sized buffer memory (515) may be required, and that size may be relatively large. Such a buffer memory may be implemented in an adaptive size and may also be implemented at least partially in an operating system or similar element (not shown) outside of the video decoder (510).
[0038]
[0057] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. These categories of symbols may include information used to manage the operation of the video decoder (510), and potentially information for controlling a rendering device, such as a rendering device (512) (e.g., a display screen), which may be an integrated part of an electronic device (530), as shown in Figure 5, but which can be coupled to the electronic device (530). The rendering device control information may be in the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). The entropy coding of the video sequence to be coded can follow video coding techniques or standards, and can follow various principles including variable-length coding, Huffman coding, and arithmetic coding with or without context influence. The parser (520) can extract from the coded video sequence a set of subgroup parameters for at least one subgroup of pixels in the video decoder, based on at least one parameter corresponding to a subgroup. Subgroups can include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), predictive units (PU), etc. The parser (520) can also extract information from the coded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, and motion vectors.
[0039]
[0058] The parser (520) can perform entropy decoding / analysis on the video sequence received from buffer memory (515) in order to generate symbols (521).
[0040]
[0059] The reconstruction of the symbol (521) may include multiple different processing or functional units, depending on the type of coded video picture or part thereof (inter and intra picture, inter and intra block) and other factors. How and which units are included can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following multiple processing or functional units is not depicted for clarity.
[0041]
[0060] Beyond the functional blocks already described, the video decoder (510) can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units can interact closely with each other and be at least partially integrated. However, for the purpose of illustrating the various functions of the disclosed subject matter, a conceptual subdivision into functional units is employed in the following disclosure.
[0042]
[0061] The first unit may be a scaler / inverse unit (551). The scaler / inverse unit (551) can receive not only quantized transformation coefficients but also control information (including which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrix, etc.) from the parser (520) as symbols (521). The scaler / inverse unit (551) can output a block containing sample values that can be input to the aggregator (555).
[0043]
[0062] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from previously reconstructed portions of the current picture. Such predictive information can be provided by the intra-picture predictive unit (552). In some cases, the intra-picture predictive unit (552) can generate blocks of the same size and shape as the block being reconstructed by using surrounding block information that has already been reconstructed and stored in the current picture buffer (558). The current picture buffer (558) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. The aggregator (555) may, in some cases, add the predictive information generated by the intra-predictive unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a sample-by-sample basis.
[0044]
[0063] Otherwise, the output samples of the scaler / inverse unit (551) may be associated with intercoded, motion-compensated blocks. In such cases, the motion-compensated prediction unit (553) can access the reference picture memory (557) to retrieve samples to be used for interpicture prediction. After motion-compensating the retrieved samples according to the symbols (521) associated with the blocks, these samples are added by the aggregator (555) to the output of the scaler / inverse unit (551) (the output of unit 551 may be called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples can be controlled by motion vectors available to the motion-compensated prediction unit (553), for example, in the form of symbols (521) which may have X, Y components (shift), and a reference picture component (time). Furthermore, motion compensation may involve interpolation of sample values, such as those retrieved from reference picture memory (557), when accurate motion vectors of sub-samples are used, and may also be related to motion vector prediction mechanisms, etc.
[0045]
[0064] The output samples of the aggregator (555) may be subject to various loop filtering techniques within the loop filter unit (556). The video compression technique may include an in-loop filtering technique controlled by parameters contained in the coded video sequence (also called the coded video bitstream), which are made available to the loop filter unit (556) as symbols (521) from the parser (520). The in-loop filtering technique may respond to metadata obtained during the decoding of earlier parts (in the order of decoding) of the coded picture or coded video sequence, as well as to previously reconstructed loop-filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as will be described in more detail below.
[0046]
[0065] The output of the loop filter unit (556) can be a sample stream that can be output to the rendering device (512) as well as stored in reference picture memory (557) for use in future inter-picture prediction.
[0047]
[0066] Once a given coded picture is fully reconfigured, it can be used as a reference picture for future inter-picture predictions. For example, once the coded picture corresponding to the current picture is fully reconfigured and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before the reconfiguration of subsequent coded pictures begins.
[0048]
[0067] The video decoder (510) is capable of performing decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Rec.H.265. A coded video sequence can conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and a profile documented in the video compression technique or standard. Specifically, a profile allows for the selection of a particular tool from among all tools available in the video compression technique or standard, with that tool being the only tool usable under that profile. Furthermore, to conform to the standard, the complexity of the coded video sequence may be within a range defined by the level of the video compression technique or standard. In some cases, this level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted by the Hyperbolic Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.
[0049]
[0068] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, a time, space, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction code, etc.
[0050]
[0069] Figure 6 shows a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of Figure 4.
[0051]
[0070] The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of Figure 6) that is capable of capturing video images to be coded by the video encoder (603). In another example, the video source (601) may be implemented as part of the electronic device (620).
[0052]
[0071] The video source (601) can provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream that can be of any appropriate bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any appropriate sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device capable of storing pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures or images that convey motion when viewed in sequence. The picture itself can be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following explanation focuses on samples.
[0053]
[0072] According to some exemplary embodiments, the video encoder (603) can encode and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required by the application. Enforcing an appropriate coding speed is one function of the controller (650). In some embodiments, the controller (650) can be functionally coupled to and control other functional units as described below. The coupling is not depicted for clarity. Parameters set by the controller (650) can include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, ...), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.
[0054]
[0073] In some exemplary embodiments, the video encoder (603) can be configured to operate in a coding loop. In an extremely simplified explanation, in one example, the coding loop may include a source coder (630) (for example, responsible for generating symbols such as a symbol stream based on the input picture and reference picture to be coded) and a (local) decoder (533) built into the video encoder (603). Even if the built-in decoder 633 processes the coded video stream by the source coder 630 without entropy coding, the decoder (633) reconstructs the symbols to generate sample data in a similar manner to that produced by a (remote) decoder (because any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since the decoding of the symbol stream yields bit-exact results that are independent of the decoder's location (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local encoder and the remote encoder. In other words, the predictive part of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" if it were using predictions during decoding. This fundamental principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained due to, for example, channel errors) is used to improve coding quality.
[0055]
[0074] It is possible to assume that the operation of the “local” decoder (633) is the same as that of a “remote” decoder, such as the video decoder (510) which has already been described in detail above in relation to Figure 5. However, as can be briefly seen in Figure 5, since symbols are available and the encoding / decoding of symbols to a coded video sequence by the entropy coder (645) and parser (520) is lossless, the entropy decoding unit of the video decoder (510), which includes the buffer memory (515) and parser (520), may not be fully realized in the local decoder (633) in the encoder.
[0056]
[0075] An insight that can be gained at this point is that any decoder technique other than analysis / entropy decoding, which can only exist within the decoder, may also need to exist in substantially the same functional form within the corresponding encoder. For this reason, the disclosed subject matter can often focus on the decoder portion of the encoder and the associated decoder operation. A description of encoder techniques can be omitted, as it is the inverse of a comprehensive description of decoder techniques. A more detailed description of encoders is provided below only in specific areas or aspects.
[0057]
[0076] During operation, in some exemplary implementations, the source coder (630) may perform motion-compensated predictive coding, predicting and coding the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this way, the coding engine (632) codes the difference (or residual) in the chroma channel between the pixel blocks of the input picture and the pixel blocks of the reference picture that can be selected as predictive references for the input picture.
[0058]
[0077] The local video decoder (633) can decode coded video data of a picture that can be designated as a reference picture based on symbols generated by the source coder (630). The operation of the coding engine (632) may, advantageously, be a non-lossless process. If the coded video data can be decoded by a video decoder (not shown in Figure 6), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) can repeat the decoding process that can be performed by the video decoder with respect to the reference picture, causing the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture having common content as the reconstructed reference picture obtained by the video decoder at the far end (assuming no transmission errors).
[0059]
[0078] The predictor (635) can perform predictive searches for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (534) for sample data (as candidate reference pixel blocks) or given metadata (reference picture motion vectors, block shapes, etc.), which may serve as appropriate predictive references for the new picture. The predictor (535) can operate on a sample block-pixel block basis to find appropriate predictive references. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635).
[0060]
[0079] The controller (650) can manage the coding operations of the source coder (630), including, for example, setting parameters and subgroup parameters used to encode video data.
[0061]
[0080] All outputs of the aforementioned functional units can be subjected to entropy coding in the entropy coder (645). The entropy coder (645) converts the symbols generated by the various functional units into coded video sequences by lossless compression of symbols, following techniques such as Huffman coding, variable-length coding, and arithmetic coding.
[0062]
[0081] The transmitter (640) can buffer coded video sequences, such as those created by the entropy coder (645), to prepare them for transmission over the communication channel (660), which may be a hardware / software link to a storage device that stores coded video data. The transmitter (640) can merge coded video data from the video encoder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (source not shown).
[0063]
[0082] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each coded picture, which may affect the coding techniques that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types:
[0064]
[0083] An intra-picture (I-picture) is one that can be encoded and decoded without using any other picture in the sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.
[0065]
[0084] A predictive picture (P-picture) can be encoded and decoded using intra-prediction or inter-prediction, which uses at most one motion vector and a reference index to predict the sample value of each block.
[0066]
[0085] A bidirectional prediction picture (B-picture) can be encoded and decoded using intra-prediction or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple prediction pictures can use more than two reference pictures and associated metadata to reconstruct a single block.
[0067]
[0086] A source picture can typically be spatially subdivided into multiple sample coding blocks (e.g., 4x4, 8x8, 4x8, or 16x16 sample blocks, respectively) and coded block by block. Blocks can be predictively coded by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded predictively by spatial or temporal prediction, referencing one previously coded reference picture. Blocks of picture B may be coded predictively by spatial or temporal prediction, referencing one or two previously coded reference pictures. A source picture or a picture undergoing intermediate processing may be subdivided into other types of blocks for other purposes. The subdivision of coding blocks and other types of blocks may or may not follow the same methods as those described in further detail below.
[0068]
[0087] The video encoder (603) can perform coding operations in accordance with a specified video coding technique or standard, such as ITU-T Rec.H.265. In this operation, the video encoder (603) can perform various compression operations, including predictive coding operations that leverage temporal and spatial redundancy in the input video sequence. The coded video data can conform to the syntax specified by the video coding technique or standard being used.
[0069]
[0088] In some exemplary embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, and other forms of redundant data (such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.).
[0070]
[0089] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes temporal or other correlations between pictures. For example, a particular picture under encoding / decoding, referred to as the current picture, may be partitioned into blocks. Similar to reference blocks in a previously coded and still buffered reference picture in video, blocks in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are used.
[0071]
[0090] In some exemplary embodiments, a dual-prediction technique can be used for inter-picture prediction. According to the dual-prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (although they may be past and future in display order, respectively). Blocks in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Blocks can be predicted by working together as a combination of the first and second reference blocks.
[0072]
[0091] Furthermore, merge mode technology can be used for inter-picture prediction to improve coding efficiency.
[0073]
[0092] According to some embodiments of this disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture can have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU can contain three parallel coding tree blocks (CTBs), which are one lumen CTB and two chroma CTBs. Each CTU can be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU and four 32x32 pixel CUs. Each of the one or more 32x32 blocks can be further divided into four 16x16 pixel CUs. In some exemplary embodiments, each CU can be analyzed during coding to determine the prediction type of the CU among various prediction types, such as inter-prediction type or intra-prediction type. A CU can be divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction process in coding (encoding / decoding) is performed in units of prediction blocks. The division of a CU into PUs (or PBs for different color channels) can be performed in various spatial patterns. For example, a luma or chroma PB can include a matrix of values (e.g., luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.
[0074]
[0093] Figure 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values in the current video picture in a sequence of video pictures, and to encode the processing block into a coded picture which is part of a coded video sequence. For example, the video encoder (703) can be used instead of the video encoder (403) in the example of Figure 4.
[0075]
[0094] For example, the video encoder (703) receives a matrix of sample values for a processing block, such as an 8x8 sample prediction block. The video encoder (703) then determines whether the processing block is best coded, for example using rate-distortion optimization (RDO), using intra-mode, inter-mode, or bi-predictive mode. If it is determined that the processing block should be coded in intra-mode, the video encoder (703) can use intra-predictive technique to encode the processing block into a coded picture; if it is determined that the processing block should be coded in inter-mode or bi-predictive mode, the video encoder (703) can use inter-predictive technique or bi-predictive technique, respectively, to encode the processing block into a coded picture. In some exemplary embodiments, merge mode may be used as a submode of inter-picture prediction, in which case the motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the target block. Therefore, the video encoder (703) may include components not explicitly shown in Figure 7, such as a mode determination module, to determine the predictive mode of the processing block.
[0076]
[0095] In the example shown in Figure 7, the video encoder (703) includes an interencoder (730), an intraencoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725), all coupled together as shown in the exemplary arrangement in Figure 7.
[0077]
[0096] The inter-encoder (730) is configured to receive a sample of the current block (e.g., a processing block), compare that block to one or more reference blocks in the reference picture (e.g., blocks in earlier and later pictures in display order), generate inter-prediction information (e.g., descriptions of redundant information by inter-coding techniques, motion vectors, merge mode information), and compute inter-prediction results (e.g., predicted blocks) based on the inter-prediction information using any appropriate technique. In some examples, the reference picture is a decoded reference picture decoded based on encoded video information using a decoding unit 633 (shown as a residual decoder 728 in Figure 7, which will be described in more detail below) incorporated into the exemplary encoder 620 in Figure 6.
[0078]
[0097] The intra encoder (722) is configured to receive a sample of the current block (e.g., a processing block), compare that block to a block already coded within the same picture, generate quantized coefficients after transformation, and optionally also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). Based on the intra prediction information and reference block within the same picture, the intra encoder (722) can compute an intra prediction result (e.g., a prediction block).
[0079]
[0098] The general-purpose controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on that general control data. For example, the general-purpose controller (721) determines the prediction mode of a block and provides control signals to the switch (726) based on that prediction mode. For example, if the prediction mode is intra-mode, the general-purpose controller (721) controls the switch (726) to select the intra-mode result for use by the residual calculator (723) and the entropy encoder (725) to select the intra-prediction information and include the intra-prediction information in the bitstream; and if the prediction mode of a block is inter-mode, the general-purpose controller (721) controls the switch (726) to select the inter-prediction result for use by the residual calculator (723) and the entropy encoder (725) to select the inter-prediction information and include the inter-prediction information in the bitstream.
[0080]
[0099] The residual calculator (723) can be configured to calculate the difference (residual data) between the received block and the prediction result for a block selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) can be configured to encode the residual data to generate conversion coefficients. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain and generate conversion coefficients. The conversion coefficients are then quantized to obtain quantized conversion coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse conversion and generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (722) and inter-encoder (730). For example, an inter-encoder (730) can generate a decoded block based on decoded residual data and inter-prediction information, and an intra-encoder (722) can generate a decoded block based on decoded residual data and intra-prediction information. The decoded block is appropriately processed to generate a decoded picture, which is buffered in a memory circuit (not shown) and can be used as a reference picture.
[0081]
[0100] The entropy encoder (725) can be configured to format a bitstream to include encoded blocks and to perform entropy coding. The entropy encoder (725) can be configured to include various types of information in the bitstream. For example, the entropy encoder (725) can be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. Residual information may not be present when coding blocks in either inter-mode or bi-prediction mode merge submodes.
[0082]
[0101] Figure 8 shows an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive a coded picture which is part of a coded video sequence, decode the coded picture, and produce a reconstructed picture. In one example, the video decoder (810) may be used instead of the video decoder (410) in the example of Figure 4.
[0083]
[0102] In the example shown in Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconfiguration module (874), and an intra-decoder (872), coupled together as shown in the exemplary arrangement in Figure 8.
[0084]
[0103] The entropy decoder (871) can be configured to reconstruct from the coded picture certain symbols representing the syntactic elements in which the coded picture is constructed. Such symbols may include, for example, the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-predictive mode, merge sub-mode, or another sub-mode), prediction information (e.g., intra-predictive information or inter-predictive information) that can identify specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), residual information (e.g., in the form of quantized transformation coefficients), etc. In one example, if the prediction mode is inter or bi-predictive mode, inter-predictive information is provided to the inter-decoder (880); if the prediction type is intra-predictive type, intra-predictive information is provided to the intra-decoder (872). The residual information can be inversely quantized and provided to the residual decoder (873).
[0085]
[0104] The inter-decoder (880) can be configured to receive inter-prediction information and generate inter-prediction results based on the inter-prediction information.
[0086]
[0105] The intra decoder (872) can be configured to receive intra prediction information and generate prediction results based on the intra prediction information.
[0087]
[0106] The residual decoder (873) can be configured to perform inverse quantization to extract de-quantization conversion coefficients and process these de-quantization conversion coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also use certain control information (including quantization parameters (QP)), which may be provided by the entropy decoder (871) (the data path is not drawn because this may only be a small amount of control information).
[0088]
[0107] The reconstruction module (874) can be configured to combine the residuals as output from the residual decoder (873) and the prediction results (which may be output by an inter or intra prediction module) in the spatial domain to form reconstructed blocks that form part of the reconstructed picture as part of the reconstructed video. It should be noted that other appropriate processes, such as deblocking, may be performed to improve visual quality.
[0089]
[0108] The video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using any suitable technology. In some exemplary embodiments, the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using one or more processors that execute software instructions.
[0090]
[0109] Returning to the intra-prediction process, which generates a prediction block, the samples within a block (e.g., a luminance or chroma prediction block, or, if not further divided into prediction blocks, a coded block) are predicted by adjacent, next adjacent, or other line(s) or multiple line(s), or combinations thereof. The residuals between the coded actual block and the prediction block are then processed by a transformation and followed by quantization. Various intra-prediction modes may be available, and parameters related to the selection of the intra-mode and other parameters can be signaled in the bitstream. Various intra-prediction modes may relate, for example, to the line(s) position(s) for predicting the samples, the direction along which the prediction samples are selected from the line(s), and other special intra-prediction modes.
[0091]
[0110] For example, a set of intra-prediction modes (referred to interchangeably as "intra-mode") may include a predetermined number of directional intra-prediction modes. As described above in relation to the exemplary implementation in Figure 1, these intra-prediction modes may correspond to a predetermined number of directions, along which out-of-block samples are selected as predictions for samples predicted within a particular block. In another particular exemplary implementation, eight principal directional modes corresponding to angles of 45 to 207 degrees with respect to the horizontal axis may be supported and predetermined.
[0092]
[0111] In some other implementations of intra prediction, the directional intra-mode can be further extended to angles set at a finer granularity to further leverage the greater diversity of spatial redundancy in directional textures. For example, the eight-angle implementation described above can be configured to provide eight nominal angles, designated V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, as shown in Figure 9, with a predetermined number (e.g., 7) finer angles added to each nominal angle. Such extensions would make a larger total number of directional angles (e.g., 56 in this example) available for intra prediction, corresponding to the same number of given directional intra-modes. The predicted angle may be expressed as the nominal intra-angle plus the angle delta. In the specific example above, which has seven finer angular directions for each nominal angle, the angular delta may be -3 to 3, multiplied by a step size of 3 degrees.
[0093]
[0112] The directional intra-prediction described above may also be called unidirectional intra-prediction, which is distinct from bidirectional intra-prediction (also called intra-bidirectional prediction) as described later in this disclosure.
[0094]
[0113] In some implementations, a predetermined number of non-directional intra-prediction modes may be available as alternatives or in addition to the directional intra-modes described above. For example, five non-directional intra-modes called smooth intra-prediction modes may be specified. These non-directional intra-prediction modes may specifically be called DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra-modes. The prediction of samples for a particular block under these exemplary non-directional modes is shown in Figure 10. As an example, Figure 10 shows a 4x4 block 1002 predicted by samples from the top-adjacent line and / or left-adjacent line. A particular sample 1010 in block 1002 may correspond to sample 1004 directly above sample 1010 within the upper-adjacent line of block 1002, sample 1006 to the upper left of sample 1010 as the intersection of the upper and left-adjacent lines, and sample 1008 directly to the left of sample 1010 within the left-adjacent line of block 1002. For example, in DC intra-prediction mode, the average of the left and upper adjacent samples 1008 and 1004 can be used as the predictor for sample 1010. For example, in PAETH intra-prediction mode, reference samples 1004, 1008, and 1006 are taken from the upper, left, and upper-left, and then the value of these three reference samples closest to (upper + left - upper-left) can be set as the predictor for sample 1010. For example, in SMOOTH_V intra prediction mode, sample 1010 can be predicted by quadratic interpolation in the vertical direction between the upper left neighbor sample 1006 and the left neighbor sample 1008. For example, in SMOOTH_H intra prediction mode, sample 1010 can be predicted by quadratic interpolation in the horizontal direction between the upper left neighbor sample 1006 and the upper neighbor sample 1004.For example, in the SMOOTH_H intra prediction mode, sample 1010 can be predicted by the average of quadratic interpolation in the vertical and horizontal directions. The above implementation of the non-directional intra mode is merely illustrative as a non-limiting example. Other adjacent lines, other non-directional selections of samples, and methods of combining prediction samples to predict specific samples within a prediction block are also conceivable.
[0095]
[0114] The encoder's selection of a specific intra-predictive mode from the above-mentioned directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) can be signaled in the bitstream. In some exemplary implementations, eight exemplary nominal directional modes, along with five non-angle smooth modes (a total of 13 options), can be signaled first. Then, if the signaled mode is one of the eight nominal angle intra-modes, an index is further signaled to specify the selected angle delta for the corresponding signaled nominal angle. In some other exemplary implementations, all intra-predictive modes may be indexed together for signaling (e.g., 56 directional modes plus 5 non-directional modes resulting in 61 intra-predictive modes).
[0096]
[0115] In some exemplary implementations, for example, 56 or other numbers of directional intra-prediction modes may be implemented using a unified directional predictor that projects each sample in a block to a reference sub-sample position and interpolates the reference sample by a two-tap bilinear filter.
[0097]
[0116] In some implementations, an additional filter mode called a FILTER INTRA mode may be designed to incorporate decaying spatial correlation using references on edges. In these modes, in addition to out-of-block samples, samples predicted within a block can be used as intra-predictive reference samples for any patch within the block. These modes can be, for example, predefined and made available for intra-prediction for at least Luma blocks (or Luma blocks only). A predefined number (e.g., five) of filter-intra modes may be pre-designed, each represented by a set of n-tap filters (e.g., 7-tap filters) that reflect the correlation between a 4x2 patch and its n adjacent neighboring samples. In other words, the weighting coefficients of the n-tap filters may be position-dependent. As shown in Figure 11, taking an 8x8 block, 4x2 patch, and 7-tap filtering as an example, the 8x8 block 1102 can be divided into eight 4x2 patches. These patches are shown in Figure 11 as B0, B1, B1, B3, B4, B5, B6, and B7. For each patch, the seven neighbors of the patch, shown as R0 to R7 in Figure 11, may be used to predict the samples within the current patch. In the case of patch B0, all neighbors may have already been reconstructed. However, for the other patches, some of the neighbors may be within the current block and therefore not reconstructed, and thus the predicted values of the immediate neighbors are used as a reference. For example, as shown in Figure 11, not all of the neighbors of patch B7 have been reconstructed, and therefore, predicted samples from the neighbors, e.g., parts of B4, B5, and / or B6, are used instead.
[0098]
[0117] In some implementations of intra-prediction, one color component can be predicted using one or more other color components. The color components can be any of the components in the YCrCb, RGB, XYZ color space, etc. For example, the prediction of a chroma component (e.g., a chroma block) from a luma component (e.g., a luma reference sample) is called Chroma from Luma (CfL) and may be implemented. In some exemplary implementations, cross-color prediction may only be permitted from luma to chroma. For example, a chroma sample in a chroma block can be modeled as a linear function of the luma samples being reconstructed simultaneously. CfL prediction may be implemented as follows:
[0099]
number
[0118] Here, L AC α represents the AC contribution of the luma component, α is a parameter of the linear model, and DC represents the DC contribution of the chroma component. The AC component is obtained, for example, for each sample in a block, while the DC component is obtained for the entire block. Specifically, the reconstructed luma samples can be subsampled to the chroma resolution, and then the average luma value (DC of luma) can be subtracted from each luma value to form the AC contribution in luma. The AC contribution of luma is then used in the linear mode of Eq. (1) to predict the AC value of the chroma component. Instead of requiring the decoder to calculate scaling parameters to approximate or predict the chroma AC component from the luma AC contribution, an implementation of CfL, for example, may determine the parameter α based on the original chroma samples and signal them in the bitstream. This reduces the complexity of the decoder and results in more accurate predictions. With respect to the DC contribution of the chroma component, this may be calculated using the intra-DC mode in the chroma component in some exemplary implementations.
[0100]
[0119] Returning to the explanation of intra-prediction, in some exemplary implementations, the prediction of a sample in a coding block or prediction block can be based on one of a set of reference lines. In other words, instead of always using the nearest neighborhood line (e.g., the neighborhood line immediately above or immediately to the left of the prediction block, as shown in Figure 1 above), multiple reference lines may be provided as selection options for intra-prediction. Such an implementation of intra-prediction may be called Multiple Reference Line Selection (MRLS). In these implementations, the encoder signals which of the multiple reference lines will be used to generate the intra-predictor. On the decoder side, after analyzing the reference line index, it is possible to generate the intra-prediction for the current intra-prediction block by identifying the reconstructed reference sample by searching for the specified reference line according to the intra-prediction mode (directional, non-directional, and other intra-prediction modes, etc.). In some implementations, reference line indices may be signaled at the coding block level, and only one of multiple reference lines may be selected and used for intra-prediction of a single coding block. In some examples, multiple reference lines may be selected together for intra-prediction. For example, more than one reference line may be combined, averaged, interpolated, or otherwise used to generate predictions with or without weights. In some implementations, MRLS may be applied only to the luminal component and not to the chroma component.
[0101]
[0120] Figure 12 shows an example of four reference lines (MRLS). As shown in the example in Figure 12, intra coding block 1202 can be predicted based on one of four horizontal reference lines 1204, 1206, 1208, and 1210, and four vertical reference lines 1212, 1214, 1216, and 1218. Of these reference lines, 1210 and 1218 are the nearest neighbor reference lines. The reference lines may be indexed according to their distance from the coding block. For example, reference lines 1210 and 1218 may be referred to as zero reference lines, and the other reference lines may be referred to as non-zero reference lines. Specifically, Reference lines 1208 and 1216 may also be referenced as the first reference line; Reference lines 1206 and 1214 may also be referenced as second reference lines; Reference lines 1204 and 1212 may be referenced as third reference lines.
[0102]
[0121] In some embodiments, offset-based refinement for intra prediction (ORIP) may be used after generating intra-prediction samples to improve video encoding / decoding performance. When ORIP is applied, the prediction samples are refined by adding an offset value.
[0103]
[0122] As shown in Figure 13, intra-prediction (1330) is performed based on reference samples. The reference samples may include samples from one or more left reference lines (1312) and / or one or more upper reference lines (1310). Offset-based refinement (ORIP) for intra-prediction can generate offset values using the vicinity reference samples. In some implementations, the vicinity reference samples for ORIP may be the same set as the reference samples for intra-prediction. In some other implementations, the vicinity reference samples for ORIP may be set differently from the reference samples for intra-prediction.
[0104]
[0123] In some implementations related to Figures 14A and 14B, ORIP may be executed at the 4x4 subblock level. For each 4x4 subblock (1471, 1472, 1473, and / or 1474), the offset is generated from the samples in its vicinity. For example, for the first subblock (1471), the offset is generated from the samples in its vicinity above (P1, P2, P3, and P4 in 1420), the samples in its vicinity to the left (P5, P6, P7, and P8 in 1410), and / or the samples in its upper left vicinity (P0)(1401). In some implementations, the samples in its vicinity above may include both the samples in its vicinity above (P1, P2, P3, and P4 in 1420) and the samples in its upper left vicinity (P0)(1401). In some other implementations, the samples in the left vicinity may include both the samples in the left vicinity (P5, P6, P7, and P8 in 1410) and the samples in the upper left vicinity (P0) (1401).
[0105]
[0124] The first sub-block (1471) contains 4x4 pixels, where each pixel of the 4x4 pixels corresponds to predN, which are the Nth neighboring predicted samples before refinement, e.g., pred0, pred1, pred2, ..., pred16.
[0106]
[0125] In various embodiments, the offset value of each pixel in a given subblock can be calculated based on nearby samples according to a formula. The formula may be a predefined formula or a formula specified by parameters coded in a coded bitstream.
[0107]
[0126] In some implementations related to Figure 14B, the offset value (offset(k)) of the k-th position of a given subblock may be generated as follows:
[0108]
number
[0127] W kn P is a predefined weight for offset calculation. n These are the values of nearby samples (e.g., P0, P1, P2, ..., P8). k This is the predicted value for the pixel after applying intra-prediction or other predictions (e.g., inter-prediction). k is the refined value for the pixel after applying ORIP. clip3() is the clip3 mathematical function. n is an integer from 0 to 8, including both ends. k is an integer from 0 to 15, including both ends.
[0109]
[0128] In some implementations, W kn This may be predefined, or it may be obtained according to Table 1.
[0110] Table 1: Predefined weights for offset calculation
[0111]
number
[0129] In some other implementations, subblock-based ORIP may apply only to a predefined set of intra-prediction modes and / or may differ for luma and chroma depending on the intra-prediction mode. Table 2 shows an example implementation of subblock-based ORIP with various intra-prediction modes and either luma or chroma channels. Using luma channels as an example: if the prediction mode is either DC or SMOOTH, ORIP is always ON and no additional signaling is required; if the prediction mode is HOR / VER and angle_delta is equal to 0, block-level signaling is required to enable / disable ORIP; and / or if the intra-prediction mode is any other mode, ORIP is always OFF and no additional signaling is required.
[0112] Table 2: Mode-dependent ON / OFF of proposed method
[0113]
number
[0130] Returning to the description of the second 4x4 subblock (1473), due to its relative position to the first 4x4 subblock (1471), samples near the top of the second subblock may be some pixels of the first subblock; P1 of the second subblock may be pred12 of the first subblock. P2 of the second subblock may be pred13 of the first subblock. P3 of the second sub-block may be pred14 of the first sub-block. P4 in the second subblock may be pred15 in the first subblock. A sample near the upper left of the second subblock (P0) may be a sample near the left of the first subblock (P8).
[0114]
[0131] There may be some issues / problems related to certain ORIP implementations. For example, if the intra-prediction mode for the block's luma component is either vertical or horizontal, there may be three different options: 1) Select an adjacent reference line and apply ORIP to the block; 2) Select adjacent reference lines and do not apply ORIP to the block; and / or 3) Select a non-adjacent reference line and do not apply ORIP to the block. Since reference samples from adjacent and non-adjacent reference lines are usually similar, there may be some redundancy between option 2) and option 3, and this redundancy may degrade the performance of video encoding / decoding.
[0115]
[0132] This disclosure describes various embodiments relating to offset refinement for intra-prediction and multiple reference line intra-prediction in video coding and / or decoding, which address at least one of the challenges / problems described above.
[0116]
[0133] In various embodiments, with reference to Figure 15, an offset refinement method 1500 for intra-prediction and multiple reference line intra-prediction in video decoding is shown, which may include all or part of the following steps: step S1510, a device comprising a memory for storing instructions and a processor communicating with the memory receives a coded video bitstream relating to a block; step S1520, the device determines, based on mode information of the block, whether offset refinement for intra-prediction is applied to the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; step S1530, in response to the decision that offset refinement for intra-prediction is applied to the block, the device performs offset refinement to generate an intra-predictor for intra-prediction of the block. In some implementations, step 1520 may include a step in which the device determines, based on the mode information of the block, whether offset refinement for intra-prediction is applied to the block, the mode information of the block includes at least one of the block's reference line index, the block's intra-prediction mode, or the block's size.
[0117]
[0134] In various embodiments of this disclosure, the size of a block (e.g., a coding block, a prediction block, or a transformation block) may refer to the width or height of the block. The width or height of the block may be an integer in pixels.
[0118]
[0135] In various embodiments of this disclosure, the size of a block (e.g., a coding block, a prediction block, or a transformation block) may refer to the area size of the block. The area size of the block may be an integer calculated in pixels by multiplying the width of the block by the height of the block.
[0119]
[0136] In various embodiments of this disclosure, the size of a block (e.g., a coding block, a prediction block, or a transformation block) may refer to the maximum width or height of the block, the minimum width or height of the block, or the aspect ratio of the block. The aspect ratio of the block may be calculated by dividing the width of the block by the height, or by dividing the height of the block by the width.
[0120]
[0137] In this disclosure, a reference line index specifies a reference line among several reference lines. In various embodiments, a reference line index of 0 for a block can indicate an adjacent reference line to the block, which is also the reference line closest to the block. For example, referring to block (1202) in Figure 12, the upper reference line (1210) is the upper adjacent reference line to block (1202), which is the upper reference line closest to the block; the left reference line (1218) is the left adjacent reference line to block (1202), which is also the left reference line closest to the block. A reference line index greater than 0 for a block indicates a non-adjacent reference line to the block, which is the non-nearest reference line to the block. For example, referring to block (1202) in Figure 12, A reference line index of 1 can indicate the reference line above (1208) and / or the reference line to the left (1216); A reference line index of 2 can indicate the reference line above (1206) and / or the reference line to the left (1214); and / or A reference line index of 3 can indicate the reference line above (1204) and / or the reference line to the left (1212).
[0121]
[0138] In various embodiments for video coding and / or decoding, when a directional intra-prediction mode is selected for a block (e.g., a coding block or a coded block) to generate an intra-predictor from a sample in a non-adjacent reference line, it is possible to determine whether offset refinement for intra-prediction (ORIP) is applied to the block. This decision may depend on the block's mode information, which may include, but is not limited to, the reference line index, one or more intra-prediction angles, and / or the block size(s). When this decision is based on the block's mode information, no additional signaling is required to indicate whether ORIP is applied, reducing overhead in video coding / decoding and / or improving video coding / decoding performance. In various embodiments, offset refinement for intra-prediction (ORIP) may refer to offset-based refinement for intra-prediction, which is a type of intra-prediction in which the prediction result is further refined (or modified) according to offset values generated based on one or more neighboring reference samples.
[0122]
[0139] Referring to step 1510, the device may be the electronic device (530) in Figure 5 or the video decoder (810) in Figure 8. In some implementations, the device may be the decoder (633) within the encoder (620) in Figure 6. In other implementations, the device may be part of the electronic device (530) in Figure 5, part of the video decoder (810) in Figure 8, or part of the decoder (633) within the encoder (620) in Figure 6. The coded video bitstream may be the coded video sequence in Figure 8, or the intermediate coded data in Figure 6 or Figure 7. A block may refer to a coding block or a coded block.
[0123]
[0140] Referring to step 1520, the device can determine whether offset refinement for intra-prediction is applied to a block based on the block's mode information, which may include at least one of the block's reference line index, the block's intra-prediction mode, and the block's size.
[0124]
[0141] In various embodiments, step 1520 may include determining that offset refinement for intra-prediction is applied to a block depending on whether the block's reference line index is less than a predetermined threshold and the block's intra-prediction mode belongs to a directional intra-prediction mode; and / or determining that offset refinement for intra-prediction is not applied to a block depending on whether the block's reference line index is not less than a predetermined threshold or the block's intra-prediction mode belongs to a directional intra-prediction mode.
[0125]
[0142] In some implementations, ORIP is always applied to a set of directional intra-prediction modes if the reference line index value is less than a predetermined value; and / or ORIP is never applied if the reference line index value is greater than or equal to the predetermined value. In one example, this set of directional intra-prediction modes includes a vertical mode and a horizontal mode. In another example, this set of directional intra-prediction modes includes vertical, horizontal, 45-degree, and 135-degree directional modes. In yet another example, the predetermined value is N and is set to 1. In yet another example, N is set to 2. In yet another example, N is set to 0 or 3.
[0126]
[0143] In various embodiments, step 1520 may include the step of determining that offset refinement for intra-prediction is applied to a block depending on whether the block's reference line index is even and the block's intra-prediction mode belongs to the directional intra-prediction mode; and / or determining that offset refinement for intra-prediction is not applied to a block depending on whether the block's reference line index is odd and the block's intra-prediction mode belongs to the directional intra-prediction mode.
[0127]
[0144] In some implementations, the decision of whether ORIP applies to a set of directional intra-prediction modes depends on whether the reference line index value is even or odd. In one example, ORIP is applied to a set of directional intra-prediction modes if the reference line index value is even; ORIP is not applied if the reference line index value is odd. In another example, ORIP is applied to a set of directional intra-prediction modes if the reference line index value is odd; ORIP is not applied if the reference line index value is even. In yet another example, this set of directional intra-prediction modes includes vertical and horizontal modes. In yet another example, this set of directional intra-prediction modes includes vertical, horizontal, 45-degree, and 135-degree directional modes.
[0128]
[0145] In various embodiments, step 1520 may include the steps of: determining that an offset refinement for intra-prediction is applied to a block depending on whether the block's intra-prediction mode belongs to a first selected set of intra-prediction modes; determining that an offset refinement for intra-prediction is applied to a block depending on whether the block's intra-prediction mode belongs to a second selected set of intra-prediction modes and the reference line index indicates an adjacent reference line; and / or determining that an offset refinement for intra-prediction is not applied to a block depending on whether the block's intra-prediction mode belongs to a second selected set of intra-prediction modes and the reference line index indicates a non-adjacent reference line. In some implementations, the first selected set of intra-prediction modes does not overlap with the second selected set of intra-prediction modes. In some other implementations, the first selected set of intra-prediction modes may include a diagonal intra-prediction mode.
[0129]
[0146] In some implementations, ORIP applies to both adjacent and non-adjacent reference lines for a selected set of intra-prediction modes, but to adjacent reference lines only for the remaining sets of intra-prediction modes. For example, if intra-prediction is applied by using only integer samples, e.g., in the diagonal (45 / 225 degree) intra-prediction mode, ORIP may also apply to non-adjacent reference lines; and / or, if intra-prediction does not use integer samples, ORIP applies only to adjacent reference lines.
[0130]
[0147] In various embodiments, step 1520 may include determining that an offset reference for intra-prediction is applied to a block, depending on whether the block's intra-prediction mode belongs to a directional intra-prediction mode and the block's reference line index indicates a non-adjacent reference line.
[0131]
[0148] In some implementations, step 1520 may further include a step in which, during entropy decoding, the mode information of the block is used as context for parameters related to offset refinement for intra-prediction.
[0132]
[0149] In some other implementations, if a directional intra-prediction mode is selected for a block and intra-predictors are generated from samples in non-adjacent reference lines, ORIP may still be applicable, and / or ORIP may be signaled by a parameter in the coded bitstream (e.g., an ORIP flag or ORIP index), regardless of the value of the block's mode information.
[0133]
[0150] In some other implementations, the context used for entropy coding of ORIP parameters (e.g., flags / indexes) indicating whether and how ORIP is applied during video encoding depends on the mode information of the current block. In some implementations, the context for entropy coding of syntax values indicating whether and how other modes are applied to the current block depends on ORIP flags / indexes indicating whether and how ORIP is applied to the block. Specific examples of syntax for other modes of the current block include, but are not limited to, the reference line index, one or more intra-prediction angles, and / or the block size.
[0134]
[0151] Referring to step 1530, depending on the decision to apply the offset refinement for intra prediction to the block, the device performs the offset refinement for intra prediction on the block. In some implementations, the block is a 4x4 block or contains multiple 4x4 blocks, and the offset refinement for intra prediction is performed on the block according to a predetermined method, for example, according to the method described above using Eq.(2) and Eq.(3).
[0135]
[0152] In various embodiments, step 1530 may include: determining a predictor from a non-adjacent reference line specified by the block's reference line index; determining a predicted value for the block according to an intra-prediction based on the predictor; and / or modifying the predicted value according to an offset refinement for the intra-prediction based on a sample from an adjacent reference line. In some implementations, the non-adjacent and adjacent reference lines are from two sides; the two sides include the upper and left sides.
[0136]
[0153] In some implementations, when ORIP is applied, a predictor (e.g., predictor A) is generated from samples in non-adjacent reference lines selected by MRLS in order to generate an intra predictor from samples in non-adjacent reference lines. ORIP then further modifies the predicted value from predictor A by using samples from adjacent reference lines.
[0137]
[0154] In some other implementations, the reference samples used to generate predictor A and to further modify the predictor's value from (or based on) predictor A are from different sides of the neighboring reference samples. Specific examples of the neighboring reference sample sides include, but are not limited to, the upper and left reference samples. In one example, one or more upper reference samples are used to generate predictor A, and one or more left reference samples are used to modify the predictor's value from (or based on) predictor A. In another example, one or more left reference samples are used to generate predictor A, and one or more upper reference samples are used to modify the predictor's value from (or based on) predictor A.
[0138]
[0155] In various embodiments, step 1530 may include: determining a predictor from non-adjacent reference lines specified by the block's reference line index; determining a predicted value for the block according to an intra-prediction based on the predictor; and / or modifying the predicted value according to an offset refinement for the intra-prediction based on multiple samples from multiple reference lines.
[0139]
[0156] In some implementations, step 1530 may further include a step of acquiring a sample based on multiple samples and corresponding weights using a linear weighted average, where the corresponding weights are predetermined according to the relative positions of the samples.
[0140]
[0157] In some other implementations, when ORIP is applied, the predicted samples are refined by ORIP using samples from multiple (more than one) reference lines in order to generate an intra predictor from samples in non-adjacent reference lines. In one example, ORIP is generated using a linearly weighted sum of samples from multiple reference lines. The weights are predetermined according to the relative positions of the samples.
[0141]
[0158] Embodiments of the present disclosure may be used separately or in any combination of any order. Furthermore, each method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium. Embodiments of the present disclosure may be applied to a lumina block or a chroma block; in a chroma block, the embodiments may be applied separately to two or more color components or to two or more color components together.
[0142]
[0159] In this disclosure, any step or process in the various embodiments may be combined in any quantity or in any order as desired. In this disclosure, two or more steps or processes in the various embodiments may be performed in parallel.
[0143]
[0160] The technologies described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 16 shows a computer system (2600) suitable for realizing a particular embodiment of the disclosed subject matter.
[0144]
[0161] Computer software can be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code that includes instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or instructions that are executed via interpretation, microcode execution, etc.
[0145]
[0162] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.
[0146]
[0163] The components shown in Figure 16 with respect to the computer system (2600) are essentially illustrative and are not intended to imply any limitations on the scope or functionality of the computer software that enables embodiments of this disclosure. Furthermore, the configuration of the components should not be construed as having any dependency or requirement on any one or combination of components shown in the exemplary embodiments of the computer system (2600).
[0147]
[0164] The computer system (2600) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, applause), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., conversations, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), or video (e.g., 2D video, 3D video including stereoscopic pictures).
[0148]
[0165] Input human interface devices may include one or more of the following (though only one of each is depicted): keyboard (2601), mouse (2602), trackpad (2603), touchscreen (2610), data glove (not shown), joystick (2605), microphone (2606), scanner (2607), and camera (2608).
[0149]
[0166] The computer system (2600) may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via touch screens (2610), data gloves (not shown), joysticks (2605), although there may be haptic feedback devices that do not serve as input), auditory output devices (e.g., speakers (2609), headphones (not shown)), visual output devices (e.g., screens (2610) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input functionality, each with or without haptic feedback functionality, some of which may be capable of outputting two-dimensional visual output, three-dimensional or more output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).
[0150]
[0167] The computer system (2600) may also include human-accessible storage devices and associated media, such as optical media including CD / DVD ROM / RW (2620) using media such as CD / DVD (2621), thumb drives (2622), removable hard drives or solid-state drives (2623), legacy magnetic media such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).
[0151]
[0168] Those skilled in the art will also understand that the term “computer-readable medium” as used in relation to the subject matter disclosed herein does not include a transmission medium, carrier wave, or other transient signal.
[0152]
[0169] A computer system (2600) may also include a network interface (2654) to one or more communication networks (2655). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, velodrome and industrial, real-time, latency-tolerant, etc. Examples of networks include Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), wired or wireless wide-area digital networks for television (including cable TV, satellite TV, and terrestrial TV), and velodrome and industrial networks including CANBus. Certain networks generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (2649) (e.g., a USB port on a computer system (2600)); others are generally integrated into the core of the computer system (2600) by being attached to a system bus, as described below (e.g., an Ethernet interface is integrated into a PC computer system, and a cellular network interface is integrated into a smartphone computer system). Using any of these networks, the computer system (2600) can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcast television), one-way transmit-only (e.g., CANbus to a specific CANbus device), or two-way, for example, to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks can be used for each of these networks and network interfaces, as described above.
[0153]
[0170] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be mounted on the core (2640) of the computer system (2600).
[0154]
[0171] The core (2640) may include one or more central processing units (CPUs) (2641), graphics processing units (GPUs) (2642), special programmable processing devices in the form of field-programmable gate arrays (FPGAs) (2643), hardware accelerators for specific tasks (2644), graphics adapters (2650), etc. These devices, along with read-only memory (ROM) (2645), random-access memory (2646), and internal mass storage devices (e.g., internal non-user-accessible hard drives, SSDs, etc.) (2647), may be connected via a system bus (2648). In some computer systems, the system bus (2648) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (2648) or via a peripheral bus (2649). For example, a screen (2610) can be connected to a graphics adapter (2650). The peripheral bus architecture includes PCI, USB, etc.
[0155]
[0172] The CPU (2641), GPU (2642), FPGA (2643), and accelerator (2644) can be combined to execute specific instructions that constitute the aforementioned computer code. The computer code can be stored in ROM (2645) or RAM (2646). Temporary data can be stored in RAM (2646), while persistent data can be stored, for example, in internal mass storage (2647). High-speed storage and retrieval of any memory device may be possible by utilizing cache memory, which can be closely associated with one or more CPUs (2641), GPUs (2642), mass storage (2647), ROMs (2645), RAM (2646), etc.
[0156]
[0173] Computer-readable media can have computer code therein for performing various computer implementation operations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the art in the field of computer software.
[0157]
[0174] As a non-limiting example, a computer system having architecture (2600), specifically a core (2640), can provide the ability to execute software embodied in one or more tangible computer-readable media as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media can be media related to user-accessible mass storage as described above, as well as specific storage of the core (2640) of a non-transient nature, such as mass storage (2647) or ROM (2645) within the core. Software that implements various embodiments of the present disclosure can be stored in such devices and executed by the core (2640). The computer-readable media can include one or more memory devices or chips, depending on the specific needs. The software can cause the core (2640), specifically the processor (including CPU, GPU, FPGA, etc.) therein, to execute specific processes or specific parts of specific processes as described herein, including defining data structures stored in RAM (2646) and modifying such data structures according to processes defined by the software. Furthermore, or alternatively, a computer system may provide functionality as a result of logic wired or otherwise embodied within a circuit (e.g., an accelerator (2644)), such circuit may perform, in place of or in conjunction with, a particular process or a particular part of a particular process described herein. References to software may include logic, and vice versa, as appropriate. References to computer-readable media may include circuits (such as integrated circuits (ICs)) that store software for execution, circuits that embody logic for execution, or both, where appropriate. This disclosure encompasses any appropriate combination of hardware and software.
[0158]
[0175] While certain inventions are described in relation to exemplary embodiments, the descriptions are not intended to be restrictive. Various modifications of the exemplary embodiments and additional embodiments of the invention will be apparent to those skilled in the art from this description. Those skilled in the art will readily recognize that these and various other modifications may be made to the exemplary embodiments illustrated and described herein without departing from the spirit and scope of the invention. Accordingly, it is assumed that the appended claims will cover any such modifications and alternative embodiments. Certain proportions in the drawings may be exaggerated, while other proportions may be minimized. Accordingly, this disclosure and drawings should be construed as illustrative, not restrictive.
Claims
1. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The decision step includes: The step of determining that ORIP is applied to the block if the reference line index of the block is less than a predetermined threshold; The step of determining that ORIP is not applied to the block if the reference line index of the block is not smaller than the predetermined threshold; Methods that include...
2. The method according to claim 1, wherein the directional intra-prediction mode includes a vertical mode and a horizontal mode.
3. The method according to claim 1, wherein the directional intra-prediction mode includes a vertical mode, a horizontal mode, a 45-degree directional mode, and a 135-degree directional mode.
4. The method according to claim 1, wherein the predetermined threshold includes one of 1 or 2.
5. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The decision step includes: The steps include determining that ORIP is applied to the block depending on whether the reference line index of the block is even and the intra-prediction mode of the block belongs to the directional intra-prediction mode; The steps include determining that ORIP is not applied to the block, depending on whether the reference line index of the block is odd and the intra-prediction mode of the block belongs to the directional intra-prediction mode; Methods that include...
6. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The decision step includes: The steps include determining that ORIP is applied to the block depending on whether the intra-prediction mode of the block belongs to a first selected set of intra-prediction modes; The step of determining that the ORIP is applied to the block in accordance with the fact that the intra-prediction mode of the block belongs to a second selected set of intra-prediction modes and the reference line index indicates an adjacent reference line; The step of determining that ORIP is not applicable to the block, depending on that the intra-prediction mode of the block belongs to a second selected set of intra-prediction modes and the reference line index indicates a non-adjacent reference line; A method that includes such that the first selected set of intra-prediction modes does not overlap with the second selected set of intra-prediction modes.
7. The method according to claim 6, wherein the first selected set of intra-prediction modes includes diagonal intra-prediction modes.
8. The method according to claim 1, the step of performing the offset refinement to generate an intra predictor for intra prediction of the block is: The steps include determining a predictor from non-adjacent reference lines specified by the reference line index of the block; The steps include: determining the predicted value of the block according to the intra-prediction based on the predictor; The steps include: modifying the predicted value according to the offset refinement for intra-prediction based on a sample from an adjacent reference line; Methods that include...
9. In the method according to claim 8: The non-adjacent and adjacent reference lines are from two sides; and The two sides include the upper and left sides, in this method.
10. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The step of running ORIP to generate intra predictors for intra prediction of the block includes: The steps include determining a predictor from non-adjacent reference lines specified by the reference line index of the block; The steps include: determining the predicted value of the block according to the intra-prediction based on the predictor; The steps include: modifying the predicted value according to the offset refinement for intra-prediction based on multiple samples from multiple reference lines; Methods that include...
11. The method according to claim 10, further: A method comprising the step of obtaining a sample based on a plurality of samples and corresponding weights using a linear weighted average, wherein the corresponding weights are predetermined according to the relative positions of the samples.
12. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The decision step includes: A method comprising the step of determining that the ORIP is applied to the block, in that the intra-prediction mode of the block belongs to a directional intra-prediction mode and the reference line index of the block indicates a non-adjacent reference line.
13. The method according to claim 12, further: A method for entropy coding that includes the step of using the mode information of the block as a context for the parameters relating to the ORIP.
14. Memory for storing instructions; A processor that communicates with the aforementioned memory; A device comprising the above, wherein the instruction is configured to cause the processor to execute the method according to any one of claims 1 to 13.
15. A computer program that causes a processor to perform the procedure described in any one of claims 1 to 13.
16. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: transmitting a bitstream containing the encoded blocks in the encoding step; The decision step includes: The step of determining that ORIP is applied to the block if the reference line index of the block is less than a predetermined threshold; The step of determining that ORIP is not applied to the block if the reference line index of the block is not smaller than the predetermined threshold; Methods that include...
17. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: storing the bitstream containing the blocks encoded in the above encoding step into a storage medium; The decision step includes: The step of determining that ORIP is applied to the block if the reference line index of the block is less than a predetermined threshold; The step of determining that ORIP is not applied to the block if the reference line index of the block is not smaller than the predetermined threshold; Methods that include...
18. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: transmitting a bitstream containing the encoded blocks in the encoding step; The decision step includes: The steps include determining that ORIP is applied to the block depending on whether the reference line index of the block is even and the intra-prediction mode of the block belongs to the directional intra-prediction mode; The steps include determining that ORIP is not applied to the block, depending on whether the reference line index of the block is odd and the intra-prediction mode of the block belongs to the directional intra-prediction mode; Methods that include...
19. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: storing the bitstream containing the blocks encoded in the above encoding step into a storage medium; The decision step includes: The steps include determining that ORIP is applied to the block in accordance with the fact that the reference line index of the block is even and the intra-prediction mode of the block belongs to the directional intra-prediction mode; The steps include determining that ORIP is not applied to the block, depending on whether the reference line index of the block is odd and the intra-prediction mode of the block belongs to the directional intra-prediction mode; Methods that include...
20. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: transmitting a bitstream containing the encoded blocks in the encoding step; The decision step includes: The steps include determining that ORIP is applied to the block depending on whether the intra-prediction mode of the block belongs to a first selected set of intra-prediction modes; The step of determining that the ORIP is applied to the block in accordance with the fact that the intra-prediction mode of the block belongs to a second selected set of intra-prediction modes and the reference line index indicates an adjacent reference line; The step of determining that ORIP is not applicable to the block, depending on that the intra-prediction mode of the block belongs to a second selected set of intra-prediction modes and the reference line index indicates a non-adjacent reference line; A method that includes such that the first selected set of intra-prediction modes does not overlap with the second selected set of intra-prediction modes.
21. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: storing the bitstream containing the blocks encoded in the above encoding step into a storage medium; The decision step includes: The steps include determining that ORIP is applied to the block depending on whether the intra-prediction mode of the block belongs to a first selected set of intra-prediction modes; The step of determining that the ORIP is applied to the block in accordance with the fact that the intra-prediction mode of the block belongs to a second selected set of intra-prediction modes and the reference line index indicates an adjacent reference line; The step of determining that ORIP is not applicable to the block, depending on that the intra-prediction mode of the block belongs to a second selected set of intra-prediction modes and the reference line index indicates a non-adjacent reference line; A method that includes such that the first selected set of intra-prediction modes does not overlap with the second selected set of intra-prediction modes.
22. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: transmitting a bitstream containing the encoded blocks in the encoding step; The step of running ORIP to generate intra predictors for intra prediction of the block includes: The steps include determining a predictor from non-adjacent reference lines specified by the reference line index of the block; The steps include: determining the predicted value of the block according to the intra-prediction based on the predictor; The steps include: modifying the predicted value according to the offset refinement for intra-prediction based on multiple samples from multiple reference lines; Methods that include...
23. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: storing the bitstream containing the blocks encoded in the above encoding step into a storage medium; The step of running ORIP to generate intra predictors for intra prediction of the block includes: The steps include determining a predictor from non-adjacent reference lines specified by the reference line index of the block; The steps include: determining the predicted value of the block according to the intra-prediction based on the predictor; The steps include: modifying the predicted value according to the offset refinement for intra-prediction based on multiple samples from multiple reference lines; Methods that include...
24. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: transmitting a bitstream containing the encoded blocks in the encoding step; The decision step includes: A method comprising the step of determining that the ORIP is applied to the block, in that the intra-prediction mode of the block belongs to a directional intra-prediction mode and the reference line index of the block indicates a non-adjacent reference line.
25. A coding method performed by an encoder that makes intra-predictions based on multiple reference lines: A determination step of determining whether offset refinement for intra-prediction (ORIP) is applied to a block to be coded, based on the mode information of the block, wherein the mode information of the block includes at least one of the reference line index of the block, the intra-prediction mode of the block, and the size of the block; Depending on the decision to apply the ORIP to the block, the steps include: running the ORIP on the block to generate an intra predictor for intra prediction of the block; The steps include: encoding the block using the intra predictor; The steps include: storing the bitstream containing the blocks encoded in the above encoding step into a storage medium; The decision step includes: A method comprising the step of determining that the ORIP is applied to the block, in that the intra-prediction mode of the block belongs to a directional intra-prediction mode and the reference line index of the block indicates a non-adjacent reference line.