Multi-reference intra prediction using variable weights - Patents.com

Abstract

A method for performing intra prediction for coding or decoding uses multiple layers of reference samples. The layers are formed into a reference array that is used by a function, such as a weighted combination, to form a final prediction. The weights may be variable and are selected from among several sets of weights. The prediction is used in coding or decoding a block of video data. The weights may be determined in several ways, and for a given prediction mode, the same weight or different weights may be used for all pixels in the target block. If the weights are different, they may depend on the distance of the target pixel from the reference array. An index may be transmitted indicating which weight set should be used. [Selection diagram] Figure 8

Description

[Technical field]

[0001] At least one of the embodiments relates generally to a method or apparatus for video coding or decoding, and more particularly to a method or apparatus for using multiple reference intra prediction with variable weights for a video encoder or video decoder. [Background technology]

[0002] To achieve high compression efficiency, image and video coding schemes typically exploit spatial and temporal redundancy in video content using prediction and transformation, including motion vector prediction. In general, intra- or inter-prediction is used to exploit correlation within or between frames, and then the difference between the original and predicted image, often called prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by an inverse process corresponding to entropy coding, quantization, transformation, and prediction. Summary of the Invention

[0003] The shortcomings and disadvantages of the prior art are addressed by the general aspects described herein directed to block shape adaptive intra prediction direction in coding and decoding.

[0004] According to a first aspect, there is provided a method, the method comprising: forming a plurality of reference arrays from reconstructed samples neighboring a block of video data, the reconstructed samples being located at pixel locations that are one or more locations from the block of video data, predicting a target pixel of the block of video data, respectively, by applying a weight set selected from a plurality of weight sets to one or more of the plurality of reference arrays, computing a final intra prediction of the target pixel of the block of video as a function of the respective predictions from one or more of the weighted reference arrays, and coding the block of video using the predictions.

[0005] According to another aspect, a second method is provided, the method including forming a plurality of reference arrays from reconstructed samples neighboring a block of video data, the reconstructed samples being located at pixel locations that are one or more locations from the block of video data, predicting a target pixel of the block of video data by applying a weight set selected from a plurality of weight sets to one or more of the plurality of reference arrays, respectively, computing a final intra prediction of the target pixel of the block of video as a function of the respective predictions from one or more of the weighted reference arrays, and decoding the block of video using the predictions.

[0006] According to another aspect, an apparatus is provided, the apparatus including a processor, the processor may be configured to code a block of a video or decode a bitstream by performing any of the methods described above.

[0007] According to another general aspect of at least one embodiment, a device is provided that includes an apparatus according to any of the decoding embodiments and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a band of frequencies including the video block; or (iii) a display configured to display an output representative of the video block.

[0008] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated in accordance with any of the described coding embodiments or variations.

[0009] According to another general aspect of at least one embodiment, there is provided a signal including video data generated according to any of the described coding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described coding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a computer program product is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0012] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0013] [Figure 1] 13 shows a reference sample for intra prediction in HEVC and a pixel value at coordinates indicated by P(x, y). [Diagram 2] Indicates the intra prediction direction in HEVC. [Diagram 3] 1 shows an example of multi-reference intra prediction. [Figure 4] Figure 4 shows intra prediction using two references. Figure 4(a) shows that the prediction direction is extended to obtain a predictor on a second reference array that is between the two reference samples of the target pixel. Figure 4(b) shows the second predictor that is directly on the reference sample. [Diagram 5] Denotes standard and common video compression methods. [Figure 6] This shows a standard common video decompression method. [Figure 7] 1 shows an exemplary flow diagram for decoding (analyzing) intra prediction information. [Figure 8] 1 shows an example flow diagram for coding intra-prediction information. [Figure 9] 1 shows a coding scheme with pre-coding and image segmentation sections. [Figure 10] 1 shows a decoding scheme with post-decoding processing. [Figure 11] 1 illustrates an exemplary processor configuration in which the described embodiments may be implemented. [Figure 12] 1 illustrates one embodiment of a coding method using multiple reference intra prediction with variable weights. [Figure 13] 1 illustrates an embodiment of another decoding method using multiple reference intra prediction with variable weights. [Figure 14] 1 illustrates an embodiment of an apparatus for coding or decoding using multiple reference intra prediction with variable weights. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The general aspects described herein concern the field of image and video compression, with a focus on spatial prediction. Conventional codecs divide frames into large non-overlapping square blocks, e.g., coding tree units (CTUs) in H.265 / HEVC (High Efficiency Video Coding, ITU-T H.265 ITU Telecommunications Standardization Sector, "Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services - Coding of moving video, High efficiency video coding", October 2014.), or superblocks in VP9 ("AV1 Bitstream & Decoding Process Specification", The Alliance for Open Media). These are referred to as CTUs in the following. These blocks are then divided into smaller blocks, i.e., coding units (CUs), according to a recursive tree. In HEVC and AV1, the maximum size of a block is set to 64x64. This size is likely to increase to at least 128x128 in the next standard, H.266 / VVC (Versatile Video Coding), which is being researched by the Joint Video Experts Team (JVET).

[0015] To code these blocks, intra prediction is used to provide an estimated version of the block using previously reconstructed neighboring samples. The difference between the source block and the prediction is then coded. In the conventional codecs mentioned above, a single line of reference samples to the left and above the current block is used.

[0016] Recent work has introduced the use of multiple reference lines: a weighted combination of samples from multiple lines along a given prediction direction is computed and embedded into the current block.

[0017] In the aspects described herein, this method is extended to allow the encoder to select a different set of weights to be applied to the line under consideration. The selected set is then transmitted to the decoder on a block-by-block basis.

[0018] One problem solved by the described aspects is how to efficiently predict a block using its neighboring blocks that have been previously reconstructed.

[0019] First, we will describe the intra prediction performed in the latest standard H.265 / HEVC. The intra prediction process in HEVC consists of three steps, namely (1) reference sample generation, (2) intra-sample prediction, and (3) post-processing of predicted samples. Figure 1 illustrates the reference sample generation process. For a prediction unit (PU) of size N × N, a row of 2N decoded samples on the top is formed from the previously reconstructed top and top-right pixels of the current PU. Similarly, a column of 2N samples on the left is formed from the reconstructed left and bottom-left pixels. The corner pixel at the top-left position is also used to fill the gap between the reference of the top row and the reference of the left column. If some of the samples on the top or left are unavailable, for example because the corresponding CU is not in the same slice or the current CU is at a frame boundary, a method called reference sample replacement is performed, in which the missing samples are copied from the available samples in the clockwise direction. Then, depending on the current CU size and prediction mode, the reference samples are filtered using a specified filter.

[0020] The next step, i.e., intra-sample prediction, consists of predicting the pixels of the target CU based on the reference samples. As mentioned before, to efficiently predict different types of content, HEVC supports different prediction modes. Planar and DC prediction modes are used to predict smooth and gradually changing regions, while angular prediction modes are used to capture different directional structures. HEVC supports 33 directional prediction modes, indexed from 2 to 34. These prediction modes correspond to different prediction directions, as shown in Figure 2.

[0021] Multi-reference intra prediction refers to intra prediction using reference pixels from multiple rows and columns (Figure 3). This is also called any-tier reference intra prediction or multi-line intra prediction. In image processing, it is common to use a first-order prediction model because of its high correlation. The reason for using one row and column as reference in intra prediction was originally based on this, and also on the fact that other decoded rows and columns are far from the pixels of the target block and do not add much additional information about the target block. This was not a problem when the number of prediction directions was small (e.g., ≦7) and block sizes were also small (e.g., ≦8) in previous standards. However, in HEVC, the size of the prediction unit has been increased to 64×64, with a corresponding increase in the number of angular prediction directions to 33. Block sizes up to 256×256 with a number of angular prediction modes up to 129 are being studied for future standards. Considering this, it makes sense to use decoded pixels from additional rows and columns for reference when the trade-off between the cost of additional memory and complexity and the coding gain is acceptable.

[0022] Consider the vertical angle prediction mode (e.g., Fig. 4(a)) when the predicted sample of the target pixel is between two reference samples on the reference array above. In this case, in HEVC, the predicted sample is linearly interpolated as mentioned in the previous section. The predicted sample is just an estimate. If we extend the prediction direction by one more row, the line will intersect with the second row at a different point. The second row of pixels is slightly farther than the first row, but due to the orientation of the object structure in the image, the estimate of the second row may be a better estimate than the first row. In some cases, it may even intersect with the second row at the location of the pixel that was just decoded (e.g., Fig. 4(b)). Based on this idea, it makes sense to predict the target block based on multiple references.

[0023] When referring to different reference row / column pairs, the term "reference layer" is used. Thus, reference layer 1 means the reference row and column next to the target block (i.e., the normal reference in HEVC), reference layer 2 means the reference row and column next to reference layer 1, and so on.

[0024] To perform prediction, we form top and left reference arrays corresponding to the reference layers as described in the previous section. These are called reference array 1, reference array 2, etc. Note that reference array n, n=1, 2, ... can mean either the top reference array or the left reference array depending on whether the prediction mode corresponds to the vertical or horizontal direction, respectively.

[0025] In previous studies, the reference samples for each layer follow the reference sample generation algorithm of HEVC or JEM. These samples are then used to form a reference array. For positive angle modes (modes 2-10 and 26-34 in HEVC), the nth reference array is simply the reference sample above the nth reference layer.

number

number

number

[0026] In this method, weights are selected and fixed for all blocks of the sequence, which is one problem addressed by aspects described herein.

[0027] The aspects described herein extend the multi-reference intra-prediction scheme proposed in previous work by having the encoder select the optimal configuration among different sets of weights applied to different lines of reference samples for intra-prediction. The selected weights, i.e., their index in the list of possible choices, are then transmitted to the decoder for each block.

[0028] As an example, using the shortened unary encoding, it is possible to transmit two bits to indicate one choice among three possible sets. Exemplary weight sets {{1,0}, {3,1}, {1,1}} are tested. For each of the three possible sets, the first number represents the weight applied to the nearest line of the reference sample, and the second number corresponds to the weight applied to the second line of the reference sample. However, the general aspect is not limited to the use of two lines of reference samples.

[0029] This exemplary configuration has been implemented and tested, and Table 1 shows the resulting bitrate reduction compared to JVET's test model VTM for the intra configuration on the first frame of each sequence in the JVET test set. The affected codec module is the intra prediction block step 160 in FIG. 9 or step 260 in FIG. 10. [Table 1]

[0030] Aspects described herein extend previous work such that an encoder can optimize prediction using a different set of weights for each block. In previous work, the weights are fixed. Activation of multi-reference intra prediction is done using a transmitted flag. However, aspects described herein describe several variations that differ in how information related to the selection of the weights to be applied to each block is coded.

[0031] In Fig. 7, an example corresponding to one of the main embodiments is described, in which the coding of weights depends on the intra mode. Using multiple lines is advantageous for directional intra modes over other modes, such as DC mode and planar mode. In that case, the first step 710 consists of decoding the prediction mode. If the mode can be adapted to multi-reference prediction (720), the index of the weight set is decoded (730). Otherwise, conventional intra prediction is performed.

[0032] A similar flow chart describes the process of encoding the required transmitted index and is shown in Figure 8. The weight set can be encoded with any binary encoding.

[0033] As an example, using the shortened unary encoding, it is possible to transmit two bits to indicate one choice among three possible sets. The exemplary weight sets {{1,0}, {3,1}, {1,1}} are tested and applied only to the luma block, with the results shown in Table 1. For each of the three possible sets, the first number represents the weight applied to the nearest line of reference samples, and the second number corresponds to the weight applied to the second line of reference samples. However, the described aspects are not limited to the use of two lines of reference samples. The final weights are normalized, i.e., divided by {1,4,2}, respectively. In this example, the first pair of weights corresponds to conventional intra prediction.

[0034] On the encoder side, the basic idea is to try all different sets of weights and select the solution that provides the best rate / distortion (RD) cost using traditional rate-distortion optimization (RDO). This requires, for each intra mode, to code each version of prediction with its weights and compare the costs, which results in an increase in the encoder runtime. We discuss various solutions to reduce the overhead in terms of complexity.

[0035] First, traditional encoders usually include two passes for intra prediction decisions: the first (fast) pass selects a subset of optimal modes among a large number of intra modes (33 in HEVC) by estimating their rate / distortion (RD) cost without performing a full coding of the block; then a full RD calculation is performed on the selected modes to select the optimal mode.

[0036] It is then possible to perform the weight selection in a first pass and retain the pairs {intra mode, weight index} to be processed in a second pass, thus reducing the coding run time while retaining most of the gain.

[0037] It is also possible to add another fast pass that selects the mode in a fast pass and evaluates the optimal weights only for the selected intra mode. Again, this estimation does not rely on the complete coding of the block, but on an estimate of the cost of the residual.

[0038] Regarding the syntax, several embodiments can be considered as explained below.

[0039] EMBODIMENT 1 Multi-reference intra prediction with variable weights is applied to only directional intra modes or to a set of directional modes. The decoding or coding of intra prediction information is performed according to the steps illustrated in Figure 7 or Figure 8, respectively.

[0040] EMBODIMENT 2 Multi-reference intra prediction with variable weights is applied to all intra modes. In this case, there is no need to decode the intra modes, and the index of the weight set is analyzed for each block. The decoding and coding of the intra prediction information is performed according to the steps described in Figure 7 or Figure 8, respectively.

[0041] EMBODIMENT 3 Embodiment 1 or 2, in which multi-reference prediction with variable weights is applied to the luma component only, thus reducing the coding of additional bits for the chroma component. The prediction process of the chroma block can be either a conventional prediction using a single line of reference samples, or a multi-reference sample with a fixed set of weights that do not change across the block.

[0042] EMBODIMENT 4 Embodiment 1 or 2, where the prediction of the chroma components follows the choice made for luma. This requires that the division of the chroma blocks follows the luma, or that a specific mapping has been set, e.g. using the center or top-left pixel of a chroma block to derive its position within the luma division.

[0043] EMBODIMENT 5 3. The method of claim 1, wherein the index is encoded using a unary contraction encoding. Using the example above of three weight sets {{1,0}, {3,1}, {1,1}}: The first bin indicates whether the first set {1,0} is selected (value 0). If the first bin is equal to 1, the second bin is analyzed and the second set {3,1} (value 0) or the third {1,1} (value 1) is selected.

[0044] EMBODIMENT 6 3. Embodiment 1 or 2, in which multiple reference intra prediction is performed only for large blocks, e.g., blocks having a height or width greater than a size such as 16 samples.

[0045] EMBODIMENT 7 Embodiment 1 or 2, where the weight set, the number of reference lines depends on the transmitted index, as well as the prediction direction, or the block size / shape (square / rectangle...), component ID. A predefined table (indexed by prediction direction and block shape) can be used to convert the index to an actual weight set.

[0046] EMBODIMENT 8 Weight set, the number of reference lines depends on the position of the predictor, embodiment 1 or 3. This embodiment uses weights that depend on each pixel. The embodiment can use subset 1 if the predictor comes from the top line and subset 2 if the predictor comes from the left column.

[0047] A wide variety of aspects are described in this document, including tools, features, embodiments, models, approaches, and the like. Many of these aspects are described specifically and often in a manner that may sound limiting, at least to indicate their individual characteristics. However, this is for clarity of description and does not limit the application or scope of these aspects. Indeed, all of the various aspects can be combined and interchanged to provide further aspects. Additionally, these aspects can also be combined and interchanged with aspects described in previous applications.

[0048] The aspects described and contemplated in this document can be implemented in many different forms. Figures 9, 10, and 11 below provide some embodiments, but other embodiments are contemplated, and discussion of Figures 9, 10, and 11 is not intended to limit the breadth of implementations. At least one of these aspects generally relates to video coding and decoding, and at least one other aspect generally relates to transmitting a generated or coded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for coding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0049] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, while "decoded" is used on the decoder side.

[0050] Various methods are described herein, each method including one or more steps or actions for achieving the described method. Insofar as a specific order of steps or actions is not required for the proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0051] Various methods and other aspects described in this document can be used to modify modules, such as intra prediction modules, entropy coding modules, and / or decoding modules (160, 360, 145, 330) of video encoder 100 and decoder 200, as shown in Figures 9 and 10. Furthermore, the aspects are not limited to VVC or HEVC, and can be applied to other standards and recommendations, whether existing or developed in the future, and extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this document can be used individually or in combination.

[0052] In this document, various numerical values ​​are used, for example, {{1,0}, {3,1}, {1,1}}, etc. These specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0053] 9 illustrates an encoder 100. Although variations of this encoder 100 are contemplated, the encoder 100 is described below for clarity without describing all possible variations.

[0054] Before being coded, a video sequence may undergo a pre-coding process (101), such as applying a color transformation to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and may be attached to the bitstream.

[0055] In the encoder 100, a picture is coded by the elements of the encoder as described below. The picture to be coded is divided (102) and processed, for example, in units of CUs. Each unit is coded, for example, using either intra mode or inter mode. When a unit is coded in intra mode, intra prediction is performed (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether intra mode or inter mode is used to code the unit, and indicates the intra / inter decision, for example, by a prediction mode flag. A prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0056] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, plus motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may also bypass both the transform and quantization, i.e., the residual is directly coded without applying a transform or quantization process.

[0057] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual is combined (155) with the predicted block to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture, for example to perform deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0058] Figure 10 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by elements of the decoder as described below. The video decoder 200 generally performs a decoding pass that is the inverse of the coding pass as described in Figure 9. The encoder 100 also generally performs video decoding as part of encoding the video data.

[0059] In particular, the input of the decoder includes a video bitstream, which may be generated by the video encoder 100. First, the bitstream is entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition the picture according to the decoded picture partition information (235). The transform coefficients are dequantized (240) and inverse transformed (250), and the prediction residual is decoded. The decoded prediction residual is combined with a predicted block (255) to reconstruct an image block. The predicted block may be obtained from intra prediction (260) or motion compensated prediction (i.e., inter prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0060] The decoded picture may further undergo post-decoding processing (285), such as an inverse color conversion (e.g., converting from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-coding process (101). The post-decoding process may use metadata derived in the pre-coding process and signaled in the bitstream.

[0061] FIG. 11 illustrates a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television sets, personal video recording systems, connected home appliances, and servers. The elements of system 1000, either alone or in combination, may be embodied in an integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other similar systems or to other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0062] The system 1000 includes at least one processor 1010 configured to execute loaded instructions, for example, in implementing various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits, as is known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 1040 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0063] The system 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, coded or decoded video, which may include its own processor and memory. The encoder / decoder module 1030 represents a module or modules that may be included in a device that performs coding and / or decoding functions. As is well known, a device may include one or both of a coding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated within the processor 1010 as a combination of hardware and software, as is well known to those skilled in the art.

[0064] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during execution of processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and operational logic.

[0065] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, an external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as MPEG-2, HEVC, or VVC (Versatile Video Coding).

[0066] Input to the elements of system 1000 may be provided through various input devices as shown in block 1130. Such input devices may include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air by, for example, a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0067] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements necessary to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulate the down-converted, band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, e.g., a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner that performs a variety of these functions, including down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and filtering again an RF signal transmitted over a wired (e.g., cable) medium to a desired frequency band. In various embodiments, the order of the above (and other) elements is rearranged, some of these elements are removed, and / or other elements that perform similar or different functions are added. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0068] Also, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1000 to other electronic devices over USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 1010, as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor 1010, and the encoder / decoder 1030, operating in combination with memory and storage elements to process the data stream, as desired, for example, for presentation on an output device.

[0069] The various elements of system 1000 may be provided within a unitary housing in which the various elements may be interconnected and transmit data therebetween using a suitable connection arrangement 1140, e.g., an internal bus as is well known in the art, including an I2C bus, hardwire, and printed circuit board.

[0070] The system 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0071] In various embodiments, data is streamed to the system 1000 using a wireless network such as IEEE 802.11. The wireless signals in these embodiments are received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication, for example. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to outside networks, including the Internet, to enable streaming applications and other over-the-air communication. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers data via an HDMI connection in the input block 1130. Still other embodiments provide streamed data to the system 1000 using an RF connection in the input block 1130.

[0072] The system 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The other peripheral devices 1120, in various example embodiments, include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 1000. In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV link, CEC, or other communication protocols that allow device-to-device control with or without user intervention. The output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to the system 1000 using a communication channel 1060 via the communication interface 1050. The display 1100 and speakers 1110 may be integrated with other components of the system 1000 in an electronic device, such as a television, in a single unit. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0073] Alternatively, the display 1100 and speakers 1110 may be separate from one or more of the other components, for example, if the RF section of the input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0074] The embodiments may be implemented by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment, and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories. The processor 1010 may be of any type suitable for the technical environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0075] Various implementations include decoding. As used herein, "decoding" may encompass all or part of the processes performed, for example, on a received coded sequence to generate a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes may also or alternatively include processes performed by the decoder of various implementations described herein, such as extracting indexes of weights used for various intra-prediction reference arrays.

[0076] As a further example, in one embodiment, "decoding" refers to only entropy decoding, in another embodiment, "decoding" refers to only differential decoding, and in another embodiment, "decoding" refers to a combination of entropy and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process in general will be clear based on the context of a particular description and will be well understood by one of ordinary skill in the art.

[0077] Various implementations include encoding. Similar to the above discussion of "decoding", "encoding" as used in this application may encompass all or part of the processes performed, for example, on an input video sequence to generate a coded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential coding, transforming, quantization, and entropy coding. In various embodiments, such processes also or alternatively include processes performed by the encoder of various implementations described in this application, such as weighting of intra-prediction reference arrays.

[0078] As a further example, in one embodiment, "encoding" refers to only entropy encoding, in another embodiment, "encoding" refers to only differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process generally will be clear based on the context of the particular description and will be well understood by one of ordinary skill in the art.

[0079] It should be noted that the syntax elements used herein are descriptive terms, so they do not preclude the use of other syntax element names.

[0080] It should be understood that when a diagram is presented as a flow diagram, it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flow diagram of the corresponding method / process.

[0081] Various embodiments refer to rate-distortion calculation or rate-distortion optimization. In particular, during the coding process, a balance or trade-off between rate and distortion is usually considered, often given a computational complexity constraint. Rate-distortion optimization is usually conceived as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, an approach may be based on an extensive test of all coding alternatives, including all modes or coding parameter values ​​considered, with a full evaluation of the coding cost and associated distortion of the reconstructed signal after encoding and decoding. In particular, a fast approach involving a calculation of approximate distortion based on a prediction or prediction residual signal, rather than on the reconstructed signal, may also be used to reduce coding complexity. A combination of these two approaches may also be used, such as by using approximate distortion for only some of the possible coding alternatives and full distortion for others. Other approaches evaluate only a subset of the possible coding alternatives. More generally, the optimization does not necessarily involve a full evaluation of both the coding cost and the associated distortion, although many approaches employ any of a variety of techniques to perform the optimization.

[0082] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. A method may be implemented, for example, in a processor, which refers generally to processing devices including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the transfer of information between end users.

[0083] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" in various places throughout this document, as well as any other variations, are not necessarily all referring to the same embodiment.

[0084] Additionally, this document may refer to "determining" various portions of information. Determining the information may include, for example, one or more of estimating the information, calculating the information, predicting the information, or retrieving the information from a memory.

[0085] Additionally, this document may refer to "accessing" various portions of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or inferring information.

[0086] Additionally, this document may refer to "receiving" various portions of information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Additionally, "receiving" typically involves in some manner, for example, an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0087] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as the number of items listed, as would be apparent to one of ordinary skill in the art and related arts.

[0088] Also, as used herein, the word "signal" refers to, among other things, instructing a corresponding decoder to do something. For example, in a particular embodiment, an encoder signals a particular one of a number of weights to be used for an intra-prediction reference array. In this way, in one embodiment, the same parameters are used at both the encoder and decoder sides. Thus, for example, the encoder sends a particular parameter to the decoder (explicit signaling), which allows the decoder to use the same particular parameter. Conversely, if the decoder already has a particular parameter and other parameters, signaling without transmission may be used (implicit signaling), simply allowing the decoder to know and select the particular parameter. By avoiding the transmission of any actual function, bit reduction is achieved in various embodiments. It should be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. Although the above is directed to the verb form of the word "signal", the word "signal" may also be used as a noun in this specification.

[0089] As will be apparent to one of ordinary skill in the art, implementations can generate a wide variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is well known. The signal can be stored on a processor-readable medium.

[0090] Embodiments may include one or more of the following features or entities, alone or in combination, across a variety of different claim categories and types. Modifying the intra prediction process applied to the decoder and / or encoder. · Enabling several weighted intra prediction methods in the decoder and / or encoder. Inserting syntax elements into the signaling that allow the decoder to identify the weights to be applied to the intra prediction references. · Selecting a weight or set of weights to apply at the decoder based on these syntax elements. Applying weighted intra prediction at the decoder. Matching residues with an encoder according to any of the considered embodiments. A bitstream or signal containing one or more of the described syntax elements or variants thereof. Inserting syntax elements into the signaling that allow the decoder to adapt the residual in a way that corresponds to the residual used by the encoder. · Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof. A television, set-top box, mobile phone, tablet, or other electronic device that performs intra-prediction reference weighting according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that performs intra-prediction reference weighting according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display). A television, set-top box, mobile phone, tablet, or other electronic device that tunes to a channel (e.g., using a tuner) to receive a signal including the coded image and performs intra-prediction reference weighting according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing a coded image over wireless communication (e.g., using an antenna) and performs intra-prediction reference weighting according to any of the described embodiments.

[0091] Various other generalized and specific inventions and claims are also supported and contemplated throughout this disclosure.

[0092] FIG. 12 illustrates one embodiment of a method 1200 for coding a block of video data using the general aspects described herein. The method begins at start block 1201 and control passes to function block 1210, which forms multiple reference arrays from reconstructed samples adjacent to the block of video data, the reconstructed samples being located at pixel locations that are one or more locations from the block of video data. Control then passes from block 1210 to block 1220, which predicts target pixels of the block of video data by applying a weight set selected from the multiple weight sets to one or more of the multiple reference arrays, respectively. Control then passes from block 1220 to block 1230, which calculates a final intra prediction of a target pixel of the block of video as a function of the respective prediction from one or more of the weighted reference arrays. Control then passes from block 1230 to block 1240, which codes the block of video using the predictions.

[0093] FIG. 13 illustrates one embodiment of a method 1300 for decoding a block of video data using the general aspects described herein. The method begins at start block 1301 and control proceeds to function block 1310, which forms multiple reference arrays from reconstructed samples adjacent to the block of video data, the reconstructed samples being located at pixel locations that are one or more locations from the block of video data. Control then proceeds from block 1310 to block 1320, which predicts a target pixel of the block of video data by applying a weight set selected from the multiple weight sets to one or more of the multiple reference arrays, respectively. Control proceeds from block 1320 to block 1330, which calculates a final intra prediction of a target pixel of the block of video as a function of the respective prediction from one or more of the weighted reference arrays. Control then proceeds from block 1330 to block 1340, which decodes the block of video using the prediction.

[0094] 14 illustrates one embodiment of an apparatus 1400 for encoding or decoding blocks of video data. The apparatus includes a processor 1410 and may be interconnected via at least one port to a memory 1420. Both the processor 1410 and the memory 1420 may also have one or more additional interconnections to external connections.

[0095] The processor 1410 is configured to code or decode the video data by forming a plurality of reference arrays from reconstructed samples of the block of video data, predicting target pixels of the block of video data, respectively, by applying a weight set selected from a plurality of weight sets to one or more of the plurality of reference arrays, calculating a final prediction of the target pixel of the block of video as a function of the respective predictions from one or more of the reference arrays, and encoding or decoding the block of video using the final prediction.

Claims

1. predicting a target pixel of a block of video data by applying weights to one or more of a plurality of reconstructed samples from one or more rows or columns adjacent to the block of video data; calculating at least one intra prediction of the target pixel of the block of video data as a function of a prediction from one or more of the weighted predictions; and coding the block of video data using the prediction; A method in which an index corresponding to a weight set is assigned to the weight set for applying the weights to a reference line set for all intra modes, a weight set is assigned per block for luma and color components use a fixed weight set for all blocks, and the reference lines used with the weight set include reference lines that are not directly adjacent to the block of video data.

2. An apparatus including a processor, the processor comprising: predicting a target pixel of a block of video data by applying weights to one or more of a plurality of reconstructed samples from one or more rows or columns adjacent to the block of video data; calculating at least one intra prediction of the target pixel of the block of video data as a function of a prediction from one or more of the weighted predictions; coding the block of video data using the prediction; and An apparatus in which an index corresponding to a weight set is assigned to a weight set for applying the weights to a reference line set for all intra modes, a weight set is assigned per block for luminance, and color components use a fixed weight set for all blocks, and the reference lines used with the weight set include reference lines that are not directly adjacent to the block of video data.

3. predicting a target pixel of a block of video data by applying weights to one or more of a plurality of decoded samples from one or more rows or columns adjacent to the block of video data; calculating at least one intra prediction of the target pixel of the block of video data as a function of a prediction from one or more of the weighted predictions; and decoding the block of video data using the prediction; A method in which an index corresponding to a weight set is assigned to the weight set for applying the weights to a reference line set for all intra modes, a weight set is assigned per block for luma and color components use a fixed weight set for all blocks, and the reference lines used with the weight set include reference lines that are not directly adjacent to the block of video data.

4. An apparatus including a processor, the processor comprising: predicting a target pixel of a block of video data by applying weights to one or more of a plurality of decoded samples from one or more rows or columns adjacent to the block of video data; calculating at least one intra prediction of the target pixel of the block of video data as a function of a prediction from one or more of the weighted predictions; and decoding the block of video data using the prediction; An apparatus in which an index corresponding to a weight set is assigned to a weight set for applying the weights to a reference line set for all intra modes, a weight set is assigned per block for luminance, and color components use a fixed weight set for all blocks, and the reference lines used with the weight set include reference lines that are not directly adjacent to the block of video data.

5. The method of claim 1 or 3, wherein the weights are applied to a directional intra mode.

6. The method of claim 1 or 3, wherein the weights are applied to all intra modes.

7. The method of claim 5 or 6, wherein the weights are applied to the luminance component only.

8. The method of claim 1 or 3, wherein three weight sets are used.

9. The method according to claim 5 or 6, wherein multiple reference intra prediction is performed for blocks whose height or width is greater than N samples.

10. The method of claim 5 or 7, wherein the weights and the number of reference lines are based on the position of the predictor.

11. An apparatus according to claim 4; (i) an antenna configured to receive a signal, the signal including the block of video data; (ii) a band limiter configured to limit the received signal to a band of frequencies that includes the block of video data; and (iii) a display configured to display an output representative of the blocks of video data; and at least one of:

12. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to perform the method according to any one of claims 1, 3 and 5 to 10.

Images