Data dependencies in encoding / decoding

Abstract

When the processing of a video encoder or decoder is parallelized, it processes portions of the video with less latency and avoids latency due to dependency on the completion of the previous processing. In one embodiment, a motion vector predictor from a neighboring video block is used in a subsequent subsequent video block prior to completion of refinement for use in the neighboring block. In another embodiment, information from neighboring blocks is constrained to include blocks in the same coding tree unit. In another embodiment, before the motion vector predictor is added to the list of candidates, it is checked whether the motion vector predictor has been in the list of candidates to accelerate the processing.

Description

[0001] This divisional application is a divisional application with an application date of April 26, 2019, application number 201980030855.X, and invention name “Data Dependence in Encoding / Decoding”. Technical Field

[0002] This aspect relates to video compression as well as video encoding and decoding. Background Art

[0003] In the HEVC (High Efficiency Video Coding, ISO / IEC 23008-2, ITU-T H.265) video compression standard, motion compensated temporal prediction is used to exploit the redundancy between consecutive pictures of a video.

[0004] To do this, a motion vector is associated with each prediction unit (PU). Each CTU is represented by a coding tree in the compressed domain. This is a quadtree partitioning of the CTU, where each leaf is called a coding unit (CU), such as Figure 1 shown.

[0005] Each CU is then given some intra or inter prediction parameters (prediction information). To this end, it is spatially partitioned into one or more prediction units (PUs), each of which is assigned some prediction information. Intra or inter coding modes are assigned at the CU level, such as Figure 2 shown.

[0006] A motion vector is assigned to each PU in HEVC. This motion vector is used for temporal prediction of motion compensation of the considered PU. Therefore, in HEVC, the motion model of the linked prediction block and its reference blocks includes translation.

[0007] In the Joint Exploration Model (JEM) developed by the JVET (Joint Video Exploration Team) group, certain motion models are supported to improve temporal prediction. To this end, a PU can be spatially divided into sub-PUs, and a dedicated motion vector can be assigned to each sub-PU using a model.

[0008] In other versions of JEM, CU is no longer divided into PU or TU (transform unit), and some motion data is directly allocated to each CU. In this new codec design, CU can be divided into sub-CUs, and motion vectors can be calculated for each sub-CU.

[0009] For inter-frame motion compensation, a collection of new tools using decoder-side parameter estimation has been developed in JEM, including, for example, FRUC merge, FRUC bilateral, and IC. Summary of the invention

[0010]

[0006] The shortcomings and disadvantages of the prior art may be addressed by one or more embodiments described herein, including embodiments for reducing data dependencies in encoding and decoding.

[0011] According to a first aspect, a method is provided. The method comprises the steps of obtaining information for a current video block from neighboring video blocks before the information is refined for use in neighboring video blocks; refining the information for use in the current video block; and encoding the current video block using the refined information.

[0012] According to another aspect, a second method is provided. The method includes the steps of obtaining information for a current video block from a neighboring video block before the information is refined for use in a reconstructed neighboring video block; refining the information for use by the current video block; and decoding the current video block using the refined information.

[0013] According to another aspect, a device is provided. The device includes a memory and a processor. The processor can be configured to encode a video block or decode a bit stream by executing any one of the aforementioned methods.

[0014] According to another general aspect of at least one embodiment, a device is provided, comprising an apparatus according to any decoding embodiment; and at least one of: (i) an antenna configured to receive a signal over the air, the signal comprising a video block; (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, and (iii) a display configured to display an output.

[0015] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided, containing data content generated according to any of the described encoding embodiments or variations.

[0016] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variations.

[0017] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0018] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0019] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Coding tree units and coding tree concepts representing compressed HEVC pictures are shown.

[0021] Figure 2 It shows that the coding tree unit is divided into coding units, prediction units and transform units.

[0022] Figure 3 An example of a bilateral matching cost function is shown.

[0023] Figure 4 An example of a template matching cost function is shown.

[0024] Figure 5 It is shown that the L-shape in reference 0 or 1 is compared with the L-shape of the current module to derive IC parameters.

[0025] Figure 6 An example of a processing pipeline with data flow dependencies is shown.

[0026] Figure 7 An example of a pipeline in which data dependencies arise in a motion compensation module is shown.

[0027] Figure 8 A general encoding embodiment to which the present embodiment can be applied is shown.

[0028] Fig. 9 A general decoding embodiment to which the present embodiment can be applied is shown.

[0029] Fig.10 An overview of the default FRUC processing for motion vector derivation is shown.

[0030] Fig.11 An overview of an embodiment of a modified FRUC process showing motion vector derivation.

[0031] Fig.12 is an example of a CU using the FRUC template mode.

[0032] Fig.13 An example of motion vector predictor derivation for merge candidates in JEM is shown.

[0033] Fig.14 Examples are shown from left to right under the exemplary embodiment: default check, alternative check, simplified check.

[0034] Fig.15 A block diagram of an exemplary communication channel is shown in which various aspects and exemplary embodiments are implemented.

[0035] Fig.16 One embodiment of a method for encoding according to the generally described aspects is shown.

[0036] Fig.17 One embodiment of a method for decoding according to the generally described aspects is shown.

[0037] Fig.18 One embodiment of an apparatus for encoding or decoding according to the generally described aspects is shown. DETAILED DESCRIPTION

[0038] The described embodiments are generally in the field of video compression. One or more embodiments are directed to improving compression efficiency compared to existing video compression systems.

[0039] In the HEVC (High Efficiency Video Coding, ISO / IEC 23008-2, ITU-T H.265) video compression standard, motion compensated temporal prediction is used to exploit the redundancy between consecutive pictures of a video.

[0040] To do this, a motion vector is associated with each prediction unit (PU). Each CTU is represented by a coding tree in the compressed domain. This is a quadtree partitioning of the CTU, where each leaf is called a coding unit (CU), such as Figure 1 shown.

[0041] Each CU is then given some intra or inter prediction parameters (prediction information). To this end, it is spatially partitioned into one or more prediction units (PUs), each of which is assigned some prediction information. Intra or inter coding modes are assigned at the CU level, such as Figure 2 shown.

[0042] In HEVC a motion vector is assigned to each PU. This motion vector is used for temporal prediction of motion compensation of the considered PU. Therefore, in HEVC the motion model of the linked prediction block and its reference blocks includes translation.

[0043] In the Joint Exploration Model (JEM) developed by the JVET (Joint Video Exploration Team) group, certain motion models are supported to improve temporal prediction. To this end, a PU can be spatially divided into sub-PUs, and a dedicated motion vector can be assigned to each sub-PU using a model.

[0044] In other versions of JEM, CU is no longer divided into PU or TU (transform unit), and some motion data is directly allocated to each CU. In this new codec design, CU can be divided into sub-CUs, and motion vectors can be calculated for each sub-CU.

[0045] For inter-frame motion compensation, a collection of new tools using decoder-side parameter estimation has been developed in JEM, including, for example, FRUC merge, FRUC bilateral, and IC.

[0046] The FRUC (Frame Rate Up Conversion) tool is described below.

[0047] FRUC allows the motion information of a CU to be derived at the decoder side without signaling.

[0048] This mode is signaled at the CU level using the FRUC flag and the additional FRUC mode flag to indicate which matching cost function (bilateral or template) to use to derive the motion information of the CU.

[0049] On the encoder side, the decision whether to use FRUC merge mode for a CU is based on RD (rate distortion) cost selection. Two matching modes (bilateral and template) are checked for a CU. The one that results in the minimum RD cost is further compared with other coding modes. If the FRUC mode is the most efficient in terms of RD, the FRUC flag is set to true for the CU and the relevant matching mode is used.

[0050] The motion derivation process in FRUC merge mode is divided into two steps. First, a CU-level motion search is performed, and then a sub-CU-level motion refinement is performed. At the CU level, an initial motion vector is derived from a list of MV (motion vector) candidates for the entire CU based on bilateral or template matching. The candidate that results in the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral or template matching around the starting point is performed, and the MV that results in the minimum matching cost is used as the MV for the entire CU. Subsequently, the motion information is further refined at the sub-CU level, starting from the derived CU motion vector.

[0051] like Figure 3 As shown, the bilateral matching cost function is used to derive the motion information of the current CU by finding the best match between two blocks in two different reference pictures along the motion trajectory of the current CU. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distance between the current picture and the two reference pictures (TD0 and TD1).

[0052] like Figure 4 As shown, the template matching cost function is used to derive the motion information of the current CU by finding the best match between the template in the current picture (the top and / or left neighboring blocks of the current CU) and the block in the reference picture (the same size as the template).

[0053] Note that this FRUC mode using the template matching cost function can also be applied to the AMVP (Advanced Motion Vector Prediction) mode in the embodiment. In this case, there are two candidates for AMVP. A new candidate is derived using the FRUC tool with template matching. If this FRUC candidate is different from the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list, and then the list size is set to two (meaning the second existing AMVP candidate is deleted). When applied to AMVP mode, only CU level search is applied.

[0054] Illumination Compensation (IC)

[0055] In inter-frame mode, IC allows to correct the block prediction samples obtained via motion compensation (MC) by taking into account spatial or temporal local illumination variations. IC parameters are estimated by comparing the set S of reconstructed neighboring samples (L-shape-cur) with the neighboring samples (L-shape-ref-i) of the reference i block (i=0 or 1), as Figure 5 Shown

[0056] The IC parameters minimize the difference between the samples in L-shape-cur and the samples of L-shape-ref-i corrected with the IC parameters (least squares method). Typically, the IC model is linear: IC(x) = a*x+b, where x is the value of the sample to be compensated.

[0057] The parameters a and b are derived by solving an L-shaped least squares minimization at the encoder (and decoder):

[0058]

[0059] Finally, a i Convert to integer weights (a i ) and shift (sh i ), and correct the MC block by IC:

[0060] Pred i = (a i * x i >> sh i ) + b i (3)

[0061] One problem addressed by at least one of the described embodiments is how to relax data dependencies generated by tools such as FRUC. Figure 6 An example showing the processing pipeline for decoding an inter frame:

[0062] - First, parse the bitstream and decode all symbols for a given unit (here we set the unit to CU)

[0063] -Then, the symbols are processed to calculate values ​​used to reconstruct the CU. Examples of such values ​​are motion vector values, residual coefficients, etc.

[0064] -When the value is ready, execute the processing. Figure 6 An example of a motion compensation and residual reconstruction pipeline is shown. Note that these modules can run in parallel and have very different runtimes than other modules (like parsing or decoding), and also have varying times that depend on the CU size.

[0065] - When all modules of a particular CU are run, the final result is calculated. Here, as an example, the final reconstruction consists in adding the motion compensated block and the residual block.

[0066] When considering such a pipeline, one issue raised by tools such as FRUC is that since the final motion vector of CU0 depends on the result of motion compensation and CU1 should wait for this value before starting to decode parameters, a dependency is introduced between the parameter decoding module and the compensation module.

[0067] Another problem is that, depending on the availability of sample data from each neighboring CU, certain data used to perform motion compensation (eg, for FRUC mode or IC parameter calculation) may not be available.

[0068] Figure 7 An example of a pipeline showing data dependencies arising in a motion compensation module.

[0069] At least one of the embodiments described herein uses an approach that avoids this dependency and allows for highly parallel pipelining at the decoder.

[0070] FRUC and IC are new modes in JEM, and therefore pipeline stalling is a relatively new problem.

[0071] A basic idea of ​​at least one of the proposed embodiments is to break the dependency between the decoding and motion compensation modules.

[0072] At least one of the proposed embodiments involves a modification of the codec specification: the encoding and decoding processes are completely symmetrical. The affected codec modules of one or more embodiments are Fig.10 Motion compensation 170 and motion estimation 175 and Fig.11 Estimation of motion 275.

[0073] Independent Motion Vector Predictor

[0074] In the default FRUC template processing, the motion vector for a particular block is refined using samples from the top and left templates of neighboring blocks. After refinement, the final value of the motion vector is known and can be used to decode the motion vectors for subsequent blocks in the frame (see Fig.10 ). However, since motion compensation and refinement can take a long time (especially waiting for data from other blocks to be ready), decoding of the current parameters is stalled, or the motion compensation pipeline waits for the slowest block to continue.

[0075] Instead of using the final motion vector (after the FRUC process is complete) as the predictor for the neighboring block, the predictor for the neighboring block itself is used as the predictor for the current block (see Fig.11 ). In this case, the motion compensation process can start immediately without waiting for the motion compensation process of the previous block to be completed.

[0076] Independent of motion compensation

[0077] The motion compensation process still has some dependency on neighboring block values ​​(typically using the samples used in the top and left templates to start the motion refinement process). To break this dependency, the FRUC mode can be restricted to CUs inside a CTU (or, in an alternative embodiment, a region of a given size).

[0078] exist Fig.12 In JEM FRUC, we show an example of such restriction. For example, if the top and left templates are to be used, CU0, CU1 and CU3 will not be able to use FRUC mode because it uses samples from another CTU. However, CU2 can use FRUC template mode because the data dependency is restricted to inside the CTU. In JEM FRUC, the availability of the left and top neighboring templates are tested independently, and, if at least one is available, FRUC is performed. In this case, it is not possible for CU0, but it is possible only for CU3 with the left template and only for CU1 with the top template.

[0079] In another embodiment, the restriction applies only to the CTU on the left, and CU3 is then allowed to have FRUC template mode.

[0080] This allows parallelization of several CTUs in the motion compensation module.

[0081] Note that this method applies to both FRUC and IC calculations.

[0082] In another embodiment, the above restrictions only apply to the update of the motion vector predictor: when a neighboring CU uses a predictor outside the CTU, only the motion vector predictor of the neighboring CU can be used, not the final motion vector value, but when the CU uses the motion vector predictor from the CU inside the CTU, the final motion vector is used as the predictor for the current CU.

[0083] This allows parallelization of several CTUs in a decoding module, thereby allowing more parallelization on further modules.

[0084] The associated syntax may be signaled at one or more of the slice, PPS (picture parameter set) or SPS (sequence parameter set) levels, such as one or more flags, selections from lists, other indicators, such as limits on FRUC or IC. Other levels, high-level syntax or other means are used in other embodiments. The associated syntax for this signaling includes, for example, one or more flags, selections from lists, other indicators.

[0085] Independent motion vector decoding

[0086] In order to make the motion vector decoding not to stall or wait for the final result of motion compensation, another method is to make the derivation process of the motion vector independent of the motion vector value itself. In this case, the motion vector derivation uses a modified process.

[0087] Fig.13 An example of motion vector predictor derivation is shown.

[0088] In the default process, each new candidate vector is compared with the vectors already in the list before it is added to the list. The comparison here can refer to motion vector equality, equal reference pictures and optionally IC usage equality.

[0089] The new approach consists in replacing the vector equality check in the module's "check list" with another check: checking the predicted values ​​(instead of the final motion vector values), or bypassing the check (see Fig.14 ).

[0090] Various embodiments include one or more of the following:

[0091] - Using the motion vector predictor instead of the final motion vector value as the predictor for the neighboring CU. Several of these embodiments solve the dependency problem of FRUC between the decoding and motion compensation modules.

[0092] - Restrict the reconstructed samples for FRUC and IC to a region.

[0093] - Allows decoding of parameters independent of the final value of the motion vectors.

[0094] Fig.16 One embodiment of a method 1600 for reducing data dependencies in an encoder is shown. The method begins at start block 1601, and control passes to block 1610 to obtain information from neighboring video blocks before the information of the current video block is refined for the neighboring video blocks. Control passes from block 1610 to block 1620 to refine the information for the current video block. Control passes from block 1620 to block 1630 to encode the current video block using the refined information.

[0095] Fig.17 One embodiment of a method 1700 for reducing data dependencies in a decoder is shown. The method begins at start block 1701, and control proceeds to block 1710 for obtaining information from reconstructed neighboring video blocks before the information of the current video block is refined for use in neighboring video blocks. Control proceeds from block 1710 to block 1720 for refining the information for use in the current video block. Control proceeds from block 1720 to block 1730 for decoding the current video block using the refined information.

[0096] Fig.18 One embodiment of an apparatus 1800 for encoding or decoding video blocks with reduced data dependencies is shown. The apparatus includes a processor 2010 having one or more input and output ports and interconnected with a memory 2020 via one or more communication ports. The apparatus 2000 is capable of executing Fig.16 or Fig.17 or any variation thereof.

[0097] This article describes multiple aspects including tools, features, embodiments, models, methods, etc. Many of these aspects are described in a specific way, and at least to illustrate the various features, they are often described in a way that may appear to be limited. However, this is for clarity of description and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous applications.

[0098] The aspects described and contemplated herein may be implemented in many different forms. Figure 8 , Fig. 9 and Fig.15 Some embodiments are provided, but other embodiments are contemplated, and Figure 8 , 9The discussion of 15 does not limit the scope of the embodiments. At least one of these aspects generally relates to video encoding and decoding, and at least another aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0099] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0100] Various methods are described above, and each method includes one or more steps or actions for implementing the described method. Unless a specific step or action sequence is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0101] Various methods and other aspects described herein can be used to modify modules, such as, for example, Figure 8 Motion compensation 170 and motion estimation 175 and Fig. 9 Motion estimation 275. Furthermore, the present aspects are not limited to JVET or HEVC, and may be applied, for example, to other standards and recommendations, whether pre-existing or developed in the future, as well as extensions of any such standards and recommendations (including JVET and HEVC). Unless otherwise specified or technically excluded, the various aspects described herein may be used alone or in combination.

[0102] Various numerical values ​​may be shown in this document. The specific values ​​are for exemplary purposes, and the described aspects are not limited to these specific values.

[0103] Figure 8 An exemplary encoder 100 is illustrated. Variations of the encoder 100 are contemplated, but for clarity, the encoder 100 is described below without describing all contemplated variations.

[0104] Before being encoded, the video sequence may undergo a pre-encoding process (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0105] In an exemplary encoder 100, a picture is encoded by an encoder element as described below. The picture to be encoded is partitioned (102) and processed, for example, in units of CUs. Each unit is encoded, for example, using intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode is used to encode the unit, and indicates the intra / inter decision, for example, by a prediction mode flag. The prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0106] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients are entropy encoded (145) along with motion vectors and other syntax elements to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can also bypass both the transform and quantization, i.e., directly encode the residual without applying the transform or quantization process.

[0107] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to, for example, perform deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0108] Fig. 9 A block diagram of an exemplary video decoder 200 is illustrated. In the exemplary decoder 200, a bitstream is decoded by decoder elements, as described below. The video decoder 200 generally performs the same Figure 1 The decoding path is the inverse of the encoding path described in . The encoder 100 also typically performs video decoding as part of encoding the video data.

[0109] Specifically, the input to the decoder includes a video bitstream that may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoding information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can divide (235) the picture according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored at a reference picture buffer (280).

[0110] The decoded pictures may further undergo post-decoding processing (285), such as inverse color conversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0111] Fig.15 A block diagram illustrating an example of a system in which various aspects and embodiments are implemented. System 1000 may be embodied as a device including various components described below, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1000 are embodied in a single integrated circuit, multiple ICs and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other systems or other electronic devices, for example, via a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0112] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 1010 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 1000 also includes at least one memory 1020 (e.g., a volatile memory device, a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, the storage device 1040 may include an internal storage device, an attached storage device, and / or a network accessible storage device.

[0113] The system 1000 may also include an encoder / decoder module 1000, which is configured to process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a (multiple) module that may be included in the device to perform encoding and / or decoding functions. As known, the device may include one or both of the encoding and decoding modules. In addition, as known to those skilled in the art, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software.

[0114] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to an exemplary embodiment, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. Such stored items include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results resulting from the processing of equations, formulas, operations, and operational logic.

[0115] In several embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations such as MPEG-2, HEVC, or VVC (Versatile Video Coding).

[0116] Input to the elements of system 1000 may be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted over the air, such as by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0117] In various embodiments, the input device of block 1130 has associated corresponding input processing elements, as known in the art. For example, the RF portion may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band limiting the frequency band to a narrower frequency band to select a signal frequency band that (for example) may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF portion of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (e.g., cable) medium, and by filtering, down-convert and filter to the desired frequency band again to perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions. The interpolation element can be included in the insertion element between existing elements, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.

[0118] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across the USB and / or HDMI connections. It is to be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within the processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processor 1010. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and an encoder / decoder 1030 operating in conjunction with memory and storage elements to process the data stream for presentation on an output device.

[0119] The various elements of the system 1000 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and data may be transferred between them using a suitable connection arrangement 1140, such as an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards.

[0120] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0121] In various embodiments, a wireless network such as IEEE 802.11 is used to transmit the data stream to the system 1000. The wireless signals of these embodiments are received on a communication channel 1060 and a communication interface 1050 adapted for wireless communications such as Wi-Fi communications. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other air communications. Other embodiments use a set-top box to provide streaming data to the system 1000, which passes data through the HDMI connection of the input block 1130. Other embodiments also use the RF connection of the input block 1130 to provide streaming data to the system 1000.

[0122] The system 1000 can provide output signals to various output devices including a display 1100, speakers 1110, and other peripherals 1120. In various examples of embodiments, the other peripherals 1120 include one or more independent DVRs, disk players, stereo systems, lighting systems, and other devices that provide functions based on the output of the system 1000. In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripherals 1120 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 using a communication channel 1060 via a communication interface 1050. The display 1100 and speakers 1110 can be integrated into a single unit with other components of the electronic device (e.g., television) system 1000. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0123] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 can be separate from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal can be provided via a dedicated output connection (including, for example, an HDMI port, a USB port, or a COMP output).

[0124] The exemplary embodiments may be performed by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the exemplary embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, optical storage devices, magnetic storage devices, semiconductor-based storage devices, fixed memory, and removable memory. As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0125] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). The device can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a device (e.g., a processor), which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0126] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations appearing in various places throughout this document are not necessarily all referring to the same embodiment.

[0127] Additionally, this document may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0128] Additionally, this document may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0129] Additionally, this document may refer to "receiving" various information. As with "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory device). Furthermore, "receiving" typically involves, in one way or another, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information during an operation.

[0130] It will be apparent to those skilled in the art that implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method or data generated by one of the described embodiments. Such signals may be formatted to carry a bitstream of the described embodiments. Such signals may be formatted as, for example, electromagnetic waves (e.g., using a radio frequency portion of a spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0131] The foregoing description has described a number of embodiments. These embodiments include the following optional features, alone or in any combination, across a variety of claim categories and types:

[0132] - Relax, reduce, or otherwise modify data dependencies created by encoding and / or decoding tools

[0133] - Tools include FRUC

[0134] - Use forecasts instead of final values

[0135] -Data dependencies are dependencies between the block being decoded and its neighboring blocks

[0136] - Using the predictor of the motion vector (or other encoding / decoding parameter, such as e.g. quantization parameter) of a block as the predictor of another block instead of the final motion vector (or other encoding / decoding parameter) value of the block.

[0137] - Block is CU

[0138] - The other block is a neighboring block

[0139] - Relax, reduce or otherwise modify the FRUC dependency between the decoding and motion compensation modules.

[0140] - Restrict the reconstructed samples used for FRUC and IC to within a region of the image.

[0141] - The area is all or part of a CTU

[0142] - Allows decoding of motion vectors independent of the final value of the motion vector.

[0143] - Relax, reduce, or otherwise modify data dependencies between the block being decoded and neighboring blocks

[0144] -FRUC mode is limited to using the CU inside the CTU

[0145] -FRUC mode is limited to constraining data dependencies within CTUs or other blocks

[0146] -FRUC mode is limited to constraining data dependencies within the CTU and one additional CTU

[0147] - a bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0148] - Insertion of signalling syntax elements enabling the decoder to process the bitstream in the reverse manner to that performed by the encoder.

[0149] - creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of said syntax elements or variations thereof.

[0150] - A television, set-top box, cellular phone, tablet computer or other electronic device implementing any of the described embodiments.

[0151] - A television, set-top box, cellular phone, tablet computer, or other electronic device that executes any of the described embodiments and displays the resulting images (e.g., using a monitor, screen, or other type of display).

[0152] - A television, set-top box, cellular phone, tablet computer, or other electronic device that tunes (e.g., using a tuner) to a channel to receive a signal including an encoded image, and performs any of the described embodiments.

[0153] - A television, set-top box, cellular telephone, tablet computer, or other electronic device that receives over the air (e.g., using an antenna) a signal including the encoded image and performs any of the described embodiments.

[0154] - Throughout this disclosure, various other generalized and specific features are also supported and contemplated.

Claims

1. A method comprising: obtaining information from a neighboring video block before the information for the current video block is refined for use in the neighboring video block; Refining the information for use by the current video block; When the coding unit is outside the current coding tree unit, the motion vector predictor from the neighboring coding unit is used, and when the coding unit uses the motion vector predictor from the coding unit within the current coding tree unit, the final motion vector is used; as well as, The current video block is encoded using the refined information.

2. An apparatus for encoding a video block, comprising: Memory, and Processor, configured as: obtaining information from a neighboring video block before the information for the current video block is refined for use in the neighboring video block; Refining the information for use by the current video block; When the coding unit is outside the current coding tree unit, the motion vector predictor from the neighboring coding unit is used, and when the coding unit uses the motion vector predictor from the coding unit within the current coding tree unit, the final motion vector is used; as well as, The current video block is encoded using the refined information.

3. A method comprising: obtaining information from a neighboring video block before the information for the current video block is refined for use in a reconstructed neighboring video block; Refining the information for use by the current video block; When the coding unit is outside the current coding tree unit, a motion vector predictor from a neighboring coding unit is used, and when the coding unit uses a motion vector predictor from a coding unit within the current coding tree unit, a final motion vector is used; as well as, The current video block is decoded using the refined information.

4. An apparatus for decoding a video block, comprising: Memory, and Processor, configured as: obtaining information from a neighboring video block before the information for the current video block is refined for use in a reconstructed neighboring video block; Refining the information for use by the current video block; When the coding unit is outside the current coding tree unit, a motion vector predictor from a neighboring coding unit is used, and when the coding unit uses a motion vector predictor from a coding unit within the current coding tree unit, a final motion vector is used; as well as, The current video block is decoded using the refined information.

5. The method according to claim 1 or 3 or the device according to claim 2 or 4, wherein The information includes a motion vector predictor; The refinement of the motion vector predictor of the current video block comprises frame rate up-conversion to generate a motion vector; and The encoding includes using the motion vector for the current block.

6. The method according to claim 5, wherein The refinement of the motion vector predictor is based on template matching.

7. The method according to claim 6, wherein: The template matching is restricted to the coding tree unit containing the current video block.

8. The method of claim 5, wherein when the coding unit is outside the current coding tree unit, a motion vector predictor from a neighboring coding unit is used, and when the coding unit uses a motion vector predictor from a coding unit within the current coding tree unit, a final motion vector is used.

9. The method according to claim 1 or 3 or the device according to claim 2 or 4, wherein: Before adding a motion vector predictor to the list of candidates, it is checked whether the motion vector predictor is in the list.

10. The method according to claim 1 or 3 or the device according to claim 2 or 4, wherein: The refinement is signaled using syntax.

11. The method according to claim 1 or 3 or the device according to claim 2 or 4, wherein: The refinement includes illumination compensation.

12. A device comprising: The device according to any one of claims 4 to 11; as well as At least one of: (i) an antenna configured to receive a signal over the air, the signal comprising a video block; (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, and (iii) a display configured to display an output.

13. A non-transitory computer readable medium comprising data content generated by the method of any one of claims 1 and 5 to 12 or by the apparatus of any one of claims 2 and 5 to 12 for playback using a processor.

14. A signal comprising video data generated by the method of any one of claims 1 and 5 to 12 or by the apparatus of any one of claims 2 and 5 to 12 for playback using a processor.

15. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 3 and 5 to 12.

Images