Method and apparatus for performing real-time VVC decoding

Abstract

An apparatus and method for implementing a real-time generic video coding (VVC) decoder uses multiple threads to address the limitations of existing parallelization techniques and fully utilizes available CPU computational resources without compromising coding efficiency. The proposed multi-threaded (MT) framework uses CTU-level parallelism without compromising memory bandwidth. Picture-level parallelism separates a sequence into temporal levels by considering the picture reference hierarchy. Embodiments are provided using various optimization techniques to achieve real-time VVC decoding on heterogeneous platforms with multi-core CPUs for bitstreams generated using a VVC reference encoder with a default configuration.

Description

[Technical Field]

[0001] At least one of the present embodiments generally relates to a method or apparatus for video decoding or compression. [Background technology]

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction, including motion vector prediction, and transformation that exploits spatial and temporal redundancy in the video content. Typically, intra- or inter-prediction is used to exploit correlation within or between frames, and then the difference between the original and predicted image, often called the prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to entropy coding, quantization, transformation, and prediction. Summary of the Invention

[0003] At least one of the present embodiments generally relates to a method or apparatus for performing real-time decoding, such as the VVC (Universal Video Coding or H.266) standard.

[0004] According to a first aspect, there is provided a method, comprising: allocating threads to CABAC decoding of video frames using parallel processing, wherein more threads are allocated to decoding frames having a low quantization parameter compared to a higher quantization parameter, or intra frames relative to inter frames, or P frames, or when frames are at a lower temporal tier; and decoding the video frames using the allocated threads.

[0005] According to a second aspect, another method is provided, the method comprising steps of generating motion vectors for a coding unit, performing inter-coding unit reconstruction, performing intra-coding unit reconstruction, and performing inverse reshaping and in-loop filtering of the video, wherein the generating and performing steps are performed simultaneously on separate portions of the video.

[0006] According to another aspect, an apparatus is provided, comprising: a device including a memory and one or more processors configured to allocate threads to CABAC decoding of video frames using parallel processing, wherein more threads are allocated to decoding of frames having a low quantization parameter compared to a higher quantization parameter, or intra frames relative to inter frames, or P frames, or when frames are at a lower temporal tier, and to decode the video frames using the allocated threads.

[0007] According to another aspect, another apparatus is provided, comprising: a device comprising a memory and one or more processors configured to generate motion vectors for coding units, perform inter-coding unit reconstruction, perform intra-coding unit reconstruction, and perform inverse reshaping and in-loop filtering of the video simultaneously for separate portions of the video.

[0008] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments and at least one of: (i) an antenna configured to receive a signal, the signal including tensors of video blocks and feature maps; (ii) a band limiter configured to limit the received signal to a frequency band including the video blocks; or (iii) a display configured to display an output representative of the video blocks, or any receiving device that analyzes features and / or decoded content.

[0009] According to another general aspect of at least one embodiment, there is provided a non-transitory computer-readable medium that includes data content decoded according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, there is provided a signal including video data decoded according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings. [Brief explanation of the drawings]

[0014] [Figure 1]1 shows an overall diagram of a VVC block-based video encoder. [Figure 2] Showing block partitions in multiple types of tree structures: (a) quaternary partitions, (b) vertical binary partitions, (c) horizontal binary partitions, (d) vertical ternary partitions, and (e) horizontal ternary partitions. [Figure 3] 1 shows an overall block diagram of a VVC video decoder. [Figure 4] 1 shows an exemplary overview of the slice / picture decoding stage. [Figure 5] An example of CABAC decoding with WPP parallelization is shown. [Figure 6] 1 illustrates an exemplary thread scheduling mechanism for the slice / picture reconstruction stage. [Figure 7] 10 illustrates coding unit motion vector generation with WPP parallelization. [Figure 8] 1 illustrates an exemplary parallelization within the CUMV decoding stage. [Figure 9] 1 illustrates parallelization within the inter-coding unit reconstruction decoding stage. [Figure 10] 10 illustrates an example of parallel non-joint intra-coding unit reconstruction. [Figure 11] 1 illustrates joint parallelization of inter-coding unit and intra-coding unit reconstruction decoding stages. [Figure 12] 1 illustrates the interleaving of horizontal and vertical deblocking filter decoding stages. [Figure 13] 1 illustrates the interleaving process of the coding unit motion vector generation and inter-coding unit reconstruction decoding stages. [Figure 14] 1 illustrates the interleaving process of the inter- and intra-coding unit reconstruction decoding stage. [Figure 15] 1 illustrates the interleaving process of the inter- and intra-coding unit reconstruction and inverse reshaping decoding stages. [Figure 16a]1 shows a) a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented, and b) a system diagram illustrating an example WTRU. [Figure 16b] 1 shows a) a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented, and b) a system diagram illustrating an example WTRU. [Figure 17] 1 illustrates one embodiment of a method under the general aspects described. [Figure 18] 1 illustrates a second embodiment of a method under the general aspects described. [Figure 19] 1 shows an apparatus for carrying out any of the described methods; DETAILED DESCRIPTION OF THE INVENTION

[0015] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Among various types of video coding systems, such as block-based, wavelet-based, and object-based systems, block-based hybrid video coding systems are the most widely used and deployed today. Examples of block-based video coding systems include international video coding standards such as MPEG1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10, AVC, and VC-1, as well as the latest video coding standard called High Efficiency Video Coding (HEVC), which was developed by the Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T / SG16 / Q.6 / VCEG and ISO / IEC / MPEG.

[0016] The first version of the HEVC standard was finalized in October 2013, providing approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard, H.264 / MPEG AVC. While the HEVC standard offers significant coding improvements over its predecessor, there is evidence that superior coding efficiency can be achieved using additional coding tools beyond HEVC. Based on this, both VCEG and MPEG have begun work on exploring new coding techniques for future video coding standardization. The Joint Video Exploration Team (JVET), formed by ITU-T VECG and ISO / IEC MPEG in October 2015, has begun significant research into advanced techniques that could enable substantial increases in coding efficiency. One reference software, called the joint exploration model (JEM), was maintained by JVET by incorporating several additional coding tools on top of the HEVC test model (HM).

[0017] In October 2017, ITU-T and ISO / IEC issued a joint call for proposals for video compression with capabilities beyond HEVC. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, which demonstrated a compression efficiency increase of approximately 40% over HEVC. Based on these evaluation results, JVET launched a new project to develop a new generation video coding standard named Versatile Video Coding (VVC). In the same month, a reference software code base called the VVC test model (VTM) was established to demonstrate a reference implementation of the VVC standard. Meanwhile, another reference software base called the benchmark set (BMS) was also created to facilitate the evaluation of new coding tools. The BMS code base includes a list of additional coding tools on top of VTM that offer higher coding efficiency and moderate implementation complexity, and will be used as a benchmark when evaluating similar coding technologies during the VVC standardization process. Specifically, in addition to JEM coding tools such as 4x4 non-separable secondary transform (NSST), generalized bi-prediction (GBi), bi-directional optical flow (BIO), decoder-side motion vector refinement (DMVR), and current picture referencing (CPR) integrated into BMS-2.0, it includes trellis coded quantization tools.

[0018] Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 1 shows a block diagram of a general block-based hybrid video coding system. An input video signal 102 is processed block by block (called a coding unit (CU)). In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC, which partitions blocks only based on a quadtree, in VTM-1.0, a coding tree unit (CTU) is partitioned into CUs to adapt to various local characteristics based on a quad / binary / ternary tree. Furthermore, the concept of multiple partition unit types in HEVC has been eliminated; that is, the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC. Instead, each CU is always used as the basic unit for both prediction and transformation without further partitioning. In the multi-type tree structure, a CTU is first partitioned by a quadtree structure. Each quadtree leaf node can then be further divided by binary and ternary tree structures. As shown in Figure 2, there are five division types: quaternary partition, horizontal binary partition, vertical binary partition, horizontal ternary partition, and vertical ternary partition. In Figure 1, spatial prediction (160) and / or temporal prediction (162) can be performed. Spatial prediction (or "intra-prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) within the same video picture / slice to predict the current video block. Spatial prediction reduces spatial redundancy inherent in video signals. Temporal prediction (also called "inter-prediction" or "motion-compensated prediction") uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces temporal redundancy inherent in video signals.The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. If multiple reference pictures are supported, a reference picture index is additionally transmitted, which is used to identify which reference picture in the reference picture store (164) the temporal prediction signal comes from. After spatial and / or temporal prediction, a mode decision block (180) in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. The prediction block is then subtracted (116) from the current video block, and the prediction residual is de-correlated (104) using a transform and quantized (106). The quantized residual coefficients are inverse quantized (110) and inverse transformed (112) to form a reconstructed residual, which is then added back (126) to the prediction block to form the reconstructed signal for the CU. Further loop filtering, such as a deblocking filter, sample adaptive offset (SAO), and adaptive in-loop filter (ALF), may be applied to the reconstructed CU (166), which is then placed into a reference picture store (164) and used to code future video blocks. To form the output video bitstream 120, the coding mode (intra or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit (108) for further compression and packing to form the bitstream.

[0019] Within the inter prediction and loop filtering stages of VVC, the following tools are new compared to HEVC: 1) Combined Inter / Intra Prediction (CIIP) 2) Decoder-side Motion Vector Refinement (DMVR) 3) Reverse reshaping The CIIP mode combines inter prediction signals with intra prediction signals. The inter and intra prediction signals are combined using weighted averaging, where the weights are based on the coding modes of the left and above blocks.

[0020] In VVC, in combined inter / intra prediction (CIIP) mode, an additional flag is signaled per CU if the following conditions are met: When a CU is coded in merge mode The CU contains more than 64 luma samples, i.e., CU width x CU height is 64 or more. CU height and CU width are less than 128 luma samples

[0021] VVC can increase MV accuracy using bidirectional matching-based motion vector correction at the decoder, also known as decoder-side motion vector correction (DMVR). Template matching is performed at the decoder to correct MVs around the initial MV in reference picture lists L0 and L1. The SAD is calculated between each corrected MV candidate within a search range (25-point full search in integer samples) around the initial MV in raster scan order. The MV candidate with the lowest SAD is used to generate a bi-predictive signal.

[0022] In VVC, DMVR can be applied to each CU using the following features: CU level merge mode with bi-predictive MV · With respect to the current picture, one reference picture is in the past and the other reference picture is in the future. · The POC (Picture Order Count) distance between both reference pictures to the current picture is the same. CU contains more than 64 luma samples Both the CU height and CU width are 8 or more luma samples CU-level weight (BCW) Biprediction with weight index indicates equal weight The current block does not have Weighted Prediction (WP) enabled

[0023] Luma mapping with chroma scaling (LMCS) is added as a new tool within VVC that can be enabled / disabled at the sequence level using a sequence parameter set (SPS) flag. LMCS is applied just before the loop filtering stage. For inter-coded blocks, motion compensated prediction is applied to the mapped domain, i.e., a forward mapping function is applied to the luma prediction block in the original domain to transform it to the mapped domain. For intra-coded blocks, forward mapping is not applied.

[0024] LMCS consists of two parts: 1. luma mapping in a piecewise linear model, and 2. luma-dependent chroma scaling. 1. Luma mapping in piecewise linear models The luma mapping model is signaled at the tile group level. If the luma mapping model present flag is detected for the current tile group, the corresponding piecewise linear model parameters are signaled. The signaled codeword is used to calculate scaling coefficients and adjust the mapping function for each of the 16 equal pieces. 2. Luma-dependent chroma residual scaling Chroma residual scaling is also signaled at the tile group level. If luma mapping is enabled and dual-tree partitioning does not apply to the current tile group, an additional flag is signaled to indicate the use of luma-dependent chroma residual scaling. Luma-dependent chroma residual scaling is disabled for chroma blocks whose area is less than or equal to 4. The purpose of using chroma residual scaling is to compensate for the interaction between the luma and chroma signals.

[0025] 3 shows an overall block diagram of a block-based video decoder. A video bitstream 202 is first unpacked and entropy decoded in an entropy decoding unit 208. Coding mode and prediction information is sent to either a spatial prediction unit 260 (for intra-coded cases) or a temporal prediction unit 262 (for inter-coded cases) to form a prediction block. Residual transform coefficients are sent to an inverse quantization unit 210 and an inverse transform unit 212 to reconstruct a residual block. The prediction block and residual block are then added together at 226. The reconstructed block may further pass in-loop filtering before being stored in a reference picture store 264. The reconstructed video in the reference picture store is then sent to drive a display device and for use in predicting future video blocks.

[0026] Several published papers have exploited the parallelization capabilities of video codecs prior to VVC on CPUs, GPUs, or heterogeneous platforms. Another approach proposed an HEVC deblocking filter that utilizes independent regions of a frame and reduces overall memory accesses. In another approach, GPUs are used to accelerate the HEVC decoding stages of inverse quantization, inverse transform, intra prediction, deblocking filter, and SAO. Another approach proposes GPU-based parallel algorithms, including parallel computation of sample classification, statistics collection for each coding tree block, parallel computation of the best offset value and minimum distortion for each class of edge offset and band offset, SAO merging, and SAO filtering.

[0027] Frame-level parallelism consists of processing multiple frames simultaneously while satisfying motion compensation dependencies. One of the main limitations of frame-level parallelism is that the level of parallelism is determined by the length of the motion vectors, which is a major bottleneck, especially for sequences with high motion.

[0028] Slice-level parallelism allows slices to be processed independently of each other. The main drawback of this level of parallelism is that the number of slices is determined by the encoder. Furthermore, in most cases, codecs may be limited to using one slice per frame, resulting in little slice-level parallelism. Furthermore, slicing reduces coding efficiency.

[0029] The above drawbacks of frame-level or slice-level parallelization can be overcome by using Wavefront Parallel Processing (WPP) and tiling. Both techniques allow each picture to be subdivided into multiple partitions that can be processed in parallel. Each partition contains an integer number of coding units that need not have dependencies on other partitions. The use of tiles reduces coding efficiency and complicates raster scan processing. However, the main drawback of using tiles is the rate-distortion loss, which would be significant with a larger number of tiles, due to the lack of dependency between partitions.

[0030] Wavefront parallelism (WPP) allows for partitioning of a picture without breaking coding dependencies or resetting CABAC probabilities as in slice or tile parallelism. The WPP method partitions a picture into CTU rows and allows prediction and entropy coding across CTU row boundaries. For this reason, WPP results in lower loss of compression efficiency compared to slice and tile parallelism methods. However, wavefront dependencies do not allow all CTU rows to start decoding at the same time. Therefore, rows also do not finish decoding at the same time. This introduces parallelism efficiencies that become more apparent when multiple WPP threads are used for decoding.

[0031] Overlapped Wavefront (OWF) parallelism improves the implementation efficiency of WPP by overlapping the execution of successive pictures. The dependency in OWF technology is caused by motion search. In OWF, the CTU is not ready for decoding until all reference pixels within the motion search region in the reference picture have been decoded. Motion search dependency limits the throughput of frame-level parallel processing threads for encoding multiple frames. This problem is exacerbated when the center of the search window (determined by the motion estimator) is located toward the bottom of the reference picture. On the other hand, limiting the motion vectors results in significant coding loss for videos exhibiting fast vertical motion.

[0032] Therefore, none of the parallelization techniques can fully utilize the CPU processing power available in today's multi-core systems.

[0033] The described embodiments address the limitations of existing parallelization techniques and fully utilize available CPU computational resources without compromising coding efficiency. The proposed multi-threaded (MT) framework uses CTU-level parallelism techniques without compromising memory bandwidth. Picture-level parallelism separates sequences into temporal levels by considering the picture reference hierarchy. This disclosure discusses various optimization techniques used to achieve real-time VVC decoding on heterogeneous platforms with multi-core CPUs for those bitstreams generated using a VVC reference encoder with a default configuration. Equivalent techniques proposed for the decoder can also be used with the VVC encoder framework.

[0034] The current VVC draft includes several aspects to make the coding process parallelizable. These include tile and wavefront parallel processing (WPP). Employing tiles for parallelization introduces coding loss because there is no dependency between tiles. WPP divides slices into CTU rows and processes the rows in parallel while maintaining coding dependencies.

[0035] This embodiment proposes a finer granularity of parallelization without compromising coding efficiency. The section entitled "Parallelization of CABAC with Slice Decoding Stage" presents the finer granularity of parallelization achieved by processing CABAC decoding and the remainder of the slice decoding stage, called the reconstruction decoding stage. An architectural diagram of the slice / picture decoding stage is shown in Figure 4. The parallelization of each of the reconstruction decoding stages, e.g., the inter, intra, reshaper, and loop filter as shown in Figure 4, is described in the section entitled "Parallelization of Slice Decoding Stage."

[0036] In summary, the main contributions to improve the finer grain parallelism of pipelined multi-stage VVC decoders are as follows: In the intra-CU reconstruction stage, all non-joint / independent intra-CTUs (CTUs with all their intra-coded CU reference pixels coded in inter mode) can be reconstructed in parallel once the reference pixel inter-CU (left, top, and top-right block) reconstruction process is completed. Inter prediction of inter CU and combined inter and intra prediction (CIIP) CU prediction processes can be parallelized across all CTUs of a picture. To distribute the computational complexity, WPP-enabled CABAC decoding can be performed in parallel with the reconstruction decoding stage. The number of threads initially allocated for CABAC decoding of a picture can be dynamically changed based on the CABAC execution of the previous picture. Multiple stages of the decoder pipeline can run in parallel within a picture and across temporal layer pictures. For example, the decoding stages of CUMV, inter CU reconstruction, and intra CU reconstruction can run in parallel within a picture or across pictures. Pre-analysis to find the location of the first dependent CU in each CTU and use this information to improve Wavefront Parallel Processing (WPP) parallelization in the CUMV and intra-CU reconstruction stages. To avoid scanning the picture multiple times, the deblocking filter is parallelized by interleaving the horizontal and vertical filtering stages. The number of horizontal and vertical edges within and between CTUs is calculated to distribute the filtering workload evenly among the threads. Add an additional condition to adaptively switch off DMVR. DMVR is used for merge candidates only when the motion vector length is less than a predefined threshold. Disabling DMVR based on the motion vector length results in the availability of reference regions and increases DMVR parallelism.

[0037] Parallelization approaches for the VVC decoding stage can be classified into one of the following three categories. The category is selected for each decoding stage based on the decoding dependencies between CTUs. They are as follows: I. CTU level parallelization (CTUP), II. Improved Overlapped Wavefront (IOWF) Parallelization III. Load-sharing based parallelization.

[0038] Approach I is selected for VVC decoding stages without CTU level dependency, e.g., reshaping, sample adaptive offset (SAO), while Approach II is selected for VVC decoding stages with CTU level dependency along with intra-CU prediction.

[0039] Pipeline design approach I adds fine-grained parallelism to the existing tile-based approach. It is based on flexible partitioning of pictures into CTUs such that dependencies between multiple CTU partitions are prohibited. The number of CUs processed per thread varies dynamically based on the QTBT partitioning.

[0040] Pipeline design approach II addresses the inefficiencies of wavefront parallel processing (WPP) by following an improved overlapped wavefront (IOWF) approach. The IOWF proposed in these embodiments uses wavefronts to enable overlapping the execution of consecutive pictures, regions, e.g., CUs, within a CTU of a picture, and decoding stages within a picture. The section below titled "Parallelization of CABAC with Slice Decoding Stages" highlights a pipeline design based on approach II.

[0041] Our pipeline design approach III parallelizes modules that are difficult to parallelize due to high branch divergence and low data parallelism. CABAC decoding falls under this category. Therefore, CABAC decoding is performed in parallel with the slice reconstruction decoding stage, as described in the section entitled "Parallelization of CABAC with Slice Decoding Stage."

[0042] Parallelization of CABAC with a slice decoding stage. The parallelization of CABAC with the slice decoding stage is based on the proposed approach III. One of the main reasons for VVC decoder latency is the data dependency of the CABAC engine's context selection. This is mainly because the bin content selection depends on the values ​​of previously decoded bins. This dependency is a bottleneck for achieving CABAC parallelization, especially in the decoder. This, in turn, adds to decoder latency. This latency can be reduced if the CABAC decoding of a picture is performed in parallel with the reconstruction decoding stage of other pictures.

[0043] The load balancing between the CABAC decoding stage and the reconstruction decoding stage consists of the following steps:

[0044] I. Thread Priority Scheduling The complexity of CABAC decoding may dictate thread execution priority. For example, the complexity of CABAC decoding for an intra-slice is much higher than that for other slices. Therefore, the CABAC decoding activity for an intra-slice may be scheduled earlier than the picture preceding it in decoding order. Similarly, the complexity of CABAC decoding for a picture in temporal layer 0 is higher than that of temporal layer 1, etc. Therefore, the CABAC decoding of a lower temporal layer picture, which is the next group of pictures (GOP), may be scheduled in parallel with the CABAC decoding of a higher temporal layer picture, which is the previous GOP. In this way, the variation in CABAC decoding time can be balanced among temporal layer pictures. Therefore, thread scheduling is prioritized based on the following: · Early scheduling of the CABAC decoding process for a picture based on the estimated complexity of that frame / slice.

[0045] II. Thread Allocation for WPP-enabled CABAC Decryption As shown in Figure 5, the initial number of threads allocated to WPP-enabled CABAC decoding of a picture depends on the complexity of frame decoding. If the complexity of frame decoding is high, more threads are allocated to CABAC decoding, for example, for low QP, intra-frame, inter-picture, or lower temporal layer. After some threads are allocated to CABAC decoding, the remaining available threads are allocated to process the reconstruction decoding stage of the previous picture. The total number of available threads is distributed between the CABAC and reconstruction decoding stages in the following manner: More threads are allocated to CABAC decoding of frames with low QP compared to high QP, intra frames compared to inter frames, P pictures, or when pictures are at lower temporal tiers. · A larger number of threads are allocated to CABAC decoding of pictures with multiple tiles and slices compared to pictures with a single tile or a single slice.

[0046] III. Dynamic Allocation The number of threads initially allocated for WPP-enabled CABAC decoding of a picture varies dynamically based on the performance of the initial allocation. The latency between CABAC and the reconstruction decoding stage based on the initial allocation can be used as a thread allocation criterion. The goal of dynamically varying the number of threads allocated to CABAC and the reconstruction decoding stage is to reduce the CABAC processing time variation from frame to frame. If the time required to process a similar previous frame is higher than the real-time constraint, the number of threads allocated to CABAC decoding of the current frame is increased. If the latency between CABAC decoding and reconstruction decoding stage is higher, the number of threads allocated to CABAC decoding of the current frame is reduced. One of multiple previous frames can be used in determining dynamic variations in allocation. The weight of the previous frame in determining the dynamic allocation of threads for the current frame varies based on POC distance, time tier, and frame type.

[0047] Parallelization of the slice decoding stage Individual decoding stages after CABAC decoding are parallelized using Approaches I and II. The pipeline design follows the principle of overlapped execution of multiple decoding blocks that can be processed in parallel. When a thread finishes executing a block, it continues executing the next available block. The parallel processing partitions the picture into CTUs or CTU rows without affecting dependencies. Such overlapped execution can occur within one decoder module or between modules. Subsections will explain this in more detail.

[0048] Thread Scheduling This section describes the thread scheduling mechanism shown in FIG. 6 for the CUMV stage, which is described in detail below in the section "CUMV Generation." A thread pool and a first-in, first-out (FIFO) activity queue are created to handle reconstruction decoding stage activities. The thread pool is created based on the number of available CPU cores in the heterogeneous multicore system. Any free thread from the thread pool fetches the first available task from the activity queue and utilizes CPU power to execute the activity. The slice decoding thread pushes a list of tasks to be executed for decoding a slice / picture into the activity queue. The slice decoding thread initially pushes CU motion vector generation for the first CTU row task with a dependency flag set to false, and the second CTU row task with a dependency flag set to true, up to the last CTU row. A free worker thread, referred to as the first worker thread, fetches the first available activity from the task queue and executes CU motion vector generation for the first CTU, continuing until the last CTU in the first row without any dependencies. Another free worker thread fetches the next activity (second CTU row CU motion vector generation) from the queue and processes the activity in parallel with the first worker thread with WPP dependency. This process continues for all slice / picture decoding stages.

[0049] GPU parallel processing The parallelization schemes proposed in the sections titled "Parallelization Within a Decoding Stage" and "Parallelization Between Decoding Stages" are also suitable for GPU parallelization. The VVC decoding architecture is modified to reduce GPU memory usage, e.g., global, cache, or constant memory usage, and to reduce GPU memory accesses. GPU vector instructions are utilized instead of the SIMD optimization used for CPU parallelization to increase parallelism. Furthermore, GPU kernel launches are performed immediately after the file read operation, which removes the execution limitations imposed by memory transfers.

[0050] Parallelization within the decoding stage The current VVC design processes the decoding stage of each CTU sequentially. This design is not parallelization friendly. To improve the degree of parallelization of the VVC decoder stage, it is proposed to divide the decoding stage of each picture into the following subtasks: 1. CU motion vector (CUMV) generation 2. Inter-CU Reconfiguration a. Combined Inter-Intra Prediction (CIIP) b. Decoder Motion Vector Compensation (DMVR) 3. Intra-CU Reconstruction 4. Reverse reshaping 5. In-loop filtering a. Deblocking filter b. Adaptive Loop Filter c. Sample Adaptive Offset

[0051] CUMV generation The CU motion vector derivation process is completed for the entire interslice / picture before proceeding to the reconstruction process. The derivation of a CU motion vector can depend on its left, top, and / or top-right neighboring CU motion vectors, which is called wavefront parallelism (WPP) dependency. To achieve better parallel processing capabilities, each CTU row is pushed into the activity queue in the order of increasing CTU number. As shown in Figure 7, a free worker thread from the thread pool fetches the first available CTU row from the activity queue and generates motion vectors for CTUs in consecutive order until the entire CTU row is completed. Each worker thread always ensures that the top-right CTU has completed generating a motion vector before proceeding to motion vector generation for the current CTU. In this way, a maximum of H (the number of CTU rows) worker threads can be used for parallel processing of the entire picture motion vector generation. Figure 8 illustrates the CUMV motion vector derivation process. The variables used in Figure 8 are explained in Table 1.

[0052] [Table 1]

[0053] The steps used to derive the CUMV motion vector can be summarized as follows: Pre-analysis to determine the location of the first CU within each CTU depending on its upper right CTU CUs up to the position of the first CU in the current CTU can be processed in parallel with the CTU immediately above it, as in step 1. The remaining CUs in each CTU may require the availability of the upper right dependent CU, rather than the entire upper right CTU.

[0054] Inter-CU Reconstruction After deriving the CU motion vectors for the whole picture, the inter CU reconstruction process is separated from the intra and CIIP mode reconstruction processes. The inter CU reconstruction algorithm is detailed in Figure 9. The variables used in Figure 9 are explained in Table 2.

[0055] [Table 2]

[0056] The rest of the decoding threads push the inter-CU reconstruction processes of CTUs or CTU groups onto the activity queue in the order of increasing CTU numbers. A free worker thread from the thread pool fetches the first available CTU or CTU group from the activity queue and performs the inter-CU reconstruction process for the entire CTU or CTU group (one CTU after another). All worker threads perform the inter-CU reconstruction process for that CTU group in parallel, taking advantage of multiple CPU / GPU cores available in the system. In this way, the overall complexity of inter-CU reconstruction of a slice / picture can be dramatically reduced.

[0057] I.CIIP When a CU is coded in merge mode where the number of luma samples exceeds 64 and both the CU width and CU height are less than or equal to 128, an additional flag signals the use of combined inter / intra CU prediction (CIIP) mode. Because inter and intra are combined for the current CU, inter prediction of the CIIP mode CU can be combined with other inter CU prediction processes. The inter CU and CIIP inter CU prediction processes can be parallelized across all CTUs of the picture. In this way, the CIIP inter CU reconstruction process time can be significantly reduced by processing them in parallel using multiple worker threads. The CIIP intra CU reconstruction process is combined with other intra CU reconstruction processes.

[0058] II.DMVR VVC allows corrected motion vectors to be obtained without transmitting them, thus reducing the computational complexity of the encoder. However, in the process, DMVR increases the computational workload and processing time of the decoder. In DMVR, the computational complexity arises due to the search scheme around the search center, the metrics used, such as sum of absolute differences (SAD) to select the correction, and bilinear interpolation to interpolate fractional pixel positions.

[0059] Because the size of the motion vectors (MVs) is limited in either the encoder or decoder, a smaller number of reference regions will be needed to calculate the DMVR output of the current CU. Therefore, by limiting the number of search points in a region or the size of the MVs, DMVR parallelism can be increased at the expense of describing fast motion.

[0060] The DMVR can be adaptively switched on and off without signaling overhead. The MV correction process only works when the following conditions are met: The coding unit is in merge mode Unidirectional, ATMVP, STMVP, affine and merge candidates using IC mode are excluded ·(POC ref0 -POC cur ) value and (POC ref1 -POC cur ) and only one of the values ​​is negative

[0061] In at least one of the described embodiments, an additional condition is added to only use DMVR if the motion vector length is less than a predefined threshold. Disabling DMVR based on the motion vector length provides reference region availability and increases DMVR parallelism for bi-predicted merge candidates.

[0062] Intra-CU Reconfiguration Conventional WPP parallelization of intra-CU prediction treats each CTU row as an independent region. At the CTU level, each CTU must wait until its left and upper right neighboring CTUs finish reconstruction. This process ensures that the current CTU row is always two CTUs more latently than its neighboring row above. To alleviate this problem, the general aspect described herein proposes the following steps for intra-CU prediction parallelization:

[0063] I. Preliminary Analysis The pre-analysis stage determines the location of the first dependent CU of each CTU that depends on the upper right CTU. Find the number of non-joint CTUs in the picture. A non-joint CTU is one whose intra-coded CU reference pixels are all coded in inter mode.

[0064] II. Parallelization Scheme Based on the results of the above pre-analysis, a CU can be processed in parallel with the CTU immediately above it until it becomes the first dependent CU in the current CTU. All non-joint intra CUs can be reconstructed in parallel once their referenced pixel inter CU (left, top, and top right blocks) reconstruction process is complete. A free worker thread can process a certain number of non-joint intra CUs. If more non-joint intra coded CUs are available, each free worker thread can process a certain number of non-joint CUs in parallel with other worker threads. The non-joint coded CTUs in the top row have a higher priority. Each non-joint CTU in the top row is processed by a free worker thread. This thread scheduling method can complete the intra-CU reconfiguration process of the top row faster and significantly reduce the waiting time for the next CTU row. As shown in Figure 10, each non-joint CTU in the first row is processed by a free worker thread after the inter-CU reconfiguration process of that CTU is completed. The WPP thread sequentially performs the first CTU row intra-CU reconstruction process for the remaining CTUs. This WPP thread runs in parallel with the non-joint CTU reconstruction process described above.

[0065] Deblocking Filter Deblocking is a module in the VVC loop filtering stage immediately following the inverse reshaper. In VVC, the vertical deblocking filtering stage is based on the horizontal deblocking filtered output. Therefore, the current design processes the horizontal and vertical filtering stages sequentially. However, there is no data dependency between CTUs for either the horizontal or vertical filtering stages. Therefore, all CTUs for the entire frame can be processed in parallel. The following section, "Interleaving Grouping," proposes a technique to interleave the horizontal and vertical filtering stages and mitigate the increase in memory accesses caused by CTU-level parallelism.

[0066] IV. Edge-based CTU grouping In VVC, the maximum allowed CTU size is 128x128. Therefore, for deblocking performed on an 8x8 grid, the maximum number of allowed vertical edges is 128 / 8=16. However, due to QTBTTT partitioning, the number of vertical and horizontal edges can vary within and between CTUs. Therefore, to distribute the processing load among threads evenly, it is processed to distribute tasks per thread based on a predefined number of edges.

[0067] V. Region of interest (ROI)-based CTU grouping Instead of grouping consecutive CTU rows for thread allocation, a region of interest (ROI) can be selected for each thread processing. Grouping CTUs for DBF processing based on the ROI will result in better memory handling.

[0068] VI. Interleaved Grouping Existing VVC designs process the horizontal and vertical filtering stages sequentially, which requires multiple scans of the picture and increases memory accesses. Instead of processing the entire picture sequentially, we propose interleaving the horizontal and vertical filtering stages. Horizontal filtering of two CTU rows is processed in the first pass, followed by vertical filtering. This ensures the availability of horizontally filtered pixels for the current CTU row and the CTU row below it before vertical filtering of the current CTU row begins. Figure 12 shows the joint parallelization of the horizontal and vertical deblocking stages. The variables used are described in Table 3.

[0069] [Table 3]

[0070] Parallelization between decoding stages Parallelization between decoding stages is based on Approach I. Data structures and dependencies between video coding stages limit the possibility of parallelization within and between stages. For example, a decoding module such as CABAC has high branch divergence and low data parallelism. Similarly, a module such as loop filtering, which has low branch divergence and higher data parallelism, is a good candidate for parallel processing. The proposed parallelization technique performs such an analysis of a VVC decoder based on the criteria of branch divergence and data parallelism to determine the effectiveness of parallelizing one module relative to another. Below, we describe steps for joint parallelization of VVC modules that simultaneously enable low branch divergence and higher data divergence.

[0071] Parallel processing of temporal layer pictures The decoding frame rate, as well as the latency, can be reduced simultaneously by parallel decoding of temporal picture layers. For example, new reference pictures occupy lower temporal layers compared to non-reference pictures, and vice versa. To process consecutive frames in parallel, the following steps are performed in succession: Complete the prediction process for the co-located CUs in the reference picture, the bottom and top right CUs. After the bottom and right CU prediction process is completed, start inverse reshaping and deblocking of the co-located CUs in the reference picture. Apply inverse reshaping and loop filtering to the co-located CU in parallel with the other CU prediction process in the reference picture. Generate a prediction buffer for the current CU.

[0072] Parallel processing of multiple decoding modules within a picture To achieve efficient parallelization of multiple decoding modules, two factors are considered: high branch divergence and percentage of total decoding time. Based on our profiling analysis, the inter-CU prediction and loop filtering stages account for a very high percentage of decoding time. On the other hand, the intra-CU prediction module had high branch divergence. Sub-stages within and between such modules are decoded in parallel to minimize the total decoding time.

[0073] I. CUMV generation and inter-CU reconstruction The parallelization between CUMV generation and inter-CU reconstruction follows Approach II and uses the following steps: Parallel picture-wide CU MV generation following WPP dependency after CABAC decoding Parallel inter-CU reconstruction of the whole picture ● Scheduling the inter-CU reconfiguration of the upper row CTU in parallel with the CUMV generation of other CTUs. After the completion of the CU MV generation process of the upper CTU row, those free worker threads are assigned to the inter-CU reconfiguration process instead of the CU MV generation process of other CTU rows. Early allocation of worker threads to the CU MV generation process of the lower row leads to resource waste because they have WPP dependencies from the CTU row above them. Allocating those worker threads to the inter-CU reconfiguration process helps to utilize resources effectively. As shown in Figure 13, the inter-CU reconstruction process of the upper CTU row is initiated using a number-free worker thread, while the CU motion vector generation process is performed by the WPP worker thread of the lower CTU row.

[0074] II. CUMV generation, inter and intra CU reconstruction The joint parallelization of the inter-CU and intra-CU reconstruction stages is shown in FIG. 11, and the related parameters are described in Table 4. The intra-CU reconstruction follows the inter-CU reconstruction process. However, the intra-CU reconstruction process of the top row CTU can be performed in parallel with the inter-CU reconstruction of other CTUs (e.g., the last row CTU). This will ensure the availability of inter-CU reconstruction samples and at the same time benefit from the joint parallelization of the inter-CU and intra-CU reconstruction.

[0075] As shown in FIG. 14, the motion vector generation and inter- and intra-CU reconstruction decoding stages can overlap. Once a WPP thread completes CU MV generation for a particular CTU row, a free worker thread can be used for inter-CU reconstruction. Once the inter-CU reconstruction of a predefined number of CTUs is complete, the free worker thread can begin executing the intra-CU reconstruction process for the first CTU row. By the time another free worker thread begins executing the intra-CU reconstruction process for the second CTU row, the intra-CU reconstruction processes for many of the CTUs in the first CTU row have been completed. In this way, delays in the intra-CU reconstruction process for the second CTU row due to WPP dependencies can be significantly minimized.

[0076] [Table 4]

[0077] III. Reverse Reshaping and Intra-CU Reconstruction The CTU-level inverse reshaping process and inter and intra CU reconstruction of other CTUs are performed in parallel. As shown in Figure 15, the inverse reshaping process of the top CTU row can be scheduled using a free worker thread before the completion of inter and intra CU reconstruction of the entire picture. This early scheduling allows the entire top row CTU reconstruction process, including deblocking, SAO, and ALF completion, before the completion of inter and intra CU reconstruction of the entire picture. Once the top CTU row reconstruction is complete, the next dependent picture decoding can begin in parallel with the current picture decoding.

[0078] 16A illustrates an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcasts, etc., to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multicarrier (FBMC), and / or the like.

[0079] 16A, communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104, a CN 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or “STA,” may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain contexts), consumer electronics devices, devices operating in commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0080] The communications system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communications networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNodeB, a Home Node B, a Home eNodeB, a gNB, an NR Node B, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each shown as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0081] The base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. The base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers per sector of the cell, for example, using beamforming to transmit and / or receive signals in desired spatial directions.

[0082] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over the air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0083] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base stations 114a of the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communications protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink Packet Access (HSDPA) and / or High-Speed ​​Uplink Packet Access (HSUPA).

[0084] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-APro).

[0085] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as New Radio (NR) radio access, which may establish the air interface 116 using NR.

[0086] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access, e.g., using a dual connectivity (DC) principle. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0087] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity, WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), GSM Evolution (Enhanced Data rates for GSM Evolution, EDGE), GSM EDGE (GERAN), or the like.

[0088] 16A may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point and may utilize any suitable RAT to facilitate wireless connectivity in a local area such as a location such as a business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may establish a picocell or a femtocell using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-APro, NR, etc.). As shown in Figure 16A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 through the CN 106.

[0089] The RAN 104 may communicate with the CN 106, which may be any type of network configured to provide voice, data, application, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions such as user authentication. Although not shown in FIG. 16A , it will be understood that the RAN 104 and / or CN 106 may communicate directly or indirectly with other RANs that use the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may utilize NR radio technology, the CN 106 may also communicate with another RAN (not shown) using GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0090] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a public switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP), and / or internet protocol (IP) of the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs that may use the same RAT as the RAN 104 or a different RAT.

[0091] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links.) For example, the WTRU 102c shown in Figure 16A may be configured to communicate with a base station 114a that can use cellular-based wireless technology and a base station 114b that can use IEEE 802 wireless technology.

[0092] 16B is a system diagram illustrating an example WTRU 102. As shown in FIG. 16B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0093] The processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other function that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 16B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0094] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0095] 16B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0096] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0097] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. Furthermore, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from and store data in memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).

[0098] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components within the WTRU 102. The power source 134 may be any suitable device for providing power to the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0099] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may obtain location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0100] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for pictures and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, a direction sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0101] The WTRU 102 may include a full-duplex radio where transmission and reception of some or all of the signals (e.g., associated with a particular subframe for both the UL (e.g., for transmission) and downlink (e.g., for reception)) may be parallel and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via hardware (e.g., chokes) or processor-based signal processing (e.g., via a separate processor (not shown) or processor 118). In one embodiment, the WTRU 102 may include a half-duplex radio for transmission and reception of either some or all of the signals (e.g., associated with a particular subframe for either the UL (e.g., for transmission) or downlink (e.g., for reception)).

[0102] Although the WTRU is depicted in Figures 16A-16B as a wireless terminal, it is contemplated that in certain representative embodiments, such a terminal may use a wired communication interface (e.g., temporarily or permanently) with the communication network.

[0103] 16A-16B and the corresponding description thereof, one or more or all of the functions described herein with respect to one or more of the WTRUs 102a-d, base stations 114a-b, eNode-Bs 160a-c, MME 162, SGW 164, PGW 166, gNBs 180a-c, AMFs 182a-b, UPFs 184a-b, SMFs 183a-b, DNs 185a-b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation devices may be used to test other devices and / or simulate network and / or WTRU functions.

[0104] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation devices may be directly coupled to another device for testing purposes and / or may perform testing using terrestrial wireless communication.

[0105] One or more emulation devices may perform one or more functions, inclusive, while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in test scenarios in a test lab and / or in an undeployed (e.g., test) wired and / or wireless communication network to implement testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, e.g., one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0106] Similar to the HEVC Experimental Model (HM), the Joint Search Model (JEM) software is also built into the block-based hybrid video coding framework (100). Figure 1 shows a block diagram of a block-based hybrid video coding system. Note that in this application, the terms "reconstructed" and "decoded" may be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, and the term "decoded" is used on the decoder side.

[0107] Before being encoded, a video sequence may undergo preprocessing, for example, applying a color transformation to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resistant to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the preprocessing and attached to the bitstream.

[0108] The input video signal 102 is processed block by block. The HEVC specification distinguishes between "blocks" and "units," where a "block" covers a specific region of a sample array (e.g., luma, Y), and a "unit" includes all coded color components (e.g., Y, Cb, Cr, or monochrome), syntax elements, and collocated blocks of prediction data (e.g., motion vectors) associated with the block. In this application, the term "block" can be used to refer to arrays of data of various sizes, including macroblocks and partitions as specified in H.264 / AVC, coding tree units (CTUs), coding units (CUs), prediction units (PUs), transform units (TUs), coding blocks (CBs), prediction blocks (PBs), and transform blocks (TBs) as in HEVC, superblocks or subpartitions in AV1, and CTUs, CUs, TUs, CBs, and TBs as in VVC (Generic Video Coding) or other video coding standards.

[0109] HEVC uses an extended block size to efficiently compress high-resolution (1080p and above) video signals. In HEVC, a CU can be up to 64x64 pixels. A CU can be further divided into prediction units, to which separate prediction methods are applied. For each input video block (MB or CU), spatial prediction (160) and / or temporal prediction (162) can be performed.

[0110] Spatial prediction (or "intra prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) within the same video picture / slice to predict a current video block. Spatial prediction reduces spatial redundancy inherent in video signals.

[0111] Temporal prediction (also called "inter-prediction" or "motion-compensated prediction") uses reconstructed pixels from an already coded video picture to predict a current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given video block is typically signaled by one or more motion vectors, which indicate the amount and direction of motion between the current block and its reference block. Also, if multiple reference pictures are supported (as in recent video coding standards such as H.264 / AVC or HEVC), then for each video block, its reference picture index is additionally transmitted, and this reference index is used to identify which reference picture in the reference picture store (164) the temporal prediction signal comes from.

[0112] After spatial and / or temporal prediction, a mode decision block (180) in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. The prediction block is then subtracted (116) from the current video block, and the prediction residual is decorrelated (104) using a transform and quantized (106).

[0113] The encoder decodes the coded block to provide reference data for further prediction. The quantized residual coefficients are inverse quantized (110) and inverse transformed (112) to form a reconstructed residual, which is then added back (126) to the prediction block to form a video block.

[0114] The encoder can also skip the transform and apply quantization directly to the untransformed residual signal. The encoder can also bypass both the transform and quantization, i.e., the residual is directly coded without applying a transform or quantization process. In direct pulse code modulation (PCM) coding, no prediction is applied and coding unit samples are coded directly into the bitstream.

[0115] Further loop filtering, such as a deblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter, may be applied to the reconstructed video block (166), which is then placed into a reference picture store (164) and used to code future video blocks. To form the output video bitstream 120, the coding mode (intra or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit (108) for further compression and packing to form the bitstream.

[0116] Figure 3 shows an overall block diagram of a block-based video decoder (200). A video decoder generally performs a decoding pass that is reciprocal to the corresponding encoding pass, performing video decoding as part of encoding video data. A video bitstream 202 is first unpacked and entropy decoded in an entropy decoding unit 208. Coding mode and prediction information is sent to either a spatial prediction unit 260 (if intra-coded) or a temporal prediction unit 262 (if inter-coded) to form a prediction block. Residual transform coefficients are sent to an inverse quantization unit 210 and an inverse transform unit 212 to reconstruct a residual block. The prediction block and residual block are then added together at 226. The reconstructed block may further pass through in-loop filtering (266) before being stored in a reference picture store 264. The reconstructed video in the reference picture store (220) may then be stored, transmitted, or used to drive a display device, as well as to predict future video blocks.

[0117] The decoded picture may further undergo post-processing, such as an inverse color transformation (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process. The post-processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0118] Both HEVC and JEM conform to a block-based motion-compensated hybrid video encoding / decoding workflow, as shown in Figures 1 and 3, and are based on the same functional modules, such as spatial prediction (i.e., intra-prediction), temporal prediction (i.e., inter-prediction), transform, quantization, entropy coding, and loop filter. However, some inter-coding modules, especially those related to motion-compensated prediction, are further extended and improved.

[0119] One embodiment of a method 1700 under general aspects described herein is shown in Figure 17. The method begins at start block 1701, and control proceeds to block 1710, which assigns threads to CABAC decoding of video frames using parallel processing, with more threads being assigned to decoding frames with lower quantization parameters compared to higher quantization parameters, or intra frames relative to inter frames, or P frames, or when frames are at lower temporal tiers. Control proceeds from block 1710 to block 1720, which decodes the video frames using the assigned threads.

[0120] A second embodiment of a method 1800 under the general aspects described herein is shown in Figure 18. The method begins at start block 1801, with control passing to block 1810, which generates motion vectors for a coding unit. Control passes from block 1810 to block 1820, which performs inter-coding unit reconstruction. Control passes from block 1820 to block 1830, which performs intra-coding unit reconstruction. Control passes from block 1830 to block 1840, which performs inverse reshaping and in-loop filtering of the video, with the generating and performing steps being performed simultaneously on separate portions of the video.

[0121] Figure 19 illustrates one embodiment of an apparatus 1900 for implementing the method of Figure 17 or 18. The apparatus includes a processor 1910 and may be interconnected through at least one port to a memory 1920. Both the processor 1910 and the memory 1920 may also have one or more additional interconnections to external connections.

[0122] Additionally, the processor 1910 is configured to insert or receive information in a bitstream and compress, encode, or decode using any of the described aspects.

[0123] The embodiments described herein include various aspects, including tools, features, examples, models, approaches, and the like. Many of these aspects are described with specificity and, often in a manner that may sound limiting, at least to illustrate their individual characteristics. However, this is for clarity of description only and does not limit the applicability or scope of the aspects. In fact, all of the different aspects may be combined or interchanged to provide further aspects. Furthermore, these aspects may be combined or substituted with aspects described in previous applications.

[0124] As will be apparent to those skilled in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted, for example. Such information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0125] The preceding sections describe several embodiments across various claim categories and types. Features of these embodiments may be provided alone or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in combination: Allocating threads to CABAC decoding of video frames using parallel processing, generating a decoded video stream; The method as described above, wherein more threads are allocated to decoding frames with low quantization parameters compared to higher quantization parameters, or intra frames relative to inter frames, or P frames, or when frames are at lower temporal layers; Decoding the video frame using the allocated thread.

Claims

1. 1. A method of decoding video, comprising: allocating threads for CABAC decoding of video frames using parallel processing, wherein more threads are allocated to decoding frames with low quantization parameters compared to higher quantization parameters, or intra frames relative to inter frames, or P frames, or when the frames are in a lower temporal tier; decoding the video frames using the allocated threads, where if the time taken to process a previous frame is greater than a real-time limit, the number of threads allocated for CABAC decoding of a current frame is increased; A method for providing

2. A device, memory and one or more processors Equipped with The one or more processors: using parallel processing to allocate threads for CABAC decoding of video frames, with more threads being allocated to decoding frames with low quantization parameters compared to higher quantization parameters, or intra frames versus inter frames, or P frames, or when the frames are in a lower temporal tier; The allocated threads are used to decode the video frame, and if the time taken to process the previous frame is greater than the real-time limit, the number of threads allocated for CABAC decoding of the current frame is increased. Devices configured to:

3. 10. The method of claim 1, wherein the number of allocated threads varies dynamically based on an initial allocation performed.

4. The method of claim 3, wherein the number of threads allocated to the current frame of the video is increased if the processing time of a particular previous frame is longer than a certain period.

5. The method of claim 3 , wherein thread allocation is based on a weight of a previous frame, the weight being based on a picture order count distance, a temporal tier, and a frame type of the previous frame.

6. 1. An apparatus comprising: A device according to claim 2; at least one of (i) an antenna configured to receive a signal including a video block; (ii) a band limiter configured to limit the received signal to a frequency band including the video block; and (iii) a display configured to display an output representing the video block; A device comprising:

7. A computer program comprising instructions for causing a computer to perform the method of claim 1.

Images