Method and apparatus for performing real-time vvc decoding

Abstract

Apparatuses and methods for implementing real-time versatile video coding (VVC) decoder use multiple threads to address limitations in existing parallelization techniques and fully utilize available CPU computing resources without impacting encoding efficiency. The proposed multi-thread (MT) framework uses CTU-level parallel processing techniques without impacting memory bandwidth. Image-level parallel processing divides a sequence into temporal levels by considering the reference hierarchy of images. Implementations are provided using various optimization techniques to achieve real-time VVC decoding on a heterogeneous platform with a multi-core CPU for those bitstreams generated using the VVC reference encoder with a default configuration.

Description

Technical Field

[0001] At least one embodiment of this implementation relates in general to a method or apparatus for video decoding and / or compression. Background Technology

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transform, including motion vector prediction, to utilize spatial and temporal redundancy in the video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image (usually represented as prediction error or prediction residual) is transformed, quantized, and entropy-coded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy coding, quantization, transform, and prediction. Summary of the Invention

[0003] At least one embodiment of this implementation relates in general to a method or apparatus for performing real-time decoding, as in the VVC (Multi-Functional Video Coding or H.266) standard.

[0004] According to a first aspect, a method is provided. The method includes the step of allocating threads to perform CABAC decoding on video frames using parallel processing, wherein a higher number of threads are allocated for decoding frames with low quantization parameters compared to decoding frames with higher quantization parameters, a higher number of threads are allocated for intra-frame decoding compared to inter-frame decoding, or a higher number of threads are allocated for P-frames, or a higher number of threads are allocated when the frame is at a lower time level; and the video frames are decoded using the allocated threads.

[0005] According to the second aspect, another method is provided. This method includes steps for generating motion vectors for coding units; performing inter-frame coding unit reconstruction; performing intra-frame coding unit reconstruction; and performing inverse reshaping and loop filtering of the video, wherein the generation and execution steps are performed simultaneously on different portions of the video.

[0006] According to another aspect, an apparatus is provided. The apparatus includes a device comprising: a memory; and one or more processors configured to allocate threads to perform CABAC decoding of video frames using parallel processing, wherein a higher number of threads are allocated for decoding frames with low quantization parameters compared to decoding frames with higher quantization parameters, a higher number of threads are allocated for intra-frame decoding compared to inter-frame decoding, or a higher number of threads are allocated for P-frames, or a higher number of threads are allocated when the frame is at a lower time level; and the allocated threads are used to decode the video frames.

[0007] According to another aspect, a different apparatus is provided. The apparatus includes a device comprising: a memory; and one or more processors configured to generate motion vectors for coding units; perform inter-frame coding unit reconstruction; perform intra-frame coding unit reconstruction; and perform inverse reshaping and loop filtering of the video, wherein the device is configured to generate and perform the above simultaneously on different portions of the video.

[0008] According to another general aspect of at least one embodiment, an apparatus is provided comprising: means according to any embodiment of the decoding embodiment; and at least one of: (i) an antenna configured to receive a signal comprising a tensor of a video block and a feature map; (ii) a bandwidth limiter configured to limit the received signal to a bandwidth including the video block; and (iii) a display configured to display the output of any receiving device representing the video block or analysis feature / decoded content.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium containing data content decoded according to any embodiment or variant of the described encoding embodiment or variant.

[0010] According to another general aspect of at least one embodiment, a signal is provided that includes video data decoded according to any embodiment or variant of the described encoding embodiment or variant.

[0011] According to another general aspect of at least one implementation scheme, the bitstream is formatted to include data content generated according to any of the described encoding implementation schemes or variations.

[0012] According to another general aspect of at least one embodiment, a computer program product is provided, the computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described coded embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments read in conjunction with the accompanying drawings. Attached Figure Description

[0014] Figure 1 A general diagram of a block-based video encoder for VVC is shown.

[0015] Figure 2Block partitioning in various tree structures is shown: (a) quadrilateral partitioning; (b) vertical binary partitioning; (c) horizontal binary partitioning; (d) vertical ternary partitioning; (e) horizontal ternary partitioning.

[0016] Figure 3 A general block diagram of the video decoder used in VVC is shown.

[0017] Figure 4 An exemplary overview of the slice / image decoding stage is shown.

[0018] Figure 5 An example of CABAC decoding using WPP parallelization is shown.

[0019] Figure 6 An exemplary thread scheduling mechanism for the slicing / image reconstruction stage is shown.

[0020] Figure 7 The generation of motion vectors for coding units using WPP parallelization is shown.

[0021] Figure 8 An example of exemplary parallelization within the CUMV decoding stage is shown.

[0022] Figure 9 Parallelization within the inter-frame coding unit reconstruction and decoding stage is illustrated.

[0023] Figure 10 An example of parallel reconstruction of separate intra-frame coding units is shown.

[0024] Figure 11 This demonstrates the joint parallelization of the reconstruction and decoding stages of inter-frame coding units and intra-frame coding units.

[0025] Figure 12 The interleaving process is shown in the horizontal and vertical deblocking filtering decoding stages.

[0026] Figure 13 The interleaving process is shown in the coding phases of motion vector generation of coding units and reconstruction coding of inter-frame coding units.

[0027] Figure 14 The interleaving process during the reconstruction and decoding stages of inter-frame coding units and intra-frame coding units is shown.

[0028] Figure 15 The interleaving process is shown in the inter-frame and intra-frame coding unit reconstruction and inverse reshaping decoding stages.

[0029] Figure 16a and Figure 16b The diagram shows a) an exemplary communication system 100 in which one or more of the disclosed embodiments may be implemented, and b) a system diagram of an exemplary WTRU.

[0030] Figure 17 An implementation of the method in the general aspects described is shown.

[0031] Figure 18 A second embodiment of the method in the general aspects described is shown.

[0032] Figure 19 An apparatus for performing any of the methods described is shown. Detailed Implementation

[0033] Video coding systems are widely used to compress digital video signals to reduce the storage requirements and / or transmission bandwidth of such signals. Among the various types of video coding systems (such as block-based, wavelet-based, and object-based systems), block-based hybrid video coding systems are currently the most widely used and deployed. Examples of block-based video coding systems include international video coding standards such as MPEG1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, and the more recent video coding standard called High Efficiency Video Coding (HEVC), developed by the JCT-VC (Joint Collaboration Group on Video Coding) of ITU-T / SG16 / Q.6 / VCEG and ISO / IEC / MPEG.

[0034] The first version of the HEVC standard was completed in October 2013, offering approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard, H.264 / MPEG AVC. Despite the significant coding improvements offered by HEVC over its predecessor, evidence suggests that superior coding efficiency can be achieved with additional coding tools. Based on this, both VCEG and MPEG began exploring new coding technologies for future video coding standardization. In October 2015, the Joint Video Exploration Group (JVET) was established by ITU-T VECG and ISO / IEC MPEG to initiate significant research into advanced technologies that could greatly improve coding efficiency. JVET maintains a reference software called the Joint Exploration Model (JEM) by integrating several additional coding tools on top of the HEVC Test Model (HM).

[0035] In October 2017, the ITU-T and ISO / IEC issued a joint call for proposals (CfP) on video compression with capabilities exceeding HEVC. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, demonstrating compression efficiency improvements of approximately 40% over HEVC. Based on these evaluation results, JVET launched a new project to develop a next-generation video coding standard called Multifunctional Video Coding (VVC). That same month, a reference software codebase called the VVC Test Model (VTM) was established to demonstrate reference implementations of the VVC standard. Simultaneously, another reference software library called the Baseline Set (BMS) was generated to facilitate the evaluation of new coding tools. In addition to the VTM, the BMS codebase includes an additional list of coding tools that offer higher coding efficiency and moderate implementation complexity, and is used as a benchmark when evaluating similar coding techniques during the VVC standardization process. Specifically, in addition to JEM coding tools, such as 4x4 Inseparable Quadratic Transform (NSST), Generalized Bidirectional Prediction (GBi), Bidirectional Optical Flow (BIO), Decoder-End Motion Vector Refinement (DMVR), and Current Image Reference (CPR) integrated into BMS-2.0, it includes grid-coded quantization tools.

[0036] Similar to HEVC, VVC is built on a block-based hybrid video coding framework. Figure 1 A block diagram of a block-based general hybrid video coding system is given. The input video signal 102 is processed block by block (called a coding unit (CU)). In VTM-1.0, a CU can be up to 128×128 pixels. However, unlike HEVC, which partitions blocks solely based on quadtrees, in VTM-1.0, a coding tree unit (CTU) is divided into CUs to accommodate the varying local characteristics based on quadtrees / binary / tritrees. Furthermore, the concept of multiple partitioned unit types is removed from HEVC; that is, the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC. Instead, each CU is always used as the basic unit for both prediction and transform, without further partitioning. In the multi-type tree structure, a CTU is first partitioned using a quadtree structure. Then, each quadtree leaf node can be further partitioned using binary and ternary tree structures. Figure 2 As shown, there are five partition types: quad partition, horizontal binary partition, vertical binary partition, horizontal ternary partition, and vertical ternary partition. Figure 1In the encoder, spatial prediction (160) and / or temporal prediction (162) can be performed. Spatial prediction (or “intra-frame prediction”) uses pixels from samples (called reference samples) of neighboring blocks encoded in the same video image / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also called “inter-frame prediction” or “motion-compensated prediction”) uses reconstructed pixels from encoded video images to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs) indicating the magnitude and direction of motion between the current CU and its temporal reference. Moreover, if multiple reference images are supported, an additional reference image index is sent to identify which reference image in the reference image repository (164) the temporal prediction signal comes from. After spatial and / or temporal prediction, a mode decision block (180) in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block (116); and the prediction residual is decorrelated using a transform (104) and quantization (106). The quantized residual coefficients are inversely quantized (110) and inversely transformed (112) to form a reconstruction residual, which can then be added back to the prediction block (126) to form the reconstructed signal of the CU. Further loop filtering, such as deblocking filtering, sample adaptive offset (SAO), and adaptive loop filtering (ALF), can be applied (166) to reconstruct the CU before being placed in the reference image repository (164) and used to encode future video blocks. To form the output video bitstream 120, the coding mode (inter-frame or intra-frame), prediction mode information, motion information, and quantized residual coefficients are sent to the entropy coding unit (108) for further compression and packing to form the bitstream.

[0037] In the inter-frame prediction and loop filtering stages of VVC, the following tools are new compared to HEVC.

[0038] 1) Inter-frame / Intra-frame Joint Prediction (CIIP) 2) Decoder-side motion vector refinement (DMVR) 3) Reverse reshaping CIIP mode combines inter-frame prediction signals with intra-frame prediction signals. A weighted average is used to combine the inter-frame and intra-frame prediction signals. The weights are based on the coding patterns of the left and upper blocks.

[0039] In VVC, each CU signals an additional tag for use in inter-frame / intra-frame joint prediction (CIIP) mode if the following conditions are met.

[0040] • When encoding CU in merge mode • The CU contains more than 64 luminance samples, meaning the product of the CU width and CU height is equal to or greater than 64. • The CU height and CU width are less than 128 luminance samples.

[0041] VVC allows for increased MV accuracy using bidirectional matching-based motion vector refinement at the decoder, also known as decoder-side motion vector refinement (DMVR). Template matching is performed at the decoder to refine the MV around the initial MV within the reference image lists L0 and L1. SAD is computed between each refined MV candidate in raster scan order within the search range (25-point full search of integer samples) around the initial MV. The MV candidate with the lowest SAD is used to generate the bidirectional prediction signal.

[0042] In VVC, DMVR can be applied to each CU that uses the following features.

[0043] • CU-level merging mode with bidirectional predictive MV • Relative to the current image, one reference image is from the past, and the other reference image is from the future. • The point of view (POC) distance between the reference image and the current image is the same. • The CU contains more than 64 luminance samples. • Both the CU height and CU width are greater than or equal to 8 luminance samples. • Bidirectional prediction (BCW) weight index with CU-level weights indicates equal weights • Weighted prediction (WP) is not enabled for the current block.

[0044] Luminance Mapping with Chroma Scaling (LMCS) is added as a new tool to VVC, which can be enabled / disabled at the sequence level using Sequence Parameter Set (SPS) flags. LMCS is applied immediately before the loop filtering stage. For inter-coded blocks, motion-compensated prediction is applied in the mapped domain; that is, a forward mapping function is applied to the luminance prediction block in the original domain to transform it into the mapped domain. For intra-coded blocks, no forward mapping is applied.

[0045] LMCS consists of two parts: 1. Luminance mapping with a piecewise linear model, and 2. Luminance-dependent chroma scaling.

[0046] 1. Luminance mapping with a piecewise linear model At the tile group level, a signal is sent to the brightness mapping model. If a marker is detected in the brightness mapping model for the current tile group, the corresponding piecewise linear model parameters are signaled. The signal codeword is used to calculate the scaling factor and adjust the mapping function for each of the 16 equal segments.

[0047] 2. Luminance-dependent chromaticity residual scaling Chroma residual scaling is also signaled at the tile group level. If luma mapping is enabled and if dual-tree partitioning is not applied to the current tile group, an additional marker is signaled to indicate the use of luma-dependent chroma residual scaling. Luma-dependent chroma residual scaling is disabled for chroma blocks with a region of 4 or less. The purpose of using chroma residual scaling is to compensate for the interaction between the luma and chroma signals.

[0048] Figure 3 A general block diagram of the block-based video decoder is shown. The video bitstream 202 is first unpacked and entropy decoded at entropy decoding unit 208. The coding mode and prediction information are sent to spatial prediction unit 260 (if intra-frame coded) or temporal prediction unit 262 (if inter-frame coded) to form prediction blocks. Residual transform coefficients are sent to inverse quantization unit 210 and inverse transform unit 212 to reconstruct residual blocks. The prediction blocks and residual blocks are then added together at 226. The reconstructed blocks may undergo further loop filtering before being stored in reference image repository 264. The reconstructed video in the reference image repository is then sent out to drive the display device, along with video blocks for predicting future frames.

[0049] Prior to VVC implementation on CPU, GPU, or heterogeneous platforms, several published papers had already leveraged the parallelization capabilities of video codecs. Another approach proposed a HEVC deblocking filter that utilizes independent regions of frames and reduces overall memory access. In other methods, GPUs are used to accelerate the HEVC decoding stages of dequantization, inverse transform, intra-frame prediction, deblocking filtering, and SAO. Yet another approach proposes a GPU-based parallel algorithm, including parallel computation for sample classification, statistical information collection for each coding tree block, parallel computation of optimal offsets, and minimum distortion for each class of edge offsets and band offsets, SAO merging, and SAO filtering.

[0050] Frame-level parallelism involves processing multiple frames while satisfying motion compensation dependencies. One of the main limitations of frame-level parallelism is that its parallelism is determined by the length of the motion vectors. This is a major bottleneck, especially for sequences with large motions.

[0051] Slice-level parallelism allows slices to be processed independently of each other. The main drawback of this level of parallelism is that the number of slices is determined by the encoder. Furthermore, in most cases, the codec can limit itself to using only one slice per frame, resulting in virtually no slice-level parallelism. Additionally, slicing reduces coding efficiency.

[0052] The aforementioned drawbacks of frame-level parallelism or slice-level parallelism can be overcome by employing wavefront parallel processing (WPP) and tile-based methods. Both techniques allow each image to be divided into multiple partitions that can be processed in parallel. Each partition contains an integer number of coding units that are independent of other partitions. The use of tiles reduces coding efficiency and complicates raster scan processing. However, the main drawback of tile-based methods is the significant rate-distortion loss with a higher number of blocks due to the lack of dependencies between partitions.

[0053] Wavefront parallel processing (WPP) allows image partitioning without compromising coding dependencies or resetting CABAC probabilities in parallel processing using slices or tiles. The WPP method partitions an image into CTU rows and allows prediction and entropy coding across CTU row boundaries. For this, WPP reduces the loss of compression efficiency compared to slice and tile parallel processing methods. However, wavefront dependencies do not allow all CTB rows to start decoding simultaneously. Therefore, the row will not complete decoding concurrently. This makes parallelization inefficient when using a high number of WPP threads for decoding.

[0054] Overlapping Wavefront (OWF) parallelism improves the implementation efficiency of WPP by enabling the execution of overlapping of consecutive images. The dependency of the OWF technique stems from motion search. In OWF, the CTU is not ready to decode until all reference pixels within the motion search region in the reference image have been decoded. This motion search dependency limits the throughput of the frame-level parallel processing threads used to encode multiple frames. This problem is exacerbated when the center of the search window (determined by the motion predictor) is positioned towards the lower part of the reference image. On the other hand, restricting motion vectors leads to significant video coding losses, manifested in rapid vertical motion.

[0055] Therefore, no single parallelization technique can fully utilize the CPU processing power available on today's multi-core systems.

[0056] The described implementation addresses the limitations of existing parallelization techniques and fully utilizes available CPU computing resources without compromising coding efficiency. The proposed multithreaded (MT) framework employs CTU-level parallel processing techniques without impacting memory bandwidth. Image-level parallel processing divides the sequence into temporal levels by considering the reference hierarchy of the image. This disclosure discusses various optimization techniques for achieving real-time VVC decoding on heterogeneous platforms with multi-core CPUs for bitstreams generated using a VVC reference encoder with default configurations. Equivalent techniques proposed for the decoder can also be used with the VVC encoder framework.

[0057] The current VVC draft includes several aspects for parallelizing the coding process. These include tile-based parallel processing (WPP). Using tiles for parallelism incurs coding loss because there are no dependencies between blocks. WPP divides slices into CTU rows and processes the rows in parallel while preserving coding dependencies.

[0058] The current implementation proposes a finer granularity of parallelism without compromising coding efficiency. The section titled "Parallelization of CABAC with Slice Decoding Stage" presents this finer granularity of parallelism achieved by handling CABAC decoding, with the remainder of the slice decoding stage referred to as the reconstruction decoding stage. Figure 4 The diagram shows the architecture of the slice / image decoding stage. Parallelization of CABAC with a slice decoding stage is described in the section titled "Parallelization of CABAC with a Slice Decoding Stage". Figure 4 The parallelization of each reconstruction decoding stage, such as inter-frame, intra-frame, reshaping, and loop filtering, is shown in the reconstruction decoding stages.

[0059] In summary, the key contributions to improving the finer parallelism granularity of pipelined multi-stage VVC decoders are as follows: • In the intra-frame CU reconstruction phase, once the inter-frame CU (left block, top block, and top right block) reconstruction process of the reference pixels of all separate / independent intra-frame CTUs (CTUs that encode all their intra-frame coded CU reference pixels in inter-frame mode) is completed, all separate / independent intra-frame CTUs can be reconstructed in parallel. • Inter-frame CU and inter-frame prediction in the Inter-frame / Intra-frame Joint Prediction (CIIP) CU prediction process can be parallelized across all CTUs of an image. • To balance computational complexity, WPP-enabled CABAC decoding can be performed in parallel with the reconstruction decoding phase. The number of threads initially allocated for CABAC decoding of the image varies dynamically based on the CABAC performance of previous images; • Multiple stages of the decoder pipeline can be executed in parallel within an image and between images at different time levels. For example, the CUMV, inter-frame CU reconstruction, and intra-frame CU reconstruction decoding stages can be executed in parallel within an image or between images; • Pre-analysis is performed to locate the first relevant CU in each CTU and this information is used to improve the parallelization of wavefront parallel processing (WPP) at the CUMV and intra-frame CU reconstruction stages; • To avoid multiple image scans, deblocking filtering is parallelized by interleaving the horizontal and vertical filtering stages. The number of horizontal and vertical edges within and between CTUs is calculated to distribute the filtering workload evenly across threads; • Add additional conditions to adaptively switch DMVR. DMVR is used to merge candidates only when the length of the motion vector is less than a predefined threshold. Disabling DMVR based on the length of the motion vector makes the reference area available and increases DMVR parallelism.

[0060] Parallelization methods for the VVC decoding stage can be categorized into one of the following three groups. The group is chosen for each decoding stage based on the decoding dependencies between CTUs. These are: I. CTU-level parallelization (CTUP); II. Improved parallelization of overlapping wavefronts (IOWF); III. Parallelization based on load balancing.

[0061] Method I is used to select VVC decoding stages that do not have CTU-level dependencies, such as reshaping and Sample Adaptive Offset (SAO). Method II is used to select VVC decoding stages that have CTU-level dependencies on intra-frame CU prediction.

[0062] Pipeline design method I adds fine-grained parallelism to existing tile-based methods. It is based on flexibly partitioning the image into CTUs, thereby eliminating dependencies between multiple CTU partitions. The number of CUs processed per thread varies dynamically based on the QTBT partitions.

[0063] Method II of the pipeline design addresses the inefficiency of wavefront parallel processing (WPP) by following an improved Overlapping Wavefront (IOWF) approach. The IOWF proposed in these implementations allows for the overlapping of the execution of decoding stages within consecutive images, regions, such as CUs within a CTU of an image, and within the image itself using wavefronts. The section below, titled "Parallelization of CABAC with Sliced ​​Decoding Stages," highlights the pipeline design based on Method II.

[0064] Pipeline design method III parallelizes modules that are difficult to parallelize due to high branch divergence and low data parallelism. CABAC decoding belongs to this category. Therefore, CABAC decoding is performed in parallel with the slice reconstruction decoding stage, as explained in the section titled "Parallelization of CABAC with Slice Decoding Stage".

[0065] Parallelization of CABAC with Slice Decoding Stage Parallelization of CABAC with a slice decoding stage is based on the proposed Method III. One of the main causes of VVC decoder latency is the data dependency of context selection for the CABAC engine. This is primarily because the content selection used for slices depends on the values ​​of previously decoded slices. This dependency is a bottleneck for achieving CABAC parallelism, especially at the decoder. This, in turn, increases decoder latency. This latency can be reduced if the CABAC decoding of an image is performed in parallel with the reconstruction decoding stages of other images.

[0066] Load balancing between the CABAC and reconstruction / decoding stages consists of the following stages.

[0067] I. Thread Priority Scheduling CABAC decoding complexity determines thread execution priority. For example, intra-frame slices have higher CABAC decoding complexity compared to other slices. Therefore, intra-frame slice CABAC decoding activities can be scheduled earlier in the decoding order than their preceding images. Similarly, the complexity of CABAC decoding for time layer 0 images is higher than that for time layer 1, etc. Therefore, CABAC decoding of lower time layer images in the next group of images (GOP) can be scheduled in parallel with CABAC decoding of higher time layer images in the previous GOP. In this way, fluctuations in CABAC decoding time can be balanced across time layer images. Therefore, thread scheduling is prioritized based on the following factors.

[0068] • Early scheduling of the CABAC decoding process for images based on the estimated complexity of frames / slices.

[0069] II. Thread allocation for CABAC decoding with WPP enabled like Figure 5 As shown, the initial number of threads allocated for WPP-enabled CABAC decoding of an image depends on the frame decoding complexity. If the frame decoding complexity is high, such as for low QP, intra-frame, inter-frame images, or lower temporal layers, more threads are allocated for CABAC decoding. Once multiple threads have been allocated for CABAC decoding, the remaining available threads are allocated to the reconstruction decoding stage of the previous image. The total number of available threads is distributed between the CABAC and reconstruction decoding stages in the following manner.

[0070] • Allocate a higher number of threads for CABAC decoding of frames with low QP compared to frames with high QP, a higher number of threads for intra-frame decoding compared to inter-frame decoding, a higher number of threads for P-images, or a higher number of threads when the image is in a lower time layer.

[0071] • A higher number of threads are allocated for CABAC decoding of images with multiple tiles and slices compared to images with a single tile or a single slice.

[0072] III. Dynamic Changes in Allocation The number of threads initially allocated for WPP-enabled CABAC decoding of the image varies dynamically based on the performance of that initial CABAC allocation. The latency between the initial CABAC allocation and the reconstruction decoding phase can be used as a thread allocation criterion. The goal of dynamically varying the number of threads allocated to the CABAC and reconstruction decoding phases is to reduce the fluctuation in CABAC processing time per frame.

[0073] • If processing a similar previous frame takes longer than the real-time constraint, the number of threads allocated for CABAC decoding of the current frame is increased.

[0074] • If the waiting time between the CABAC decoding and reconstruction decoding stages is high, the number of threads allocated for CABAC decoding of the current frame is reduced.

[0075] • One of the multiple previous frames can be used to determine the dynamic changes in the assignment.

[0076] • The weights of previous frames in the dynamic allocation of threads used to determine the current frame vary based on POC distance, time layer, and frame type.

[0077] Parallelization of the slice decoding stage Methods I and II are used to parallelize the various decoding stages after CABAC decoding. The pipeline design follows the principle of overlapping execution of multiple decoding blocks that can be processed in parallel. If a thread has completed the execution of a block, it will continue to execute the next available block. Parallel processing partitions the image into CTUs or CTU rows without affecting dependencies. Such overlapping execution can occur within a decoder module or between modules. This is described in detail in the respective subsections.

[0078] Thread scheduling This section explains Figure 6The thread scheduling mechanism shown below is used for the CUMV phases, which are described in detail in the "CUMV Generation" section. A thread pool and a first-in-first-out (FIFO) activity queue are created to handle the reconstruction decoding phase activities. The thread pool is created based on the number of available CPU cores in the heterogeneous multi-core system. Any free thread from the thread pool takes the first available task from the activity queue and executes the activity using CPU capacity. The slice decoding thread pushes a list of tasks to be executed to decode slices / images into the activity queue. The slice decoding thread initially pushes CU motion vector generation for the first CTU line of tasks, where the dependency flag is set to false, and CU motion vector generation for the second CTU line of tasks, where the dependency flag is set to true, until the last CTU line. A free worker thread, called the first worker thread, takes the first available activity from the task queue and executes the CU motion vector generation for the first CTU, continuing without using any dependencies until the last CTU in the first line. Another free worker thread takes the next activity from the queue (CU motion vector generation for the second CTU line) and processes the activity in parallel with the first worker thread, using WPP dependencies. This process continues throughout all slice / image decoding phases.

[0079] GPU parallel processing The parallelization schemes proposed in the sections titled "Parallelization within the Decoding Stage" and "Parallelization between Decoding Stages" are also applicable to GPU parallelization schemes. The VVC decoding architecture is modified to reduce GPU memory usage, such as global, cached, or constant memory usage, and to reduce GPU memory accesses. Instead of SIMD optimizations used for CPU parallelization, GPU vector instructions are utilized to increase parallelism. Furthermore, GPU kernel startup is completed immediately after the file read operation. This eliminates performance limitations caused by memory transfers.

[0080] Parallelization within the decoding phase The current VVC design processes the decoding stage of each CTU sequentially. This design is not parallelization-friendly. To improve the parallelism of the VVC decoder stages, it is recommended to divide the decoding stage of each image into the following sub-tasks.

[0081] 1. CU Motion Vector (CUMV) Generation 2. Inter-frame CU reconstruction a. Inter-frame and intra-frame joint prediction (CIIP) b. Decoder-side motion vector refinement (DMVR) 3. Intra-frame CU reconstruction 4. Reverse reshaping 5. Loop Filtering a. Deblocking filtering b. Adaptive loop filtering c. Sample adaptive offset.

[0082] CUMV generation Before proceeding with the reconstruction process, the CU motion vector derivation is performed for the entire inter-frame slice / image. CU motion vector derivation can depend on the motion vectors of its adjacent CUs to its left, above, and / or upper right; this is known as wavefront parallel processing (WPP) dependency. For better parallel processing capabilities, each CTU row is pushed into the activity queue in ascending order of CTU number. For example... Figure 7 As shown, free worker threads from the thread pool will retrieve the first available CTU row from the active queue and generate motion vectors for the CTUs sequentially until the entire CTU row is completed. Each worker thread ensures that the CTU to the right of the current CTU has always completed generating motion vectors before continuing to generate motion vectors for the current CTU. In this way, a maximum of H (number of CTU rows) worker threads can be used for parallel processing of motion vector generation for the entire image. Figure 8 The derivation process of the CUMV motion vector is shown. Figure 8 The variables used are explained in Table 1.

[0083] Table 1. Definitions of CUMV variables variable definition Module (i) The i-th block in the CUMV motion derivation process <![CDATA[NT max ]]> Maximum number of threads for the CUMV module NT Number of threads allocated for the CUMV module

[0084] The steps for deriving the CUMV motion vector can be summarized as follows.

[0085] • Pre-analysis to determine the location of the first CU in each CTU, which depends on the CTU to its upper right. • The CU at the first CU position in the current CTU, as in step 1, can be processed in parallel with the CTU to its upper right. • The remaining CUs of each CTU will require the availability of the relevant CU in the upper right corner, rather than the availability of the entire upper right CTU.

[0086] Inter-frame CU reconstruction After the derivation of the CU motion vectors for the entire image, the inter-frame CU reconstruction process has been separated from the intra-frame and CIIP mode reconstruction processes. Figure 9 The inter-frame CU reconstruction algorithm is described in detail. Figure 9 The variables used are explained in Table 2.

[0087] Table 2. Definition of Inter-Frame CU Reconstruction Variables variable definition InterCU(i) The i-th interCU reconstruction and decoding stage <![CDATA[NT max ]]> Maximum number of threads for a module NT Number of currently working threads on the module

[0088] The remainder of the decoding thread pushes the inter-frame CU reconstruction process of a CTU or CTU group into the activity queue in ascending order of CTU number. Free worker threads from the thread pool retrieve the first available CTU or CTU group from the activity queue and execute the inter-frame CU reconstruction process for the entire CTU or CTU group (one CTU after another). All worker threads can utilize multiple CPU / GPU cores available in the system to execute the inter-frame CU reconstruction process of their CTU groups in parallel. This significantly reduces the complexity of the entire slice / image inter-frame CU reconstruction.

[0089] I. CIIP When a CU is encoded in merged mode, where the number of luma samples is greater than 64 and both the CU width and CU height are less than or equal to 128, an additional marker signals the use of the inter-frame / intra-frame CU joint prediction (CIIP) mode. Due to the inter-frame and intra-frame joint nature of the current CU, the inter-frame prediction of the CIIP mode CU can be combined with other inter-frame CU prediction processes. The inter-frame CU and CIIP inter-frame CU prediction processes can be parallelized across all CTUs of the image. This allows for a significant reduction in the CIIP inter-frame CU reconstruction process time by using multiple worker threads for parallel processing. The CIIP intra-frame CU reconstruction process can be combined with other intra-frame CU reconstruction processes.

[0090] II. DMVR VVC allows obtaining refined motion vectors without sending them, thus reducing the encoder's computational complexity. However, DMVR increases the decoder's computational workload and processing time. In DMVR, computational complexity increases due to the search scheme around the search center, the metrics used to select refinement, such as the sum of absolute differences (SAD), and bilinear interpolation used to insert fractional pixel locations.

[0091] As the size of the motion vector (MV) is constrained in the encoder or decoder, the number of reference regions required to compute the DMVR output of the current CU is reduced. Therefore, limiting the number of search points within the region or the size of the MV will increase DMVR parallelism at the cost of describing rapid motion.

[0092] DMVR can be adaptively turned on and off without signaling overhead. The MV refinement process operates only when the following conditions are met.

[0093] • The encoding unit is a merged mode • One-way, ATMVP, STMVP, affine, and merge candidates using the IC pattern were excluded. • (POC ref0 – POC cur ) and (POC ref1 – POCcur Only one value in the ) is negative.

[0094] In at least one of the embodiments described, an appendix condition is added to use DMVR only when the length of the motion vector is less than a predefined threshold. Disabling DMVR based on the length of the motion vector makes the reference region available and increases DMVR parallelism for merging predicted candidates.

[0095] Intra-frame CU reconstruction Traditional WPP parallelization of intra-frame CU prediction treats each CTU line as an independent region. At the CTU level, each CTU must wait until its left and right-top adjacent CTUs have completed reconstruction. This process causes the current CTU line to always lag behind its adjacent previous line by two CTUs. To mitigate this problem, the general approach described in this paper proposes the following stages for intra-frame CU prediction parallelization.

[0096] I. Pre-analysis • The pre-analysis phase determines the location of the first relevant CU for each CTU, dependent on the upper right CTU. • Find the number of separate CTUs in the image. A separate CTU is a CTU in which all of its intra-frame coded CU reference pixels are encoded in inter-frame mode.

[0097] II. Parallelization Scheme • Based on the above pre-analysis results, the CU at the first relevant CU position in the current CTU can be processed in parallel with the CTU to its upper right.

[0098] • Once the inter-frame CU (left, top, and top right blocks) reconstruction process for its reference pixel is complete, all separate intra-frame CUs can be reconstructed in parallel. A free worker thread can process a constant number of separate intra-frame CUs. As more separate intra-frame coded CUs become available, each free worker thread can process a constant number of separate CUs in parallel with other worker threads.

[0099] • Separate intra-coded CTUs in the upper row will have higher priority. Each separate CTU in the upper row will be processed by a free worker thread. Using this thread scheduling method, the intra-CU reconstruction process in the upper row can be completed faster, and the waiting time for the next few CTU rows can be significantly reduced. Figure 10 As shown, after the CU reconstruction process of each separate CTU in the first row is completed, the CTU is processed by the free worker thread.

[0100] • The WPP thread performs the intra-CU reconstruction process for the first CTU line sequentially for the remaining CTUs. This WPP thread runs in parallel with the aforementioned separate CTU reconstruction process.

[0101] Deblocking Filtering Deblocking is the module that follows inverse reshaping in the VVC loop filtering stage. In VVC, the vertical deblocking filtering stage is based on the output of the horizontal deblocking filtering stage. Therefore, the current design processes the horizontal filtering stage and the vertical filtering stage sequentially. However, there is no data dependency between CTUs for either the horizontal or vertical filtering stage. Therefore, all CTUs of the entire frame can be processed in parallel. The "Interleaving and Grouping" section below proposes techniques for interleaving the horizontal and vertical filtering stages and for mitigating the increase in memory access due to CTU-level parallelization.

[0102] IV. Edge-based CTU grouping In VVC, the maximum allowed CTU size is 128×128. Therefore, for deblocking performed on an 8×8 grid, the maximum allowed number of vertical edges is 128 / 8 = 16. However, due to QTBTTT partitioning, the number of vertical and horizontal edges can vary within and between CTUs. Therefore, to ensure a uniform distribution of processing load across threads, it is handled by assigning tasks to each thread based on a predefined number of edges.

[0103] V. CTU grouping based on Region of Interest (ROI) Instead of grouping consecutive rows of CTUs used for thread allocation, a region of interest (ROI) can be selected for each thread processing. Grouping CTUs used for DBF processing based on ROIs results in better memory management.

[0104] VI. Interleaved Grouping Existing VVC designs process the horizontal and vertical filtering stages sequentially. This requires multiple image scans and increases memory access. Instead of processing the entire image sequentially, a method is proposed here that interleaves the horizontal and vertical filtering stages. Horizontal filtering is performed on two CTU rows in the first pass, followed by vertical filtering. This ensures the availability of horizontally filtered pixels for the current CTU row and the CTU rows below it before vertical filtering begins. Figure 12 The joint parallelization of the horizontal deblocking filtering decoding stage and the vertical deblocking filtering decoding stage is shown. The variables used are explained in Table 3.

[0105] Table 3. Variable definitions for interleaved horizontal and vertical deblocking filters variable definition HORDBF(i) The i-th level deblocking filtering and decoding stage VERDBF(i) The i-th vertical deblocking filtering and decoding stage <![CDATA[NT max ]]> Maximum number of threads for a module NT Number of currently working threads on the module

[0106] Parallelization between decoding stages Parallelization between decoding stages is based on Method I. The data structures and dependencies between video encoding stages limit the possibility of parallelism within and between stages. For example, decoding modules such as CABAC exhibit high branch divergence and low data parallelism. Similarly, modules with low branch divergence and high data parallelism, such as loop filtering, are good candidates for parallel processing. The proposed parallelization technique performs such an analysis of the VVC decoder based on the criteria of branch divergence and data parallelism to determine the effectiveness of parallelizing one module with another. The steps for joint parallelization of VVC modules, which simultaneously allows for low branch divergence and high data divergence, are explained below.

[0107] Parallel processing of time-layer images Both decoding frame rate and latency can be reduced by parallel decoding of temporal image layers. For example, a new reference image occupies a lower temporal layer compared to a non-reference image, and vice versa. To process consecutive frames in parallel, the following stages are executed sequentially: • Complete the prediction process for the juxtaposed CU, bottom CU, and top right CU in the reference image.

[0108] • After the CU prediction process is completed below and to the right, inverse reshaping and deblocking of the juxtaposed CUs in the reference image is initiated.

[0109] • Apply inverse reshaping and loop filtering to the juxtaposed CUs in parallel with other CU prediction processes in the reference image.

[0110] • Generate a prediction buffer for the current CU.

[0111] Parallel processing of multiple decoding modules in an image To achieve efficient parallelization of multiple decoding modules, two factors were considered: high branch divergence and the percentage of total decoding time. Based on graph analysis, inter-frame CU prediction and loop filtering stages accounted for a very high percentage of the decoding time. On the other hand, intra-frame CU prediction modules exhibited high branch divergence. Parallel decoding of sub-stages within and between these modules was implemented to minimize the total decoding time.

[0112] I. CUMV Generation and Inter-Frame CU Reconstruction Parallelization between CUMV generation and inter-frame CU reconstruction follows Method II and uses the following steps.

[0113] • After CABAC decoding, CUMV generation of the entire image is performed in parallel after WPP dependencies. • Perform inter-frame CU reconstruction of the entire image in parallel. • Schedule inter-frame CU reconstruction of upper row CTUs in parallel with CUMV generation of other CTUs. After completing the upper CTU line CUMV generation process, those free worker threads can be allocated to the inter-frame CU reconstruction process, rather than other CTU line CUMV generation processes. Early allocation of worker threads to the lower line CUMV generation processes leads to resource waste because they have WPP dependencies on their upper CTU lines. Allocating those worker threads to the inter-frame CU reconstruction process will help to utilize resources efficiently.

[0114] • like Figure 13 As shown, the inter-frame CU reconstruction process in the upper CTU row has been initiated using multiple free worker threads, while the CU motion vector generation process is executed by the WPP worker thread used for the lower CTU row.

[0115] II. CUMV Generation, Intra-Frame and Inter-Frame CU Reconstruction Figure 11 The joint parallelization of the inter-frame CU reconstruction phase and the intra-frame CU reconstruction phase is illustrated in Table 4, and the associated parameters are explained in Table 4. Intra-frame CU reconstruction follows the inter-frame CU reconstruction process. However, the intra-frame CU reconstruction process of the upper-line CTU can be performed in parallel with the inter-frame CU reconstruction of other CTUs (e.g., later-line CTUs). This ensures the availability of inter-frame CU reconstruction samples while benefiting from the joint parallelization of inter-frame and intra-frame CU reconstruction.

[0116] like Figure 14 As shown, the motion vector generation, inter-frame reconstruction decoding, and intra-frame CU reconstruction decoding stages can overlap. When the WPP thread completes the CUMV generation for a specific CTU line, a free worker thread can be used for inter-frame CU reconstruction. Once the CU reconstruction of a predefined number of CTUs is completed, the free worker thread can begin executing the intra-frame CU reconstruction process for the first CTU line. By the time another free worker thread begins executing the intra-frame CU reconstruction process for the second CTU line, many CTUs in the first CTU line have already completed their intra-frame CU reconstruction processes. In this way, the latency in the intra-frame CU reconstruction process for the second CTU line caused by WPP dependency can be significantly minimized.

[0117] Table 4. Definition of variables for joint inter-frame and intra-frame CU reconstruction variable definition InterCU(i) The i-th inter-frame CU reconstruction and decoding stage IntraCU(i) The i-th intra-frame CU reconstruction and decoding stage <![CDATA[NT max ]]> Maximum number of threads for both modules NT Total number of threads working on the two modules

[0118] III. Inverse reshaping and intra-frame CU reconstruction Other CTU-level reverse reshaping processes and inter-frame and intra-frame CU reconstructions are executed in parallel. For example... Figure 15As shown, before the inter-frame and intra-frame CU reconstruction of the entire image is completed, a free worker thread can be used to schedule the reverse reshaping process of the upper CTU line. This early scheduling allows the entire upper line CTU reconstruction process, including deblocking, SAO, and ALF, to be completed before the inter-frame and intra-frame CU reconstruction of the entire image is finished. Once the upper CTU line reconstruction is complete, the next relevant image decoding can be started in parallel with the current image decoding.

[0119] Figure 16A is a diagram illustrating an exemplary communication system 100 that can be implemented in one or more of the disclosed embodiments. Communication system 100 can be a multiple access system providing content such as voice, data, video, messaging, and broadcasting to multiple wireless users. Communication system 100 enables multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, communication system 100 can employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0120] As shown in Figure 16A, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104, CN 106, Public Switched Telephone Network (PSTN) 108, Internet 110, and other networks 112. However, it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d can be any type of device configured to operate and / or communicate in a wireless environment. As an example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile user units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0121] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106, Internet 110, and / or other networks 112. As an example, base stations 114a and 114b may be base transceiver stations (BTS), Node Bs, evolved Node Bs, home Node Bs, home evolved Node Bs, gNBs, NR Node Bs, site controllers, access points (APs), wireless routers, etc. Although base stations 114a and 114b are each depicted as a single element, it should be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0122] Base station 114a may be part of RAN 104, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies (which may be referred to as cells (not shown)). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage of radio services to a specific geographic area, which may be relatively fixed or changeable over time. A cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0123] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0124] More specifically, as noted above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 may implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0125] In one implementation, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as evolved UMTS terrestrial radio access (E-UTRA), which can use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro) to establish air interface 116.

[0126] In one implementation, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can use New Radio (NR) to establish air interface 116.

[0127] In one implementation, base station 114a and WTRUs 102a, 102b, and 102c can implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can, for instance, use a dual connectivity (DC) principle to implement both LTE and NR radio access together. Therefore, the air interface used by WTRUs 102a, 102b, and 102c can be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0128] In other implementations, base station 114a and WTRUs 102a, 102b, and 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), GSM Enhanced Data Rate Evolution (EDGE), and GSM EDGE (GERAN).

[0129] Base station 114b in Figure 16A can be, for example, a wireless router, a home Node B, a home eNode B, or an access point, and can utilize any suitable RAT to facilitate wireless connectivity in localized areas such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for use by drones), roads, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. As shown in Figure 16A, base station 114b may have a direct connection to Internet 110. Therefore, base station 114b may not need to access Internet 110 via CN 106.

[0130] RAN 104 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, etc., and / or perform advanced security functions such as user authentication. Although not shown in Figure 16A, it should be understood that RAN 104 and / or CN 106 can communicate directly or indirectly with other RANs using the same RAT as RAN 104 or a different RAT. For example, in addition to connecting to RAN 104 which utilizes NR radio technology, CN 106 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0131] CN 106 may also act as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 or a different RAT.

[0132] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, the WTRU 102c shown in Figure 16A may be configured to communicate with a base station 114a that may employ cellular-based radio technology and with a base station 114b that may employ IEEE 802 radio technology.

[0133] Figure 16B is a system diagram illustrating an exemplary WTRU 102. As shown in Figure 16B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138, etc. It should be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with the implementation.

[0134] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 may perform signal encoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, which may be coupled to transmitting / receiving element 122. Although Figure 16B depicts processor 118 and transceiver 120 as separate components, it should be understood that processor 118 and transceiver 120 may be integrated together in an electronic package or on a chip.

[0135] Transmitting / receiving element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, transmitting / receiving element 122 may be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 may be configured to transmit and / or receive both RF and optical signals. It should be understood that transmitting / receiving element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0136] Although the transmitting / receiving element 122 is depicted as a single element in Figure 16B, the WTRU 102 may include any number of transmitting / receiving elements 122. More specifically, the WTRU 102 may employ MIMO technology. Therefore, in one embodiment, the WTRU 102 may include two or more transmitting / receiving elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via the air interface 116.

[0137] Transceiver 120 can be configured to modulate signals transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As noted above, WTRU 102 may have multi-mode capability. Therefore, transceiver 120 may include multiple transceivers to enable WTRU 102 to communicate via various RATs such as NR and IEEE 802.11.

[0138] The processor 118 of WTRU 102 may be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and may receive user input data therefrom. The processor 118 may also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Furthermore, the processor 118 may access information from any type of suitable memory (such as non-removable memory 130 and / or removable memory 132) and store data in any type of suitable memory. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a user identity module (SIM) card, memory stick, secure digital storage (SD) card, etc. In other embodiments, the processor 118 may access information from memory that is not physically located on WTRU 102 (such as on a server or home computer (not shown)) and store data in that memory.

[0139] The processor 118 may receive power from the power supply 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 may be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell battery packs (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0140] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of the information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that, while remaining consistent with the implementation, the WTRU 102 may acquire location information using any suitable location determination method.

[0141] The processor 118 may also be coupled to other peripheral devices 138, which may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, peripheral device 138 may include an accelerometer, electronic compass, satellite transceiver, digital camera (for photos and / or video), Universal Serial Bus (USB) port, vibration device, television transceiver, hands-free headset, Bluetooth.® Modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors; altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0142] WTRU 102 may include a full-duplex radio for which the transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference through signal processing via hardware (e.g., a choke) or via a processor (e.g., a separate processor (not shown) or via processor 118). In one embodiment, WTRU 102 may include a full-duplex radio for which the transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and / or simultaneous.

[0143] Although the WTRU is described as a wireless terminal in Figures 16A to 16B, it is conceivable that in some representative embodiments, such a terminal may (e.g., temporarily or permanently) use a wired communication interface with a communication network.

[0144] Given the corresponding descriptions in Figures 16A-16B and 16A-16B, one or more of the functions described herein with reference to one or more of the following can be performed by one or more emulation devices (not shown): WTRU102a-d, base station 114a-b, evolved Node B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b, and / or any other devices described herein. An emulation device can be one or more devices configured to mimic one or more of the functions described herein. For example, an emulation device can be used to test other devices and / or simulate network and / or WTRU functions.

[0145] Simulation devices can be designed to perform one or more tests on other devices in laboratory and / or carrier network environments. For example, the one or more simulation devices may perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. The one or more simulation devices may perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communication to perform tests.

[0146] The one or more emulation devices may perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be used in test scenarios within a test laboratory and / or non-deployed (e.g., testing) wired and / or wireless communication networks to perform testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0147] Similar to the HEVC test model (HM), the Joint Exploration Model (JEM) software is also built on a block-based hybrid video coding framework (100). Figure 1 A block diagram of a block-based hybrid video coding system is shown. Note that in this application, the terms "reconstruction" and "decoding" are used interchangeably. Generally, but not necessarily, the term "reconstruction" is used at the encoding end, while "decoding" is used at the decoding end.

[0148] Before encoding, the video sequence may be preprocessed, for example, by applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or by performing remapping of the input image components to obtain a more resilient signal distribution to compression (e.g., histogram equalization using one of the color components). Metadata may be associated with the preprocessing and appended to the bitstream.

[0149] The input video signal 102 is processed block by block. The HEVC specification distinguishes between "block" and "cell," where a "block" processes a specific region (e.g., luma, Y) in a sample array, and a "cell" includes a juxtaposed block of all coded color components (e.g., Y, Cb, Cr, or monochrome), syntax elements, and prediction data (e.g., motion vectors) associated with the block. In this application, the term "block" can be used to refer to data arrays of various sizes, and it can refer to macroblocks and partitions as specified in H.264 / AVC, such as any of the coding tree unit (CTU), coding unit (CU), prediction unit (PU), transform unit (TU), coding block (CB), prediction block (PB), and transform block (TB) in HEVC, superblocks or sub-partitions in AV1, such as CTU, CU, TU, CB, and TB in VVC (Multi-Functional Video Coding) or other video coding standards.

[0150] In HEVC, the extended block size is used to efficiently compress high-resolution (1080p and above) video signals. In HEVC, a CU can be up to 64×64 pixels. A CU can be further partitioned into prediction units, to which individual prediction methods are applied. For each input video block (MB or CU), spatial prediction (160) and / or temporal prediction (162) can be performed.

[0151] Spatial prediction (or "intra-frame prediction") uses pixels from samples (called reference samples) of coded neighboring blocks in the same video image / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal.

[0152] Timing prediction (also known as "inter-frame prediction" or "motion-compensated prediction") uses reconstructed pixels from encoded video images to predict the current video block. Timing prediction reduces the inherent temporal redundancy in the video signal. Typically, a timing prediction signal for a given video block is signaled by one or more motion vectors, which indicate the magnitude and direction of motion between the current block and its reference blocks. Additionally, if multiple reference images are supported (as in the case of more recent video coding standards such as H.264 / AVC or HEVC), an index of its reference image is sent for each video block; and this reference index is used to identify which reference image in the reference image repository (164) the timing prediction signal originates from.

[0153] Following spatial and / or temporal prediction, the mode decision block (180) in the encoder selects the optimal prediction mode, for example, based on a rate-distortion optimization method. The prediction block (116) is then subtracted from the current video block; and the prediction residual is decorrelated using a transform (104) and quantization (106).

[0154] The encoder decodes the coded block to provide a reference for further prediction. The quantized residual coefficients are inversely quantized (110) and inversely transformed (112) to form the reconstructed residual, which is then added back to the prediction block (126) to form the reconstructed video block.

[0155] Encoders can also skip the transform and apply quantization directly to the untransformed residual signal. Alternatively, encoders can bypass both the transform and quantization, encoding the residual directly without applying either process. In Direct Pulse Code Modulation (PCM) coding, no prediction is applied, and the coded unit samples are directly encoded into the bitstream.

[0156] Before the reconstructed video blocks are placed into the reference image repository (164) and used to encode future video blocks, additional loop filtering (166) can be applied to the reconstructed video blocks, such as deblocking filtering, SAO (sample adaptive offset), and adaptive loop filtering. In order to form the output video bitstream 120, the coding mode (inter-frame or intra-frame), prediction mode information, motion information, and quantized residual coefficients are sent to the entropy coding unit (108) for further compression and packing to form the bitstream.

[0157] Figure 3 A general block diagram of a block-based video decoder (200) is shown. The video decoder typically performs a decoding pass that is the inverse of the corresponding encoding pass, which performs video decoding as part of the encoded video data. The video bitstream 202 is first unpacked and entropy decoded at entropy decoding unit 208. The encoding mode and prediction information are sent to spatial prediction unit 260 (if intra-frame coded) or temporal prediction unit 262 (if inter-frame coded) to form prediction blocks. Residual transform coefficients are sent to inverse quantization unit 210 and inverse transform unit 212 to reconstruct the residual blocks. The prediction blocks and residual blocks are then added together at 226. The reconstructed blocks may undergo further loop filtering (266) before being stored in reference image repository 264. The reconstructed video (220) stored in the reference image repository may subsequently be stored, transmitted, or used to drive a display device, as well as for predicting future video blocks.

[0158] The decoded image may also undergo post-processing, such as inverse color transformation (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse remapping process executed during pre-encoding. Post-processing may utilize metadata derived during pre-encoding and signaled in the bitstream.

[0159] Both HEVC and JEM are adhered to block-based motion-compensated hybrid video encoding / decoding workflows, such as Figure 1 and Figure 3As shown, it is based on the same functional modules, such as spatial prediction (i.e., intra-frame prediction), temporal prediction (i.e., inter-frame prediction), transform, quantization, entropy coding, and loop filtering. However, several coding modules, especially those associated with motion compensation prediction, have been further extended and improved.

[0160] Figure 17 The diagram illustrates one embodiment of method 1700 according to the general aspects described herein. The method begins at a start box 1701, and control proceeds to box 1710, which allocates threads to perform CABAC decoding of video frames using parallel processing. A higher number of threads are allocated for decoding frames with low quantization parameters compared to decoding frames with higher quantization parameters, a higher number of threads are allocated for intra-frame decoding compared to inter-frame decoding, or a higher number of threads are allocated for P-frames, or a higher number of threads are allocated when the frame is at a lower time level. Control advances from box 1710 to box 1720 to decode the video frames using the allocated threads.

[0161] Figure 18 A second embodiment of method 1800 according to the general aspects described herein is illustrated. The method begins at a start box 1801 and control advances to box 1810 to generate motion vectors for coding units. Control proceeds from box 1810 to box 1820 to perform inter-coding unit reconstruction. Control proceeds from box 1820 to box 1830 to perform intra-coding unit reconstruction. Control proceeds from box 1830 to box 1840 to perform inverse reshaping and loop filtering of the video, wherein the generation and execution steps are performed simultaneously on different portions of the video.

[0162] Figure 19 Implementation shown Figure 17 or Figure 18 One embodiment of the method apparatus 1900. The apparatus includes a processor 1910 and can be interconnected to a memory 1920 via at least one port. Both the processor 1910 and the memory 1920 may also have one or more additional interconnects for external connection.

[0163] The processor 1910 is also configured to insert or receive information in the bitstream and to use any of the aspects described to perform compression, encoding, or decoding.

[0164] The implementation scheme described in this article encompasses various aspects, including tools, features, implementation methods, models, and methodologies. Many of these aspects are described in detail. And at least the individual characteristics are shown, usually described in a way that may sound restrictive. However, this is for clarity and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with those described in previous submissions.

[0165] It will be apparent to those skilled in the art that embodiments can generate various signals formatted to carry, for example, storable or transmissible information. The information may include, for example, instructions for performing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of a spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating the carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that signals can be transmitted via various wired or wireless links. Signals may be stored on a processor-readable medium.

[0166] The preceding section describes multiple embodiments across various claim classes and types. Features of these embodiments may be provided individually or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, across various claim classes and types: • Generate a decoded video stream, where threads are allocated to perform CABAC decoding of video frames using parallel processing. • The above method, wherein a higher number of threads are allocated for decoding frames with low quantization parameters compared to decoding frames with higher quantization parameters, a higher number of threads are allocated for intra-frame decoding compared to inter-frame decoding, or a higher number of threads are allocated for P-frames, or a higher number of threads are allocated when the frame is in a lower time layer. • Use the allocated thread to decode the aforementioned video frames.

Claims

1. A method for parallel decoding of video, the method comprising: Generate motion vectors for the coding unit; Perform inter-frame coding unit reconstruction; Perform intra-frame coding unit reconstruction; as well as, Perform inverse reshaping and loop filtering on the video. The generation and execution steps are performed simultaneously on different parts of the video.

2. An apparatus, the apparatus comprising: Memory; and One or more processors, the one or more processors being configured to Generate motion vectors for the coding unit; Perform inter-frame coding unit reconstruction; Perform intra-frame coding unit reconstruction; as well as, Perform inverse reshaping and loop filtering on the video. The device is configured to generate and execute simultaneously on different portions of the video.

3. The method of claim 1 or the device of claim 2, wherein motion vector generation is based on adjacent coding units on the left, top, and / or upper right.

4. The method of claim 1 or the apparatus of claim 2, wherein the generation of the motion vector for the upper right coding tree unit is completed before the generation of the motion vector for the current coding tree unit.

5. The method of claim 1 or the apparatus of claim 2, wherein the reconstruction of the coding unit after inter-frame coding is performed separately from the reconstruction of the intra-frame mode and the joint prediction mode of inter-frame and intra-frame.

6. The method of claim 1 or the apparatus of claim 2, wherein encoding unit reconstruction is performed in parallel on one or more portions of the encoding unit.

7. The method of claim 1 or the device of claim 2, wherein motion vector refinement is performed using a decoder when the length of the motion vector is less than a threshold.

8. A non-transitory computer-readable medium comprising data content for playback using a processor, the data content being decoded by the method of any one of claims 1, 3 to 7 or by the device of any one of claims 2 to 7.

Images