Data processing in encoding process

Abstract

A method of processing data as part of a video data encoding process is provided. The method includes configuring a coprocessor to process data in parallel using a pipeline. The pipeline is configured according to a processing scheme comprising a plurality of processes, each process performing a discrete function of an encoding process for the video data. The data includes a plurality of processing units. The method also includes processing the data at the coprocessor such that each of the plurality of data units is processed in parallel by a corresponding one of the plurality of processes.

Description

Technical Field

[0001] The present application relates to a method of processing data using a coprocessor as part of a video data encoding process. In particular, the present application relates to the use of a coprocessor for processing data in parallel using a pipeline. In particular, but not exclusively, the encoding process uses a pipeline on the coprocessor to create an encoded bitstream according to the Moving Picture Experts Group (MPEG) 5 Part 2 Low Complexity Enhanced Video Coding (LCEVC) standard. The present application can be implemented in hardware or software. Background Art

[0002] Latency and throughput are two important parameters for evaluating data encoding techniques, for example, for encoding video data. Latency is the time required to produce an encoded frame after receiving the original frame. Throughput is the time required to produce a second encoded frame after producing the first encoded frame.

[0003] The throughput of video data encoding can be improved by improving latency. However, the cost of improving latency is high. Therefore, an efficient and economical method is needed to improve the throughput of video encoding. Summary of the invention

[0004] According to a first aspect of the present application, a method for processing data as part of a video data encoding process is provided. The method includes: configuring a coprocessor to process data in parallel using a pipeline. The pipeline is configured according to a processing scheme, the processing scheme including a plurality of processes, each process performing a discrete function of an encoding process for video data. The data includes a plurality of processing units. The method also includes processing the data at the coprocessor so that each of the plurality of data units is processed in parallel by a corresponding one of the plurality of processes. In this way, the throughput of data processing can be significantly increased in an efficient and economical manner.

[0005] Preferably, the coprocessor receives a plurality of instructions from the main processor to execute a processing scheme.

[0006] Preferably, the main processor is a central processing unit (CPU) and the coprocessor is a graphics processing unit (GPU).

[0007] Preferably, the host processor instructs the co-processor using the Vulkan application programming interface (API).

[0008] Preferably, the plurality of processes configured and executed on the coprocessor are processes before entropy encoding in the encoding process, wherein the coprocessor outputs the output of the last process of the processing scheme to the main processor for entropy encoding.

[0009] Preferably, the multiple processes include: a conversion process, an M-filtering process, a downsampling process, a basic encoder, a basic decoder, a transport stream (TS) and TS complexity extraction process, a look-ahead indicator extraction process, a perceptual analysis process, and one or more of an enhancement layer encoding process.

[0010] Preferably, the enhancement layer encoding process includes one or more of the following processes: a first residual generation process for generating first-level residual information; a second residual generation process for generating second-level residual information; a temporal prediction process operating on the second-level residual information; one or more transform processes; and one or more quantization processes.

[0011] Preferably, the first stage residual generation process comprises a comparison of a downsampled version of a processing unit with a base encoded and decoded version of the processing unit.

[0012] Preferably, the second residual generation process comprises a comparison of an input version of said block with an upsampled version of a base encoded and decoded version of said block, said block being corrected by said first level residual information of said block.

[0013] Preferably, the processing scheme offloads base encoder and base decoder operations to dedicated base codec hardware and outputs a downsampled version of the base decoded version to the dedicated base codec hardware for one processing unit and receives the downsampled version after codec processing.

[0014] Preferably, the downsampled version is the lowest spatial resolution version during the encoding process.

[0015] Preferably, the processing scheme performs forward complexity prediction on the given processing unit when the base codec processes a downsampled version of the given processing unit.

[0016] Preferably, the forward complexity prediction includes one or more of the following processes: a TS and TS complexity extraction process, a forward indicator extraction process, and a perceptual analysis process.

[0017] Preferably, the processing scheme uses multiple synchronization primitives to ensure that shared resources are only allocated to one process at a time.

[0018] Preferably, the plurality of synchronization primitives are semaphores.

[0019] Preferably, the semaphore is a binary semaphore.

[0020] Preferably, earlier processes among the plurality of processes have higher priority for any shared resource than later processes.

[0021] Preferably, the processing scheme uses a completion-or-feed-forward approach such that a plurality of earlier processes of the plurality of processes signals a next process when an earlier process completes.

[0022] Preferably, the completion-ie feed-forward method uses synchronization primitives.

[0023] Preferably, a plurality of processes having a processing scheme for a plurality of relatively more complex discrete functions are allocated more resources in the coprocessor than a plurality of processes having a processing scheme for a plurality of relatively less complex discrete functions.

[0024] Preferably, the encoding process creates an encoded bitstream according to the Moving Picture Experts Group (MPEG) 5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) standard.

[0025] Preferably, the processing unit is one of a frame or image, a data block within a frame, a coding block, and a data slice within a frame.

[0026] According to a second aspect of the present application, a coprocessor for encoding video data is provided, wherein the coprocessor is arranged to perform any of the aforementioned methods.

[0027] According to a first aspect of the present application, a computer-readable medium is provided, wherein the computer-readable medium includes a plurality of instructions, and when the instructions are executed, a processor is caused to execute any of the aforementioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0029] Figure 1 is a block diagram of a hierarchical coding technique that can be used with the principles disclosed in this application;

[0030] Figure 2 is a schematic diagram illustrating a pipeline operation at a coprocessor according to the present application; and

[0031] Figure 3 is a flow chart of a method of processing data as part of a video data encoding process according to the present application. DETAILED DESCRIPTION

[0032] Figure 1 It is a block diagram of the hierarchical coding technology implemented in the present application. Figure 1 The hierarchical coding technique in MPEG-4 complies with the Low Complexity Enhancement Video Coding (LCEVC) standard (ISO / IEC 23094-2:2021(en)) in Part 2 of Moving Picture Experts Group (MPEG) 5. LCEVC is a flexible, adaptable, efficient, and computationally inexpensive coding technique that combines a base codec (e.g., Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), or any other existing or future codec) with another different coding format that provides at least two enhancement levels of coded data.

[0033] exist Figure 1 In the example of , some processes of LCEVC are completed in the main processor 100 (such as the central processing unit (CPU)), and other processes are completed in the coprocessor 150 (such as the graphics processing unit (GPU)). The coprocessor 150 is a computer processor used to supplement the functions of the main processor 100 (CPU). The operations performed by the coprocessor 150 can be floating-point arithmetic, graphics, signal processing, string processing, cryptography, or I / O connected to peripheral devices. By offloading processor-intensive tasks from the main processor 100, the coprocessor can accelerate system performance. The coprocessor 150 mentioned in this application is not limited to the GPU, but it can be understood that any coprocessor with parallel operation capabilities can be suitable for executing this application.

[0034] By splitting the process of LCEVC between the main processor 100 and the coprocessor 150, LCEVC can be improved by utilizing the parallel operation of the coprocessor 150 (such as a GPU). The process of executing LCEVC in parallel improves the throughput of video encoding. It takes time and resources to initialize the coprocessor 150. Therefore, the time and resources for initialization should be regained by effectively utilizing the coprocessor 150. In other words, unless parallelization is used in the coprocessor 150, it is not always effective to initialize the coprocessor 150 for video encoding.

[0035] The coprocessor 150 is configured to execute a processing scheme as part of the overall encoding process by receiving a plurality of instructions from the main processor 100. The main processor 100 may instruct the coprocessor 150 to execute the processing scheme using a Vulkan application program interface (API), which provides a consistent way to interact with coprocessors from different manufacturers. However, it will be appreciated that other APIs may be used.

[0036] Some processes of the processing scheme perform discrete functions on a processing unit (e.g., a frame, residual frame, slice, tile, or block) to prepare or further process the processing unit. Some processes depend on the output of other processes and must wait until the other processes have completed processing the processing unit.

[0037] Generally speaking, Figure 1 The encoding process shown creates a converted, preprocessed and downsampled source signal, encodes it with a base codec, adds a first level of correction data to the decoded output of the base codec to produce a corrected image, and then adds another level of enhancement data to an upsampled version of the corrected image.

[0038] Specifically, an input video 152, such as a video of an original resolution, is received and converted at a converter 156. The converter 156 converts the input video 152 from an input signal format (RGB, etc.) and a color space (sRGB, etc.) to a format and color space supported by the encoding process, such as (YUV420p and BT709, BT2020, etc.).

[0039] The converted input signal is preprocessed by applying a blur filter 158 and a sharp filter 160 (collectively referred to as an M filter). The preprocessed input video signal is then downsampled by a downsampler 162. By feeding the converted, preprocessed and downsampled input video 152 to a base codec (e.g., AVC, HEVC, or any other codec), a first coded stream (coded elementary stream 154) is generated. The coded elementary stream 154 may be referred to as a base layer or base level.

[0040] A second coded stream (coded level 1 stream 102) is produced by processing the residual obtained by taking the difference between the reconstructed base codec signal and a downsampled version of the input video 152. A third coded stream (coded level 2 stream 104) is produced by processing the residual obtained by taking the difference between an upsampled version of the corrected version of the reconstructed base coded video and the input video 152. In some cases, Figure 1 The components in may provide a general low complexity encoder. In some cases, the enhancement stream may be generated by an encoding process that forms part of the low complexity encoder, and the low complexity decoder may be configured to control an independent base encoder 164 and decoder 166 (e.g., packaged as a base codec). In other cases, the base encoder 164 and decoder 166 may be provided as part of a low complexity encoder. In one case, Figure 1 The low-complexity encoder can be viewed as a wrapper around the base codec, where the functionality of the base codec can be hidden from the entity implementing the low-complexity encoder.

[0041] Looking at the process of generating the enhancement stream in more detail, to generate the encoded level 1 stream 102, the encoded elementary stream is decoded by the elementary decoder 166 (i.e., a decoding operation is performed on the encoded elementary stream 154 to generate a decoded elementary stream). The decoding can be performed by a decoding function or mode of the elementary codec. Then, a difference between the decoded elementary stream and the downsampled input video is created at the level 1 comparator 168 (i.e., a subtraction operation is applied to the downsampled input video 152 and the decoded elementary stream to generate a first set of residuals). The output of the comparator 168 can be referred to as a first set of residuals, such as residual data planes or frames, where the residual values ​​are determined for each image at the resolution of the output of the elementary encoder 164, the elementary decoder 166, and the downsampling block 162.

[0042] The difference is then transformed, quantized, and entropy encoded at transform block 170, quantization block 172, and entropy encoding block 106, respectively, to generate the encoded level 1 stream 102 (i.e., the first set of residuals are subjected to encoding operations to generate the first enhancement stream). The conversion and quantization process occurs in coprocessor 150. After quantization, coprocessor 150 passes the processed data to main processor 100, where entropy encoding occurs.

[0043] As described above, the enhancement stream may include a first level enhancement and a second level enhancement. The first level enhancement may be considered a correction stream, e.g., providing a correction level stream to a base encoded / decoded video signal at a lower resolution than the input video 152. The second level enhancement may be considered to convert the corrected stream to a further enhancement level of the original input video 152, e.g., applying an enhancement or correction level to a signal reconstructed from the corrected stream.

[0044] exist Figure 1 In the example of , a second level enhancement is created by encoding another set of residuals. This other set of residuals is generated by a level 2 comparator 174. The level 2 comparator 174 determines the difference between an upsampled version of the decoded level 1 stream (e.g., the output of the upsampling block 176) and the input video 152. The input to the upsampling block 176 is generated by applying inverse quantization and inverse transformation to the output of the quantization block 172 at the inverse quantization block 178 and the inverse transformation block 180, respectively. This will generate a set of decoded level 1 residuals. These are then combined with the output of the basic decoder 166 at the summing component 182. This effectively applies the level 1 residuals to the output of the basic decoder 166. It allows losses in the level 1 encoding and decoding process to be corrected by the level 2 residuals. The output of the summing component 182 can be viewed as an analog signal that represents the output of applying the level 1 processing to the encoded base stream 154 and the encoded level 1 stream 102 at the decoder.

[0045] As described above, the upsampled stream is compared to the input video 152 creating another set of residuals (i.e., the upsampled recreated stream is subjected to a difference operation to generate another set of residuals). This other set of residuals is then transformed, quantized, and entropy encoded at the transform block 184, the quantization block 186, and the entropy coding block 108, respectively, to generate the encoded level 2 enhancement stream (i.e., the other set of residuals is then subjected to an encoding operation to generate the encoded further enhancement stream).

[0046] Therefore, if Figure 1 As shown and described above, the output of the encoding process is a base stream and one or more enhancement streams, which preferably include a first-level enhancement and a further-level enhancement. These three streams can be combined with or without additional information such as a control header to generate a combined stream representing the video coding framework of the input video 152. It should be noted that Figure 1The components shown can operate on data slices within a frame, tile, block, or coding unit (e.g., a 2×2 or 4×4 portion of a frame corresponding to a particular resolution level). These components operate without any dependencies between processing units, so they can be applied to multiple slices, tiles, blocks, or coding units within a frame in parallel. This is different from comparative video coding schemes, where dependencies (e.g., spatial dependencies or temporal dependencies) exist between processing units (e.g., blocks). The dependencies of comparative video coding schemes limit the level of parallelism and require higher complexity.

[0047] To exploit parallelism, Figure 1 Many of the processes in are implemented in the coprocessor 150. Figure 1 The coprocessor 150 in the embodiment uses a pipeline to process the input video 152 in parallel. The pipeline allows multiple processes to occur simultaneously. For example, when the downsampling process is performed on data #n, the M filtering process can be performed on data #n+1 at the same time. This can improve the throughput of video encoding.

[0048] Figure 2 is a schematic diagram illustrating the pipeline operation at the coprocessor according to the present application. Figure 2 In the example of , the coprocessor receives data to be processed as part of a video data encoding process. In this example, the received data is frame data from a video signal, however other types of data may also be received, such as a frame of data, a tile, a block, or a slice of data within a coding unit.

[0049] The coprocessor 150 includes five encoder pipelines that perform five processes as shown in the top vertical row. These processes are: a converter process, an M filter process, a downsampling process, a forward complexity prediction process, and an enhancement layer encoding process. Other types of processes or different combinations of processes may also be used. Figure 2 Each process shown consists of its own discrete function that is applied to the data it is processing.

[0050] Each process in the encoder pipeline undergoes five operations in each frame period. These five operations are shown in the leftmost column. The five operations are: Get, Prepare, Execute, Clear, and Issue.

[0051] In a fetch operation, each process gets (not necessarily simultaneously) a frame to process. Each process has an input queue and fetches the next frame in the input queue during a fetch operation. The input queue is configured during initialization of the processing scheme that will implement the encoding process. Figure 2The example of shows the following acquisitions: the converter process acquires frame #n+7 from its input queue; the M filtering process acquires frame #n+5, while frame #n+6 is queued; the downsampling process acquires frame #n+2, while frames n+3 and n+4 are queued; the forward complexity prediction process acquires frame #n+1; and the enhancement layer encoding process obtains frame #n. Since different processes run at different speeds, some frames are queued. Therefore, if the first process has processed the first data and the second process has not yet processed the second data, the first data will be queued until the next frame is ready (i.e., after the second data is processed).

[0052] In some examples, a processing scheme at a coprocessor uses synchronization primitives to ensure that a shared resource (such as frame data stored in a shared memory) is allocated to only one process at a time. The synchronization primitive is a semaphore. The semaphore is a binary semaphore. In the processing scheme, an earlier process has a higher priority than a later process when accessing any shared resource (such as frame data stored in a shared memory). The processing scheme uses a completion-and-feedforward method so that multiple earlier processes in multiple processes signal a next process when an earlier process completes. The completion-and-feedforward method uses synchronization primitives.

[0053] In the prepare operation, resources are allocated for each process. In the execute operation, the function of each process is executed on each process's respective data. In the clear operation, resource allocations are reset. In the emit operation, processed frames are output for each process.

[0054] By using the above-mentioned parallel operation in the coprocessor, the throughput of data processing can be significantly improved. For example, if the coprocessor receives data including five processing units (e.g., five frames), the five processing units use five processes for processing, and for the five processing units using five processes for processing, 25 time cycles (e.g., frame cycles) are usually required. However, by using pipelines to execute the processes in the coprocessor in parallel, the time cycle can be reduced to 9.

[0055] In this example, the pipeline process for forward complexity prediction in the coprocessor can be compared with Figure 1 The forward complexity prediction pipeline may occur at almost the same time as the base codecs and may operate on the same data that the base codecs operate on. Alternatively, the forward complexity prediction pipeline may occur at a different time, for example, before the downsampling process. The forward complexity prediction includes one or more of the following processes: a transport stream (TS) complexity extraction process, a look-ahead indicator extraction process, and a perceptual analysis process.

[0056] Figure 1 and Figure 2As shown, a process having multiple relatively more complex discrete functions may be allocated more resources in the coprocessor than a process having multiple relatively less complex discrete functions, so that a process that normally takes longer to complete may be executed faster due to the efficient resource allocation.

[0057] Figure 3 1 is a flow chart of a method for processing data as part of a video data encoding process according to the present application. In step 310, the method configures a coprocessor to process data in parallel using a pipeline, the data including a plurality of processing units. In step 320, the method processes the data at the coprocessor so that each of the plurality of processing units is processed by a corresponding one of the plurality of processes in the parallel pipeline.

[0058] The above embodiments are to be understood as illustrative examples. Further embodiments are contemplated. It is to be understood that any feature described in connection with any one embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other embodiment, or with any combination of any other embodiment. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the present application as defined in the appended claims.

[0059] exist Figure 1 In the example of , the low complexity encoder is distributed between the main processor 100 and the coprocessor 150 so that the two processors work together to perform the overall low complexity encoding. The basic codec is a dedicated hardware device implemented in the coprocessor 150 for quickly performing basic encoding / decoding. Alternatively, the basic codec can be a computer program code executed by the coprocessor 150.

[0060] In some cases, the base stream and the enhanced stream can be transmitted separately. References to the encoded data described herein may refer to an enhanced stream or a combination of a base stream and an enhanced stream. The base stream can be decoded by a hardware decoder, while the enhanced stream can be suitable for software processing implementation with appropriate power consumption. This general encoding structure creates multiple degrees of freedom to allow great flexibility and adaptability to a variety of situations, thereby making the encoding format suitable for OTT transmission, live broadcast, ultra-high-definition (UHD) live broadcast, and other situations. Although the decoded output of the basic codec is not for viewing, it is a low-resolution fully decoded video, which makes the output compatible with existing decoders and, where appropriate, can also be used as a low-resolution output.

[0061] In some examples, each enhancement stream or enhancement streams can be encapsulated into one or more enhancement bitstreams using a set of network abstraction layer units (NALUs). NALUs are used to encapsulate enhancement bitstreams so that the enhancements are applied to correct base reconstructed frames. For example, a NALU can contain a reference index to a NALU that contains a base decoder reconstructed frame bitstream to which the enhancements must be applied. In this way, the enhancements can be synchronized to the base stream and the frames of each bitstream can be combined to produce a decoded output video (i.e., the residual of each frame at the enhancement level is combined with the frames of the base decoded stream). A set of images can represent multiple NALUs.

[0062] The skilled person will understand from this disclosure that encoding video data in the disclosed manner is not graphics rendering, nor does it involve transcoding. Instead, the disclosed video encoding involves creating an encoded video stream from an input video source. Definitions and terminology

[0063] The following terms are used in some examples described in this article:

[0064] "Access Unit" - This refers to a group of Network Abstraction Layer Units (NALUs) that are associated with each other according to a specified classification rule. They can be consecutive in decoding order and contain coded pictures (i.e., frames) of the video (in some cases only one).

[0065] "Base layer" - This is the layer associated with the base picture of the codec, where "base" refers to the codec that receives the processed input video data. It can be part of the bitstream associated with the base.

[0066] "Bitstream" - This is a sequence of bits, which may be provided as a stream of NAL units or as a byte stream. It may form a representation of coded pictures and associated data forming one or more coded video sequences (CVSs).

[0067] "Block" - an MxN (m columns by n rows) array of samples, or an MxN array of transform coefficients. The terms "coding unit" or "coding block" are also used to refer to an MxN array of samples. These terms may be used to refer to a set of image elements (e.g., pixel values ​​for a particular color channel), a set of residual elements, a set of values ​​representing the residual elements after processing, and / or a set of coded values. The term "coding unit" is sometimes used to refer to a coding block of luma samples or a coding block of chroma samples for an image with three sample arrays, or a coding block of samples for a monochrome image or an image encoded using three independent color planes and a syntax structure for encoding the samples. A coding unit may consist of an MxN array R of elements R[x][y]. For a 2x2 coding unit, there may be 4 elements. For a 4x4 coding unit, there may be 16 elements.

[0068] "Chroma" - This is an adjective used to specify that an array of samples or a single sample represents a color signal. This may be one of two color difference signals related to the primary colors, for example represented by the symbols Cb and Cr. It can also be used to refer to the channels in a set of color channels that provide information about the coloring of the image. The term chroma is used instead of the term chrominance to avoid the connotation of linear light transfer characteristics usually associated with the term chrominance.

[0069] "Coded picture" - This is used to refer to a set of coding units that represent a coded representation of a picture.

[0070] "Coded base picture" - may refer to a coded representation of a picture using a base coding process that is separate from (and usually different from) an enhancement coding process.

[0071] "Encoded representation" - a data element represented in an encoded form.

[0072] "Decoded base picture" - this refers to the decoded picture obtained by decoding the encoded base picture.

[0073] "decoded picture"—a decoded picture obtained by decoding a coded picture. A decoded picture may be a decoded frame or a decoded field. A decoded field may be a decoded top field or a decoded bottom field.

[0074] "Decoder" - a device or apparatus that embodies the decoding process.

[0075] "Decoding order" - This probably refers to the order in which the decoding process processes the syntax elements.

[0076] "Decoding process" - This is the term used to refer to the process of reading a bitstream and deriving a decoded image from it.

[0077] "Encoder" - a device or apparatus that embodies the encoding process.

[0078] "Encoding process" - this is used to refer to the process of producing a bitstream (ie, the encoded bitstream).

[0079] "Enhancement layer" - This is a layer associated with coded enhancement data, where the enhancement data is used to enhance the "base". It can be part of the bitstream containing the residual data plane. The singular term is used to refer to an encoding and / or decoding process that is different from the "base" encoding and / or decoding process.

[0080] "Enhancement sublayer" - In some examples, an enhancement layer includes multiple sublayers. For example, the first and second levels described below are "enhancement sublayers" of layers that are considered enhancement layers.

[0081] "Video Frame or Frames" - In some examples, a video frame may include a frame consisting of an array of luma samples or a luminance sample array and two corresponding arrays of chroma samples in a monochrome format. Luma and chroma samples may be provided in 4:2:0, 4:2:2, and 4:4:4 color formats (among others). A frame may consist of two fields, a top field and a bottom field (e.g., these terms may be used in the context of interlaced video). References to "frames" in these examples may also refer to frames of a particular plane, e.g., where a separate residual frame is generated for each YUV plane. As such, the terms "plane" and "frame" may be used interchangeably.

[0082] "Layer" - This term is used in some examples to refer to one of a set of syntactic structures in an unbranching hierarchical relationship, for example, when referring to a "base layer" and an "enhancement layer", or two (sub-) "layers" of an enhancement layer.

[0083] "Luminance" - This term is used as an adjective to designate an array of samples or a single sample representing a signal of brightness or monochromaticity, such as that associated with a primary color. Luminance samples may be denoted by the symbol or subscript Y or L. The term "luma" is used rather than the term luminance to avoid the connotation of a linear light transfer characteristic commonly associated with the term luminance. The symbol L is sometimes used instead of the symbol Y to avoid confusion with the symbol Y used for vertical position.

[0084] "Network Abstraction Layer (NAL) Unit (NALU)"—This is a syntax structure that contains an indication of the type of data to follow, and the bytes that contain that data in the form of a Raw Byte Sequence Payload (RBSP). An RBSP is a syntax structure that contains an integer number of bytes that are encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits that contain the syntax elements, followed by an RBSP stop bit, followed by zero or more subsequent bits equal to 0. RBSPs may be interspersed with emulation stop bytes as required.

[0085] "Network Abstraction Layer (NAL) unit stream" - a sequence of NAL units.

[0086] "Picture" - This is a general term for a field or frame. In some cases, the terms frame and picture are used interchangeably.

[0087] "Residual" - This term is defined in further examples below. It generally refers to the difference between a reconstructed version of a sample or data element and a reference of the same sample or data element.

[0088] "Residual plane" - This term is used to refer to a collection of residuals, which are organized in a planar structure similar to a color component plane, for example. A residual plane may include a plurality of residuals (i.e., residual image elements), which may be array elements having values ​​(e.g., integer values).

[0089] "Slice" - A slice is a spatially distinct region of a frame that is encoded separately from any other region in the same frame.

[0090] "Source" - This word is used in some cases to describe the video material or some attribute of it before it is encoded.

[0091] "Tile" - This term is used in some examples to refer to a rectangular area of ​​blocks or coding units within a particular image, for example, it can refer to a region of a frame containing multiple coding units, where the size of the coding unit is set based on the applied transform. For example, a tile might consist of an 8×8 array of blocks / coding units. If the blocks / coding units are 4×4, this means that each tile has 32×32 elements; if the blocks / coding units are 2×2, this means that each tile has 16×16 elements.

[0092] "Transform coefficient" (or just "coefficient") - This term is used to refer to the value produced when a transform is performed on a residual or residual data (such as a processed residual). It can be a scalar, which is considered to be in the transform domain. In one case, an M×N coding unit can be flattened into an M×N one-dimensional array. In this case, the transform may include the multiplication of the one-dimensional array with the M×N transformation matrix. In this case, the output may contain another (flattened) M*N one-dimensional array. In this output, each element may be associated with a different "coefficient", for example, for a 2×2 coding unit, there may be 4 different types of coefficients. In this way, the term "coefficient" can also be associated with a specific index in the inverse transform part of the decoding process, for example, a specific index in the one-dimensional array representing the transform residual mentioned above.

Claims

1. A method of processing video frames as part of a video data encoding process, comprising: configuring a coprocessor to process a video frame comprising a plurality of blocks in parallel using a pipeline, the pipeline configured according to a processing scheme comprising a plurality of processes, each process performing a discrete function of the encoding process on the plurality of blocks of the video frame; as well as processing the video frame at the coprocessor such that a plurality of the blocks are processed in parallel by a corresponding one of the plurality of processes; The multiple processes are multiple processes before entropy coding in the encoding process.

2. The method according to claim 1, wherein: The coprocessor receives a plurality of instructions from the main processor to execute the processing scheme.

3. The method according to claim 2, wherein: The main processor is a central processing unit (CPU), and the coprocessor is a graphics processing unit (GPU).

4. The method according to claim 2 or 3, wherein: The host processor instructs the coprocessor using a Vulkan application programming interface (API).

5. The method according to any one of claims 2 to 4, wherein: The coprocessor outputs the output of the final process of the processing scheme to the main processor for entropy encoding.

6. A method according to any preceding claim, wherein: The multiple processes include: a conversion process, an M-filtering process, a downsampling process, a basic encoder, a basic decoder, a transport stream (TS) and TS complexity extraction process, a look-ahead indicator extraction process, a perceptual analysis process, and one or more of an enhancement layer encoding process.

7. The method according to claim 6, wherein: The enhancement layer encoding process includes one or more of the following processes: a first residual generation process for generating first-level residual information; a second residual generation process for generating second-level residual information; a temporal prediction process operating on said second level residual information; one or more transform processes; and one or more quantization processes.

8. The method according to claim 7, wherein: The first stage residual generation process includes comparison of a downsampled version of a block with a base encoded and decoded version of the block.

9. The method according to claim 8, wherein: The second residual generation process comprises a comparison of an input version of the block with an upsampled version of a base encoded and decoded version of the block, the block being corrected by the first level residual information of the block.

10. A method according to any preceding claim, wherein: The processing scheme offloads base encoder and base decoder operations to dedicated base codec hardware and outputs a downsampled version of a block to the dedicated base codec hardware and receives a base decoded version of the downsampled version after processing by the codec.

11. The method according to claim 10, wherein: The downsampled version is the lowest spatial resolution version in the encoding process.

12. The method according to claim 10 or 11, wherein: The processing scheme performs forward complexity prediction on a given block while the base codec processes a downsampled version of the given block.

13. The method according to claim 12, wherein: The forward complexity prediction includes one or more of the following processes: a transport stream (TS) and TS complexity extraction process, a forward-looking indicator extraction process, and a perceptual analysis process.

14. A method according to any preceding claim, wherein: The processing scheme uses a number of synchronization primitives to ensure that shared resources are allocated to only one process at a time.

15. The method according to claim 14, wherein: The plurality of synchronization primitives are semaphores.

16. The method according to claim 15, wherein: The semaphore is a binary semaphore.

17. The method according to any one of claims 14 to 16, wherein: Earlier ones of the plurality of processes have higher priority for any shared resource than later ones of the plurality of processes.

18. The method according to any one of claims 14 to 17, wherein: The processing scheme uses a completion-and-feed-forward approach such that earlier processes of the plurality of processes signal a next process when an earlier process is completed.

19. The method according to claim 18, wherein: The completion-feedforward method uses the synchronization primitives.

20. A method according to any preceding claim, wherein: Processes of the processing scheme having a plurality of relatively more complex discrete functions are allocated more resources in the coprocessor than processes of the processing scheme having a plurality of relatively less complex discrete functions.

21. A method according to any preceding claim, wherein: The encoding process creates an encoded bitstream according to the Moving Picture Experts Group (MPEG) 5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) standard.

22. A coprocessor for encoding video data, wherein: The co-processor is arranged to perform the method of any preceding claim.

23. A computer-readable medium comprising: A plurality of instructions, when the plurality of instructions are executed, cause a processor to execute the method according to any one of claims 1 to 21.

Images