Flexible coding of components in layered scalable coding

Abstract

Examples described herein relate to signal encoding. Systems and methods of encoding and decoding a signal, such as a video signal, are described. In one case, a method of encoding a signal uses a hierarchical encoding approach in which the signal is encoded at a first layer using a first encoding module and at a second layer using a second encoding module. The signal is comprised of two or more components. The method includes the step of sending a signal from the second encoding module to the first encoding module to instruct the first encoding module to provide only a first component of the signal at the first layer to the second encoding module.

Description

TECHNICAL FIELD

[0001] The present invention relates to methods for processing signals, such as, as non-limiting examples, video, image, hyperspectral image, audio, point cloud, 3DoF / 6DoF and volumetric signals. Processing data can include, but is not limited to, obtaining, deriving, encoding, outputting, receiving and reconstructing signals in the context of a hierarchical (layer-based) encoding format, where signals are decoded in layers at successively higher quality levels, with subsequent layers (“tiers”) of reconstruction data being utilized and combined. Different layers of a signal can be encoded with different encoding formats (e.g., as non-limiting examples, traditional single-layer DCT-based codecs, ISO / IEC MPEG-5 Part 2 Low Complexity Enhancement Video Coding, SMPTE VC-6 2117, etc.), by different elementary streams that can or can not be multiplexed in a single bitstream. BACKGROUND

[0002] In a layer-based encoding format, such as ISO / IEC MPEG-5 Part 2 LCEVC (hereinafter “LCEVC”) or SMPTE VC-6 2117 (hereinafter “VC-6”), a signal is decomposed into multiple data “tiers” (also referred to as “hierarchical layers”), each corresponding to a “quality level” (also referred to herein as “LoQ”) of the signal, from a highest tier at the sampling rate of the original signal to a lowest tier that typically has a lower sampling rate than the original signal. In non-limiting examples, when the signal is a picture in a video stream, the lowest tier can be a thumbnail of the original picture, e.g., a low-resolution frame in the video stream, or even just a single picture element. Other tiers contain information about corrections to apply to a reconstructed rendition in order to produce a final output. Tiers can be based on residual information, e.g., a difference between a version of the original signal at a particular quality level and a reconstructed version of the signal at the same quality level. The lowest tier can not contain residual information, but can contain a lowest sampling of the original signal. Reconstructing a decoded signal at a given quality level proceeds by first decoding the lowest tier (thereby reconstructing the signal at a first lowest quality level), then predicting a rendition of the signal at a second next higher quality level, then decoding a corresponding second reconstruction data tier (also referred to as “residual data” at the second quality level), then combining the prediction with the reconstruction data in order to reconstruct the rendition of the signal at the second higher quality level, and so on, until the given quality level is reconstructed.

[0003] The reconstructed signal can comprise decoding residual data and using that residual data to correct a version of a particular quality level derived from a signal version from a lower quality level. Different data tiers can be encoded using different encoding formats, and different quality levels can have different sampling rates (e.g. resolutions for the case of image or video signals). Subsequent tiers can refer to the same signal resolution (i.e. sampling rate) of the signal, or to progressively higher signal resolutions. Examples of these approaches are described in more detail in the available specifications for LCEVC and VC-6.

[0004] The process of encoding and decoding signals tends to be resource intensive. For example, video encoding and decoding requires processing data frames in fractions of a second (33ms for 30Hz frames, or 16ms for 60Hz frames). Applications such as video conferencing that require audio and video encoding and transmission over a network tend to require a significant portion of the available resources on a computing device. Mobile devices also face additional challenges, with more limited operating processing resources and often powered by a battery. It is desirable to provide improved encoding and decoding methods to cope with variable real-world usage conditions. SUMMARY

[0005] Various aspects of the application are set out in the accompanying independent claims. Variations of the application are set out in the accompanying dependent claims. Additional variations and aspects are set out in the examples described herein. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 A block diagram showing an example of an encoding system according to an embodiment is shown;

[0007] Figure 2 A block diagram showing an example of a decoding system according to an embodiment is shown;

[0008] Figure 3 A flow diagram showing an example encoding method according to an embodiment is shown; and

[0009] Figure 4 A block diagram showing another example of an encoding system according to a variation is shown. DETAILED DESCRIPTION

[0010] In layer-based hierarchical coding techniques, such as those implemented in LCEVC and VC-6, a signal can require correction of the varying amount of fidelity of the predicted rendition based on a given level of quality (LoQ). This correction is provided by “residual data” (or simply “residuals”) in order to generate a signal reconstruction at the given LoQ that is most similar (or even lossless reconstruction) to the original signal. In layer-based hierarchical coding, a signal can be composed of multiple components or channels. For audio signals, these can contain components related to different speakers and / or microphones. For video signals, these can contain components related to different color channels. For example, LCEVC and VC-6 are configured to handle different chroma planes (e.g., Y or luminance, U chroma, and V chroma as non-limiting examples). The chroma planes can be defined according to a specified color encoding method and can be reconstructed to their target resolution by independent residual planes. The chroma planes can be processed serially or in parallel and can be combined in the output reconstruction for presentation on a display device. Further details of the standardized process for decoding chroma planes are described in the specifications of LCEVC and VC-6.

[0011] Encoding and / or decoding a signal requires efficient utilization of available resources. For example, hardware and / or software encoders and decoders need to efficiently control processor, memory, and power utilization (among others). For mobile encoders and decoders, such as smartphones and tablets, power is typically supplied by a battery. When battery consumption is relevant, for example when it is desirable to conserve battery consumption, encoding processing power is a relevant measure to minimize. In several devices (e.g., mobile devices as non-limiting examples), power consumption is significantly affected by the amount of memory accesses and memory copies.

[0012] Certain novel embodiments shown herein allow an encoding and / or decoding device to flexibly save significant processing power by limiting the encoding of upper layer signals to a subset of available signal components. Surprisingly, encoding only one component of a higher level signal still provides perceptible improvements in the output reconstruction, but still significantly reduces resource utilization. This makes it suitable for known layer-based hierarchical coding methods to efficiently encode and decode in the case of limited resources on, for example, a computing device. In one example, limiting the encoding of signal components limits the generation of residual hierarchies for chroma planes at higher levels of quality.

[0013] The non-limiting embodiments shown herein refer to signals as sequences of samples. These samples can contain, for example, two-dimensional images, video frames, video fields, sound frames, etc. In the description, the terms "image", "picture" or "plane" (intended to have the broadest meaning of "hyperplane", i.e. an array of elements with any number of dimensions and a given sampling grid) will be frequently used to identify the digital rendering of a signal sample along a sequence of samples, where each plane has a given resolution for each of its dimensions (e.g. X and Y) and contains a set of plane elements (or "elements", or "pixels", or display elements of a two-dimensional image generally referred to as "pixels", of a volumetric image generally referred to as "voxels", etc.) characterized by one or more "values" or "settings" (e.g. color settings in a suitable color space, settings indicative of a level of density, settings indicative of a level of temperature, settings indicative of a tone of audio, settings indicative of an amplitude, settings indicative of a depth, settings indicative of a level of transparency of an alpha channel, etc. by way of non-limiting examples). Each plane element is identified by a suitable set of coordinates, indicative of the integer position of the element in the sampling grid of the image. The signal dimensions can include only spatial dimensions (e.g. in the case of an image) or also a temporal dimension (e.g. in the case of a signal that evolves over time, such as a video signal).

[0014] As non-limiting examples, the signal can be an image, an audio signal, a multi-channel audio signal, a telemetry signal, a video signal, a 3DoF / 6DoF video signal, a volumetric signal (e.g. medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or even a signal with more than four dimensions.

[0015] For simplicity, the non-limiting embodiments shown herein generally refer to signals displayed as 2D set planes (e.g. 2D images in a suitable color space), such as video signals. The terms "picture", "frame" or "field" will be used interchangeably with the term "image" in order to indicate a temporal sample of a video signal: any concept and method shown for a video signal composed of frames (progressive video signal) can be easily applied to a video signal composed of fields (interlaced video signal) and vice versa. Although the embodiments shown herein focus on images and video signals, the skilled person can easily understand that the same concepts and methods apply also to any other type of multi-dimensional signal (e.g. audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.).

[0016] The components of the signal represent different "values" or "settings." For example, as noted above, these can include different color channels, different sensor channels, different audio channels, metadata channels, etc. For example, a different sample plane as described above can be provided for each different component, and the encoding and / or decoding process can be applied to each component plane, either serially or in parallel, to generate encoded and decoded versions of the components. For ease of explanation, reference will be made herein to the YUV color encoding of video signals, in which there are three components - Y, U, and V. Y represents the luminance or brightness channel, and U and V represent different opponent color channels. It should be noted that the described examples are not limited to YUV encoding, and can be applied to different color encodings (including RGB, Lab, YDbDr, XYZ, etc.) and non-color examples. For example, for surround sound audio, there can be 6 audio channels, including front left and right, surround left and right, center, and subwoofer channels.

[0017] In a first aspect described herein, there is a method of encoding a signal using a hierarchical or multi-layer encoding method. The signal is encoded at a first layer using a first encoding module, and at a second layer using a second encoding module. For example, the first encoding module can represent a base encoding layer, and the second layer can represent an enhancement encoding layer. Alternatively, the first and second encoding modules can represent different sub-layers of an enhancement encoding layer. The signal is comprised of two or more components.

[0018] In an example of the first aspect, the components encoded by the second encoding module include a subset of the components encoded by the first encoding module. This can be achieved by a method that includes sending a signal from the second encoding module to the first encoding module to instruct the first encoding module to provide only the first components of the signal at the first layer to the second encoding module. This signal can be sent when the second module determines that only the first components of the signal are to be encoded at the second layer. Since the second encoding module receives only a subset of the components from the first encoding module, it can encode only what it receives. This not only reduces the memory usage of the first and second encoding modules, but also reduces the computations performed by the second encoding module.

[0019] Figure 1An example encoding apparatus 100 configured to encode an input signal 110 using a hierarchical coding method is shown. In a preferred embodiment, the encoder or decoder is part of a layer-based hierarchical coding scheme or format. The term "layer" refers to the fact that a signal is encoded as a series of layers, while the term "hierarchical" refers to the fact that signal information is passed from lower layers to higher layers during encoding. In some cases, signal information related to the input signal may also be passed from higher layers to lower layers, for example, as part of a subsampling or downsampling arrangement. Examples of layer-based hierarchical coding schemes include LCEVC: MPEG-5 Part 2 LCEVC ("Low Complexity Enhanced Video Coding") and VC-6: SMPTE VC-6 ST-2117, the former described in PCT / GB2020 / 050695 (and related standard documents), and the latter described in PCT / GB2018 / 053552 (and related standard documents), all of which are incorporated herein by reference. However, the concepts shown herein are not necessarily limited to these specific hierarchical coding schemes. These concepts can also be applied to other multi-layered encoding and decoding schemes, such as those that use a base layer and an enhancement layer.

[0020] The encoding device 110 encodes the input signal 110 using at least a first layer (layer 1) of the first encoding module 120 and a second layer (layer 2) of the second encoding module 130. The input signal 110 consists of two or more components. Figure 1 The diagram illustrates three components C0, C1, and C2, each of which may contain a data plane (e.g., a 2D value array for video frames or a 1D value array for audio data). Therefore, the input signal 110 can be considered as three parallel planes within the array [C0, C1, C2]. Figure 1 The three instance components can each contain YUV channels for the video signal. Encoding device 110 can be a mobile device, such as a mobile phone, tablet, laptop, low-power portable device (e.g., a smartwatch), etc. Encoding device 110 can include a hybrid of hardware and software; for example, first encoding module 120 can include a hardware encoder (i.e., with functionality accelerated by one or more dedicated encoding chipsets), while second encoding module 130 can include a software encoder, implemented, for example, via a processor and computer program code loaded into accessible memory. In some instances, encoding device 110 can include a mobile computing device, wherein both the first and second encoding modules are implemented via a processor that processes computer program code, or both the first and second encoding modules can include dedicated chipsets. Various combinations are possible, as known from the LCEVC standard.

[0021] exist Figure 1In this implementation, the second encoding module 130 receives the input signal 110 and provides a modified version of the signal (components [C′0, C′1, C′2]) to the first encoding module 120. In the LCEVC implementation, the modified version of the input signal 110 may include a downsampled or downscaled version of the input signal, causing the first layer (layer 1) to operate at a lower spatial resolution than the second layer (layer 2). The first layer is a lower layer in a layer-based hierarchical structure and may contain layers with lower resolution, i.e., compared to the second layer. The first encoding module 120 receives the modified version of the input signal ([C′0, C′1, C′2]) and generates an encoded first stream 140. The encoded first stream may contain encoded components ([E10, E11, E12]). Although in... Figure 1 The code shows a separate encoding for each component, but in some instances, the first encoding module 120 can encode all components as a combined encoding.

[0022] exist Figure 1 In this process, the second encoding module 130 generates an encoded second stream 150. The second encoding module 130 can use the input signal 110 and the output of the first encoding module 120 to generate the encoded second stream 150. Figure 1 In this example, the second encoding module 130 receives the predicted reproduction of the signal from the first encoding module 120, which is in the form of a decoded version of the encoded first stream 150. Figure 1 The code is shown as [DE10, DE11, DE12], where a decoded version of each coded component exists in the first operating mode. If the input to the first coding module 120 in the first layer is at a first spatial resolution (i.e., forming a first quality layer), then the decoded version of the first-order coded component can also be at the same first spatial resolution. In other instances, different methods (such as different sampling parameters, different bit depths, etc.) can be used to define the quality layer. Although in Figure 1 In one instance, the first encoding module 120 provides a decoded version of the encoded first stream 150, but in other instances, the second encoding module 130 may receive the encoded first stream 150 and instruct it to be decoded as part of a second encoding. Either method can be used to allow the second encoding module 130 to access reconstructed signals from the first layer that can be used within the second encoding. Those skilled in the art familiar with the LCEVC standard will understand that the first encoding module 120 may contain a basic codec, while the second encoding module 130 may contain an LCEVC encoder. The second encoding module 130 may operate at a second spatial resolution (forming a second quality layer) and, in some cases, may involve upsampling from the first spatial resolution to the second spatial resolution.

[0023] In certain instances, such as instances similar to LCEVC, the first and second encoding modules 120 and 130 can implement different encoding methods, respectively. For example, the first encoding method can correspond to a single-layer encoding method (such as AVC, HEVC, AV1, VP9, EVC, VVC, VC-6), while the second method can correspond to a different multi-layer encoding method (such as LCEVC). In other instances, the first and second encoding modules 120 and 130 can implement the same encoding method (such as VC-6 or AVC / HEVC), respectively.

[0024] Figure 1 The instance of the second aspect differs from the implementation of an encoder for the LCEVC or VC-6 standard in that the second encoding module 130 is configured to send a control signal (CTRL) to the first encoding module 120 in order to change from a first operating mode in which all components of the signal are encoded to a second operating mode in which only a subset of the original components of the signal are encoded. The control signal instructs the first encoding module 120 to provide only the first components of the signal at the first layer to the second module. This result is indicated by the double arrow (>>) in Figure 1 After the CTRL signal indicates the second operating mode, the first encoding module 120 outputs the first decoded components [DE10] instead of the full set of decoded components ([DE10, DE11, DE12]). The second encoding module 130 therefore receives only the first decoded components and generates only a second stream containing the encoded second layer version of the first components, i.e. switches from [E20, E21, E22] to [E20]. This can contain, for example, one or more sub-layers of the enhancement stream that output only the first components. In implementations using LCEVC or similar encoding methods, the second layer encoded version of the components can contain encoded residual data for the encoded components, where the residual data, once decoded, is combined with a decoded version of the encoded first stream 140 to generate the output reconstruction. In particular instances, this can contain receiving and encoding only the luminance (Y) plane within the second encoding module 130.

[0025] Figure 2 An instance of the second aspect of the invention is shown. In this case, the second aspect forms a corresponding decoder, in which Figure 2 An instance of the second aspect of the invention is shown. In this case, the second aspect forms a corresponding decoder, in which Figure 1 A decoding apparatus 200 configured to decode a signal using the hierarchical encoding method of the encoding apparatus 100 of Figure 1 The encoded first stream 140 corresponds to the output of the first encoding module 120 in Figure 1the output of the second encoding module 130. Thus, the signal received at the decoding apparatus 200 contains a signal encoded using the first encoding module within at least a first layer and using the second encoding module within a second layer. As discussed with reference to Figure 1 the original input signal 110 received by the decoding apparatus 200 is composed of two or more components. In Figure 2 the first encoded stream 140 contains encoded versions of the three components at the first level of quality ([E10, E11, E12]). This is received by the first decoding module 220. The first decoding module 220 can contain a decoder corresponding to the first encoding module 120. The first decoding module 220 can contain a base decoder (e.g. for LCEVC) or a lowest layer (e.g. for VC-6).

[0026] In the first mode of operation, the encoded second stream 150 also contains encoded versions of the set of components (i.e. [E20, E21, E22]) for example according to the standard specification. The encoded second stream 150 is received by the second decoding module 230 which decodes the encoded second stream 150 in the first mode of operation according to the decoding process specified by the standard (e.g. as specified for the enhancement stream in LCEVC or for the tier in VC-6).

[0027] Figure 2 A second mode of operation is shown. In the second mode of operation, the second decoding module 230 receives a subset of the encoded components. For example, in Figure 2 the second decoding module 230 receives only the E20 component, as shown in Figure 1 encoded in the second mode of operation 130. Thus, the second decoding module 230 decodes only a subset of the encoded components. As mentioned above, the encoded second stream 150 can contain an encoded residual data stream. In the second mode of operation, the second decoding module 230 can decode only a set (i.e. subset) of residual data for one component. In Figure 2 the example of FIG. 2, the second decoding module 230 receives three decoded components ([DE10, DE11, DE12]) from the first decoding module 220, but in the second mode of operation, only a single decoded component data stream is used to output the reconstructed signal 240. The reconstructed signal 240 is a reconstructed version of the input signal 110. It can be output (at least initially) at the same level of quality (e.g. spatial resolution) as the input signal 110. For example, using a scheme such as LCEVC or VC-6, this can contain adding a plane of decoded residual data for only the decoded component, without adding a plane of decoded residual data for the other components within the full set of components. For example, residual data can be added to only the luminance (Y) plane, and the other chrominance planes can be reconstructed without residual data. In Figure 2In this case, three reconstructed components are shown— [C'"0, C"1, C"2]— where each reconstructed component can contain a plane of component data (e.g., color values and / or sound channel values) at the second level of quality, but the plane of component data C'"0 is reconstructed in a different manner than the planes of the other component data C"1 and C"2. Using data sent within the encoded second stream 150, the plane of component data C'"0 can have undergone a further set of enhancements. As previously mentioned, the first level of quality of the first layer can involve a first resolution, while the second level of quality of the second layer can involve a second, higher resolution (in one or more dimensions).

[0028] In one case, the decoding device 200 is a passive device and simply decodes and reconstructs based on a set of received encoded streams. For example, if encoded component data is not present in the encoded second stream 150 (e.g., as shown for components 1 and 2), that data is not used for reconstruction. In these cases, the received decoded first level data DE11 and DE12 can be upsampled to the second level of quality without adding any additional residual data; however, the decoded first level data DE10 for the first component can be upsampled, and then the decoded second level data DE20 can be added to the upsampled first component data.

[0029] In another case, even if the second decoding module 230 receives encoded data for all three components in the encoded second stream 150, it can discard data for one or more components based on local processing conditions. For example, if resources are constrained at the decoding device 200, only one component can be decoded and used for the output reconstructed signal 240.

[0030] In examples described herein, one or more of the decoding devices and encoding devices can be mobile devices, such as mobile phones, tablets, laptops, low-power portable devices (e.g., smart watches), etc. In one case, a device can contain both encoding and decoding devices, such as a mobile phone holding a video conference can simultaneously encode and decode video streams, or a voice assistant can simultaneously encode and decode audio streams.

[0031] In certain examples, the control signal (CTRL) described above is sent when the second module determines that only a first component of the signal is encoded at the second layer. For example, it can be an optional signal, and in the absence of the signal, encoding is performed according to a standardized process, such as LCEVC or VC-6. Thus, examples described herein can contain optional “non-standard” enhancements that do not affect standardized encoding or decoding; it can be added as an optional feature into certain devices (e.g., mobile or resource-constrained devices).

[0032] Figure 3An example method 300 of determining whether component encoding can be performed is shown. At block 310, resource conditions are determined. The resource conditions can include a need to encode a signal for a low power service. The low power service can include a video conferencing service. The resource conditions can relate to one or more of: processing capacity, power capacity (e.g., for a battery device), and memory capacity. The processing capacity can relate to one or more of central processing unit (CPU) and graphics processing unit (GPU) capacity. The memory capacity can relate to volatile memory capacity (e.g., random access memory) and / or non-volatile memory capacity (e.g., file storage). The capacity can also relate to bit capacity for encoding streams, e.g., the number of bits available to encode at a target bit rate. The capacity can be measured using resource utilisation, e.g., the percentage of clock cycles used or memory capacity used.

[0033] At block 320, the resource conditions determined at block 310 are evaluated to determine whether resource usage is to be reduced. This can be performed by comparing the measured resource conditions to defined thresholds. For example, this can include reducing power consumption, e.g., based on battery capacity falling below a threshold, or reducing CPU / GPU load requirements, e.g., based on exceeding a threshold utilisation. The condition can include a requirement to reduce the number of processing operations to be performed in signal encoding. The processing operations can include reads and / or writes to memory. For example, these can be memory copy operations.

[0034] Based on the evaluation at block 320, one of blocks 330 or 340 is selected. If resource usage is not to be reduced, e.g., because one or more resource metrics are within acceptable ranges, then at block 330, the full set of components are encoded at a second encoding module, such as 130 in Figure 1 In this case, the second encoding module is used to encode the full set of components, e.g., as in the case of a video conferencing service where the full set of components is to be encoded for transmission to a remote device. If resource usage is to be reduced, then at block 340, the components are encoded at a first encoding module, such as 120 in Figure 1No signal may be sent between blocks 120. Instead, a control signal indicating that all components will be encoded may be sent. If reduced resource usage is required, for example because one or more resource metrics are outside acceptable limits (or one or more other conditions are met), a determination to reduce the components encoded at the second encoding module is made at block 340. This may involve the second encoding module sending a control signal to the first encoding module to reduce the encoded components used by the second encoding module. This may involve omitting decoding operations on the omitted component set at the first encoding module (or another corresponding first decoding module) and / or not passing decoding signals for the omitted component set to the second encoding module. The determination in method 300 may include determining a condition requiring only the first component of the signal to be provided. The subset of signal components provided after block 340 may include multiple components of the signal processed by the first encoding module, but only a subset of the signal components passed from the first encoding module to the second encoding module, such as only the first component of the signal. Providing only the first component of the signal may include the first encoding module writing only the first component of the signal to memory. In other instances, providing only the first component of the signal may include only the first component of the signal encoded by the first module.

[0035] Reducing the number of encoded components can reduce resource usage in several ways. Processing resources used for encoding and / or decoding components at one or more locations in the first and second encoding modules can be saved. Memory usage can be reduced by having only one component copied from a plurality of components into memory for access by the second encoding module. The modules described herein can be configured to flexibly encode and / or decode based on received signals, requiring minimal level of control signaling to flexibly change the encoding and decoding methods (e.g., perhaps only signals from the second encoding module to the first encoding module are needed).

[0036] In some cases, the first encoding module can implement the first encoding method, and the second encoding module can implement the second encoding method. The first encoding method may differ from the second encoding method. Alternatively, the first encoding method may be the same as the second encoding method. The first layer is at a lower level than the second layer in the hierarchical structure. For example, the first layer may have a lower resolution than the second layer.

[0037] Figure 3 This method can be combined with methods that encode signals using hierarchical coding. In this case, the signal is encoded using a first coding module at the first layer, and the signal is encoded using a second coding module at the second layer, wherein the signal consists of two or more components. For example, a method similar to... Figure 1The configuration is shown. The method may include receiving a signal at the first module from the second module, the signal indicating that the second module provides only a first component of the signal at the first layer. The second module may send the signal when it determines that only the first component of the signal is encoded at the second layer. The method may also include receiving two or more components of the signal at the first module; and the first module providing only the first component of the signal.

[0038] In this method, providing only the first component of the signal may include two or more components of the signal processed by the first module, and the first module may only pass the first component of the signal to the second module. As mentioned above, providing only the first component of the signal may include the first module only writing the first component of the signal to the memory. It may also, or alternatively, include the first module only encoding the first component of the signal. The first layer may be at a lower level than the second layer in the hierarchical structure. For example, the first layer may have a lower resolution than the second layer.

[0039] It can also provide corresponding methods for decoding signals using hierarchical coding methods. This can be based on... Figure 2 The arrangement is as follows: A first encoding module encodes the signal at a first layer, and a second encoding module encodes the signal at a second layer. The signal consists of two or more components. The method includes receiving a first processing signal at a decoding module, the first processing signal being processed by the first encoding module. In this case, the first processing signal contains only a first component of the signal, and wherein the first processing signal is generated based on a signal sent from the second encoding module to the first encoding module instructing the first encoding module to provide only the first component of the signal. The method may further include decoding the second encoded signal by the decoding module to generate a decoded signal, the second encoded signal being encoded by the second encoding module. The method may further include combining the second decoded signal into the first processing signal by the decoding module. The first encoded signal corresponds to the signal encoded at the first layer, and the second encoded signal corresponds to the signal encoded at the second layer. Therefore, the method can provide a similar... Figure 2 The functions shown.

[0040] The method of encoding the signal can also be performed by the first encoding module in a set of encoding modules. In this case, the first encoding module can receive the signal from the second encoding module, for example, as shown below. Figure 1 As shown, the signal can instruct the first encoding module to provide only a first component (or a subset of the components). From the perspective of the first encoding module, the method can involve receiving two or more components of the signal at the first encoding module, and having the first encoding module provide only the first component of the signal. For example, the first encoding module can be controlled to write only one encoded and / or decoded component to memory.

[0041] In another example, there is provided a method of encoding a signal using a hierarchical encoding method, wherein the signal is encoded at a first layer using a first encoding module and at a second layer using a second encoding module, and wherein the signal is comprised of two or more components, the method comprising sending the signal from the first encoding module to the second encoding module to instruct the second encoding module to encode only a first component of the signal at the second layer. In this case, the signal can be sent from the first encoding module to the second encoding module. For example, the signal can be sent when the first encoding module determines that only the first component of the signal is to be encoded at the second layer. The determination can comprise determining a condition requiring only the first component of the signal to be provided. The condition can comprise encoding the signal for a low power service. The low power service can comprise a video conferencing service. The condition can comprise a requirement to reduce power consumption. The condition can comprise a requirement to reduce the number of processing operations to be performed in encoding the signal. The processing operations comprise reading and / or writing to memory. For example, these can be memory copy operations. The first encoding module can implement a first encoding method, and the second encoding module can implement a second encoding method. The first encoding method can be different to the second encoding method. The first encoding method can be the same as the second encoding method.

[0042] According to one particular implementation, a signal processor (e.g., computer processor hardware) is configured to receive and encode a signal composed of multiple planes ("encoder"). For example, the planes can correspond to color planes in a video or image signal, such as a luminance plane (Y) and two chrominance planes (U and V). The encoder produces a signal rendition at a first level of quality (e.g., a lower level) for each plane (e.g., color plane) of the signal, and encodes it with a first encoding method. It then produces a predicted rendition of the signal at a second level of quality (e.g., a higher level), and correspondingly produces and encodes a layer of residual data (e.g., a tier) at the second level of quality to apply to the predicted rendition of the signal at the second level of quality in order to produce a corrected rendition of the signal at the second level of quality. The predicted rendition of the signal can be generated by applying a scaling process (e.g., upscaling) to the rendition of the signal at the first level of quality. Upon detecting that chrominance processing should be limited to the lower level of quality, the encoder can generate and encode a tier of residual data at the second level of quality only for the luminance component of the signal, but not for layers (e.g., tiers) of residual data at the second level of quality for the chrominance components of the signal. The residual data can be encoded with a second encoding method. In one embodiment, the first encoding method and the second encoding method are the same encoding method. In different embodiments, the first encoding method and the second encoding method are different. Similar methods can be applied to multi-channel audio data, where residual data can be provided for certain audio channels only at a higher level of quality (e.g., higher sampling or bit rate or wider frequency range). In this case, audio output devices that typically output human speech, such as center and front speakers, can have corresponding audio channels (i.e., components) encoded by the second encoding module, and audio output devices that typically output non-speech sounds, such as surround and subwoofer speakers, can receive components encoded only by the first encoding module (e.g., components reconstructed by the second processing module without encoded elements from the enhancement stream). This can save resources, but have minimal impact on sound perception.

[0043] In a corresponding particular decoder implementation, a signal processor configured as a decoder receives an encoded signal to obtain a rendition of the signal at a first (lower) quality level and to produce a predicted rendition of the signal at a second (higher) quality level, the second quality level having a higher resolution (i.e., signal sampling rate) than the first quality level. The predicted rendition of the signal can be generated by applying a scaling process (e.g., upscaling) to the rendition of the signal at the first quality level. The decoder can then receive and decode one or more tiers of residual data to apply to the predicted rendition of the signal to produce a corrected rendition of the signal at the second quality level. When it is detected that there is no tier of encoded residual data for one or more chroma planes of the signal, the decoder outputs the predicted rendition of the plane at the second quality level for the chroma plane. In some instances, bits in the encoded bitstream signal to the decoder whether there is residual data at a given quality level for a chroma plane.

[0044] In certain instances, the encoder is configured to not process and encode the layers (e.g., tiers) of residual data for the chroma planes at the second quality level in the case of a particular application, such as by way of non-limiting example, a video conference. In other non-limiting embodiments, the encoder is configured to not process and encode the tiers of residual data for the chroma planes at the second quality level in the case that the remaining battery drops below a threshold.

[0045] According to certain instances described herein, a signal processor is configured to receive a signal and encode it with a hybrid-layer based encoding method, such as by way of non-limiting example, MPEG-5 Part 2 LCEVC (Low Complexity Enhancement Video Coding) or SMPTE VC-6 ST2117. The encoder receives the signal, down-samples it to a lower quality level, produces a first (lower) quality level rendition of the signal for each color plane of the signal, and encodes it with a codec implementing a first encoding method. In some instances, the codec implementing the first encoding method is a hardware codec. The encoder then receives a decoded reconstruction of the first encoding process from the hardware codec, produces a predicted rendition of the signal at a second (higher) quality level, and produces and encodes the tiers of residual data at the second quality level accordingly to apply to the predicted rendition of the signal at the second quality level in order to produce a corrected rendition of the signal at the second quality level. When it is detected that chroma processing should be limited to the lower quality level, the encoder signals to the codec implementing the first encoding method that chroma residual data at the higher quality level will not be produced. As a result, the codec implementing the first encoding method will not provide the decoded reconstruction of the chroma planes at the first quality level to the encoder.

[0046] In some instances, when the encoder receives a signal indicating that no chroma residual data will be generated, the codec implementing the first encoding method will not perform a mem-copy operation to provide the encoder with a decoded reconstruction of the chroma plane at a first quality level, thereby saving processing power and battery consumption. Accordingly, the encoder will not perform memory operations and computational operations on the chroma plane, further saving processing power. In another embodiment, an instantiation of an encoding pipeline is provided to allow real-time disabling of the encoding of the chroma plane as described in this specification.

[0047] In some instances, in response to the detection that a specific use case requires higher quality reconstruction, the encoder is configured to process the residual data across all chroma planes and signal to the codec implementing the first encoding method that all chroma reconstructions at the first quality level will be required.

[0048] Figure 4 It shows Figure 1 A variant 400 of the encoding device 100 is specifically designed for LCEVC implementations. In this case, according to LCEVC, the second layer is divided into at least two sub-layers. These are in Figure 4 The sublayers are shown as sublayer 1 and sublayer 2. Those familiar with the LCEVC specification will recognize that these can be implemented by enhancing sublayers in one or more directions with potentially different spatial resolutions depending on the coding configuration (e.g., sublayer 1 may have the same or higher resolution as the first layer, while sublayer 2 may have a higher resolution than sublayer 1). Figure 4 The code includes a first encoding module 420 and two sub-layer encoding modules. The first encoding module 420 may contain a basic codec used with the LCEVC encoder, while the two sub-layer encoding modules contain enhanced (layer 2) encoders—sub-layer 1 encoding module 432 and sub-layer 2 encoding module 434. Each sub-layer encoding module 432 and 434 is configured similarly to... Figure 1 The corresponding coded sub-layer streams 452 and 454 are generated in the manner shown. The second coded stream containing coded sub-layers 452 and 454 may contain an LCEVC coded enhancement stream, and the first coded stream 440 may contain a coded basic stream.

[0049] exist Figure 4 In an example, one or more of the first encoding module 420, sublayer 1 encoding module 432, and sublayer 2 encoding module 434 may be instructed to encode a subset of the signal components, as described herein. Figure 4In this case, there is a cascade of control signals, with the sublayer 1 encoding module 432 sending a first control signal CTRL1 to the first encoding module 420, and the sublayer 2 encoding module 434 sending a second control signal CTRL2 to the sublayer 1 encoding module 432. Other control configurations (e.g., a serial control signal from additional control components) can also be used. Thus, one or more of the first encoding module 420, the sublayer 1 encoding module 432, and the sublayer 2 encoding module 434 can be controlled to encode only a subset of the components, and this can be followed by a higher level module in the hierarchical structure. Figure 4 The sublayer 2 encoding module 434 is shown signaling the sublayer 1 encoding module 432 to encode only one component (e.g., only the first component such as the luminance signal), so that the sublayer 2 encoding module 434 receives a prediction reconstruction for only the selected one component, rather than for the full set of components. In these instances, both sublayer encoding modules can be controlled as described with reference to the first and second encoding modules of Figure 1 The sublayer 2 encoding module 434 is shown signaling the sublayer 1 encoding module 432 to encode only one component (e.g., only the first component such as the luminance signal), so that the sublayer 2 encoding module 434 receives a prediction reconstruction for only the selected one component, rather than for the full set of components. In these instances, both sublayer encoding modules can be controlled as described with reference to the first and second encoding modules of

[0050] In preferred instances, when resources are limited, a particular subset of components can be selected for encoding. For example, for color components, it has been found that encoding residual data for only the luminance or contrast plane, and not the chrominance planes, results in an improved video quality perception at this level of quality, but uses considerably less resources (e.g., 33% of the encoding resources) than encoding no residual data. While the quality is best when all components are encoded, this can not be possible when resources are limited, such as when an application is occupying processing resources during a video call, or when a mobile phone has a low battery charge; in these cases, reducing the components for encoding can help to slow resource consumption, but provide enough quality to continue the call. Also, the systems and methods discussed herein can be applied flexibly and dynamically during encoding, without needing to stop or start a video stream, which means that falling back to a reduced number of components is modest, and can provide a place to fall back to a lower quality level that provides an improved visual experience immediately.

[0051] The techniques described herein can be implemented in software or hardware, or can be implemented using a combination of software and hardware. They can include configuring a device to perform and / or support any or all of the techniques described herein.

[0052] The above embodiments should be understood to be illustrative only. Further embodiments are contemplated. It should be understood that any feature described in relation to any one embodiment can be used alone, or in combination with other features described, and can be used in combination with one or more features of any of the other embodiments, or any combinations of any other embodiments. Furthermore, equivalents and modifications not described above can also be employed without departing from the scope of the application, which is defined in the following claims.

Claims

1. A method of encoding a signal using a hierarchical encoding method, wherein the signal is encoded at a first layer using a first encoding module and at a second layer using a second encoding module, and wherein the signal is comprised of two or more components, the method comprising: sending a signal from the second module to the first module to instruct the first module to provide only a first component of the signal to the second module at the first layer, wherein sending the signal when the second module determines that only the first component of the signal is to be encoded at the second layer, and wherein the determining comprises determining a condition requiring only the first component of the signal to be provided.

2. The method of claim 1, wherein the condition comprises a requirement to encode the signal for a low power service.

3. The method of claim 2, wherein the low power service comprises a video conference.

4. The method of claim 1, wherein the condition comprises a requirement to reduce power consumption.

5. The method of claim 1, wherein the condition comprises a requirement to reduce a number of processing operations to be performed in the encoding of the signal.

6. The method of claim 5, wherein the processing operations comprise reads and / or writes to memory.

7. The method of any one of claims 1-6, wherein the first encoding module implements a first encoding method and the second encoding module implements a second encoding method.

8. The method of claim 7, wherein the first encoding method is different from the second encoding method.

9. The method of claim 7, wherein the first encoding method is the same as the second encoding method.

10. The method of any one of claims 1-6, wherein the first layer is at a lower level in a hierarchy than the second layer.

11. The method of claim 10, wherein a resolution of the first layer is lower than a resolution of the second layer.

12. The method of any one of claims 1-6, further comprising: receiving the two or more components of the signal at the first module; and providing only the first component of the signal by the first module to the second module.

13. The method of any one of claims 1-6, wherein providing only the first component of the signal comprises: processing the two or more components of the signal by the first module; and passing only the first component of the signal by the first module to the second module.

14. The method of any one of claims 1-6, wherein providing only the first component of the signal comprises: writing only the first component of the signal by the first module to memory.

15. The method of any one of claims 1-6, wherein providing only the first component of the signal comprises: encoding only the first component of the signal by the first module.

16. A method of encoding a signal using a hierarchical encoding method, wherein the signal is encoded at a first layer using a first encoding module and at a second layer using a second encoding module, and wherein the signal is comprised of two or more components, the method comprising: receiving a signal at the first module from the second module, the signal indicating that the second module is providing only a first component of the signal at the first layer, wherein, sending the signal when the second module determines that only the first component of the signal is encoded at the second layer, and wherein the determining comprises determining a condition requiring only the first component of the signal to be provided.

17. The method of claim 16, further comprising: receiving the two or more components of the signal at the first module; and providing only the first component of the signal by the first module.

18. The method of claim 16 or 17, wherein providing only the first component of the signal comprises: processing the two or more components of the signal by the first module; and passing only the first component of the signal by the first module to the second module.

19. The method of claim 16 or 17, wherein providing only the first component of the signal comprises: writing only the first component of the signal to memory by the first module.

20. The method of claim 16 or 17, wherein providing only the first component of the signal comprises: encoding only the first component of the signal by the first module.

21. A method of decoding a signal using a hierarchical encoding method, wherein the signal is encoded at a first layer using a first encoding module and at a second layer using a second encoding module, and wherein the signal is comprised of two or more components, the method comprising: receiving a first processed signal at a decoding module, the first processed signal processed by the first encoding module, and wherein the first processed signal contains only a first component of the signal, and wherein the first processed signal is generated by providing only the first component of the signal based on a signal sent from the second encoding module to the first encoding module; indicating the first encoding module to provide only the first component when the second module determines that only the first component of the signal is encoded at the second layer, wherein the determining comprises determining a condition requiring only the first component of the signal to be provided.

22. The method of claim 21, further comprising: decoding a second encoded signal by the decoding module to produce a decoded signal, the second encoded signal encoded by the second encoding module.

23. The method of claim 22, further comprising: combining the second decoded signal to the first processed signal by the decoding module.

24. The method of any of claims 21-23, further comprising that the first encoded signal corresponds to the signal encoded at the first layer and the second encoded signal corresponds to the signal encoded at the second layer.

25. An encoding device configured to encode a signal using a hierarchical encoding method, wherein the signal is encoded using a first encoding module at at least a first layer and using a second encoding module at a second layer, and wherein the signal consists of two or more components, the encoding device comprising the first encoding module and the second encoding module, wherein the encoding device is configured to implement the method according to any of claims 1 to 20.

26. A decoding device configured to decode a signal using a hierarchical encoding method, wherein the signal is encoded using a first encoding module at at least a first layer and using a second encoding module at a second layer, and wherein the signal consists of two or more components, the decoding device comprising a decoding module, wherein the decoding device is configured to implement the method according to any of claims 21 to 24.

27. A method of encoding a signal using a hierarchical encoding method, wherein the signal is encoded using a first encoding module at a first layer and using a second encoding module at a second layer, and wherein the signal consists of two or more components, the method comprising: sending a signal from the first module to the second module to instruct the second module to encode only a first component of the signal at the second layer, wherein, sending the signal when the second module determines to encode only the first component of the signal at the second layer, and wherein the determining comprises determining a condition requiring only the first component of the signal to be provided.

28. The encoding device of claim 25, wherein the encoding device is a mobile device.

29. The decoding device of claim 26, wherein the decoding device is a mobile device.

30. The method of any of claims 1 to 24, wherein the signal is a video signal, the components comprise luminance and chrominance components, and the first component comprises the luminance component.

Images