Techniques for providing chroma format scalability in image processing applications

Abstract

Techniques are disclosed for representing videos and images with full resolution color component information and still compatible with conventional processing systems that process images with reduced resolution information. The new image representation may include a scalable format composed of a base layer that is encoded to match the expectations of a legacy encoder. The new image representation may also include an additional enhancement layer that enables up-conversion of reduced resolution color components to higher resolution. The image representation not only provides power savings when decoding the full resolution representation, but also provides other benefits, such as scalable and power aware decoding.

Description

[0001] CLAIM OF PRIORITY

[0002] This application claims the priority benefit of U.S. Application Serial Nos. 18 / 772,408 and 63 / 519,306, both titled “Techniques For Providing Chroma Format Scalability In Image Processing Applications,” filed July 15, 2024 and August 14, 2023, respectively, the disclosures of which are incorporated herein in their entireties. BACKGROUND

[0003] The present disclosure relates to digital techniques for representing image information, and in particular to techniques for multi-color image information.

[0004] In modern computing applications, there are a variety of ways to represent multi-color image information. In many cases, image information is represented by a plurality of orthogonal color components (sometimes referred to as “planes”). For example, a multi-color image can be represented by red, green, and blue components in an “RGB” color space. In another example, the same multi-color image can be represented in a Y-Cr-Cb color space by a luminance and two chrominance color components. Standards have been developed for managing the representation of color in images, which has facilitated the exchange of images in modern computing applications.

[0005] 4:2:0 chroma format is currently the most popular chroma sampling format in consumer-facing video applications. In this format, each frame of a video sequence is represented with a luminance component and two chrominance components. However, the two chrominance components are represented with half the resolution compared to the luminance component, both vertically and horizontally. This occurs because, for most content, the characteristics of the chrominance signals allow for a reduction in their resolution with limited impact on image quality. This reduction in resolution can help reduce memory storage and bandwidth, and provide some compressibility benefits when compressing a video sequence (e.g., a scaling down process can help reduce some noise that can be present in an original 4:4:4 full resolution representation of each chrominance component, making it easier to compress the chrominance data). Alternative reduced resolution formats, such as a 4:2:2 format, are also known.

[0006] There are applications that expect higher resolution chroma information, such as screen sharing, gaming, and still image photography. In some cases, reducing chroma resolution compared to luminance resolution can cause artifacts around object edges / boundaries, especially when edges have significant chromatic aberration (e.g., a red object near a blue background). Chroma subsampling can also exacerbate chroma leakage during compression. Since chroma leakage caused by subsampling is particularly visible in HDR content, several techniques, such as luminance adjustment methods for chroma conversion, have been proposed to suppress such artifacts. Users often want to maintain the highest quality version of their content (which may include full-resolution chroma samples), for example, for archiving purposes, while distributing lower-resolution versions to others when needed.

[0007] Unfortunately, while full-resolution chroma features are highly desirable, most currently deployed consumer devices (such as set-top box decoders, mobile devices, computers, etc.) only support hardware decoding of chroma formats up to 4:2:0. While software decoding of natively encoded 4:4:4 content is possible, such software applications may consume excessive power in battery-powered devices or may be impractical in real-time encoding applications involving certain resolutions and frame rates. Attached Figure Description

[0008] Figure 1 An example of a coding system according to one embodiment of the present disclosure is shown.

[0009] Figure 2 An exemplary image is shown according to one embodiment of this disclosure.

[0010] Figure 3 An example of processing according to one embodiment of this disclosure is illustrated.

[0011] Figure 4 The process flow between the base layer image and the enhancement layer image according to one embodiment of the present disclosure is illustrated.

[0012] Figure 5 An exemplary image file is shown according to one embodiment of this disclosure.

[0013] Figure 6 An exemplary image file is shown according to another embodiment of this disclosure.

[0014] Figure 7 An exemplary image file is shown according to another embodiment of this disclosure.

[0015] Figure 8 This is a data flow diagram illustrating the encoded data flow between a base layer image and an enhancement layer image according to one embodiment of the present disclosure.

[0016] Figure 9 This is a data flow diagram illustrating the decoding data flow between a base layer image and an enhancement layer image according to one embodiment of the present disclosure.

[0017] Figure 10 An example of a coding system according to another embodiment of this disclosure is shown.

[0018] Figure 11 An example of a processing flow according to one embodiment of this disclosure is provided.

[0019] Figure 12 A process flow according to one embodiment of this disclosure is illustrated.

[0020] Figure 13 A decoding system according to one embodiment of the present disclosure is illustrated.

[0021] Figure 14 An example of a coding system according to another embodiment of this disclosure is shown. Detailed Implementation

[0022] The embodiments disclosed herein provide techniques for representing video and images using full-resolution color component information while remaining compatible with conventional processing systems that utilize reduced-resolution information to process images, such as the 4:2:2 and / or 4:2:0 representations popular in luma-chroma image representations. The image representation may include a scalable format consisting of base layers, where image data is encoded to match the expectations of a conventional encoder. The image representation may also include additional enhancement layers supporting upconversion of reduced-resolution color components to higher resolutions. The image representation not only provides power savings when decoding full-resolution representations but also offers other benefits such as scalable and power-aware decoding.

[0023] The following discussion presents the techniques proposed in this disclosure within the context of systems encoding images in a luma-chroma color plane. As discussed herein, the luma-chroma representation of images and / or videos (referred to as "images" for convenience) is typically represented in a 4:2:0 format, where the chroma image components are represented at a reduced resolution compared to the luma color components. However, the principles of this disclosure can be extended to other image formats, where it may be desirable for one color component to be represented at a reduced resolution compared to another. The use of luma-chroma examples in the following discussion should not be construed as limiting the application of the proposed techniques to any particular color space.

[0024] Figure 1An encoding system 100 according to one embodiment of this disclosure is illustrated. System 100 may include a downsampler 110, a base layer buffer 120, an upsampler 130, a residual generator 140, and an enhancement layer buffer 150. System 100 can accept image data represented in multiple colors; for images received in a non-luminance / chrominance format, a color plane converter 160 can convert the image from its native format to a luminance-chrominance color format. Therefore, in this discussion, the source image is described as being input to system 100 in a format in which the luminance, Cr, and Cb color components have the same resolution as each other (even if the source image is converted to this format upon input).

[0025] Downsampler 110 can downsample the resolution of the chroma components to a lower resolution that conforms to the color format followed by the base layer image. Therefore, in an implementation using the 4:2:2 format, downsampler 110 can downsample the chroma components (Cr, Cb) such that each chroma component has half the resolution of its corresponding luminance component in the horizontal direction. Similarly, in an implementation using the 4:2:0 format, downsampler 110 can downsample the chroma components (Cr, Cb) such that each chroma component has half the resolution of its corresponding luminance component in both the horizontal and vertical directions. Downsampler 110 can output the downsampled Cr and Cb chroma data to the base layer buffer 120.

[0026] The base layer buffer 120 can store luminance component data and downsampled chrominance component data until they are to be transferred to a file. The data stored in the base layer buffer 120 can form the base layer image of the image to be generated by the system 100. Figure 2 Typically, the base layer data is compressed by encoder 170 before being stored in a file. Compression may occur according to interoperability coding standards such as the ISO / ITU-T HEVC / H.265 standard or AOMedia's Video 1 standard (often referred to as AV1). In such cases, the downsampling provided by downsampler 110 may conform to a resolution suitable for the image data used by encoder 170 (e.g., 4:2:2, 4:2:0, or another resolution). In practice, encoder 170 may be an encoding system provided by a processing device on which system 100 operates.

[0027] Upsampler 130 can upsample downsampled chromaticity data from base layer buffer 120 to a higher resolution. For example, Cr and Cb chromaticity data can be upsampled from a 4:2:2 or 4:2:0 resolution stored in base layer buffer 120 to a full-resolution format (e.g., 4:4:4). Upsampler 130 can operate according to a predefined scaling technique (such as Lanczos5, bilinear, bicubic, or some other upsampler). Alternatively, system 100 can dynamically select parameters of upsampler 130 and provide metadata identifying the selected parameters in a file. The upsampling technique takes into account the chromaticity position type compared to the luminance chromaticity position type, i.e., whether the chromaticity position type is equal to 0, 1, 2, etc., which can affect the phase of the amplifier used.

[0028] The residual generator 140 generates residual signals of Cr and Cb chromaticity data based on a comparison between the upsampled Cr and Cb chromaticity signals and the source Cr and Cb chromaticity signals at the system input. These Cr and Cb chromaticity residual signals can be input to the enhancement layer buffer 150. These Cr and Cb chromaticity residual signals form the basis of the enhancement layer image of the source image.

[0029] Figure 2 Examples are given of those that can be generated by Figure 1 An exemplary image file 200 generated by system 100. The file is shown as including a base layer image 210 and one or more enhancement layer images 220, 230. The base layer image 210 may include a luminance plane 212 having information corresponding to the luminance components of the source image, and an enhancement plane 212 having information corresponding to the luminance components of the source image generated by downsampler 110. Figure 1 The generated Cr plane 214 contains information corresponding to the downsampled Cr chromaticity components, and the Cb plane 216 contains information corresponding to the downsampled Cb chromaticity components of the source image. The luminance component 212 of the base layer image 210, as well as the Cr chromaticity components 214 and Cb chromaticity components 216, can represent the source image using a reduced-resolution representation such as 4:2:0.

[0030] Enhancement layer images 220 and 230 may include color residuals at a higher resolution, possessing information corresponding to one or more reduced-resolution color components 214 and 216 from the base layer image 210. Continuing with the 4:2:0 example above, the Cr plane 214 and Cb plane 216 of the base layer image 210 may have half the resolution horizontally and vertically relative to the luminance plane 212 of the base layer image 210. The Cr chromaticity residual enhancement layer image 220 and Cb chromaticity residual enhancement layer image 230 can provide information corresponding to the Cr and Cb chromaticity residuals generated by the residual generator 140. Enhancement layer images 220 and 230 can provide information from which full-resolution Cr and Cb chromaticity residuals can be derived.

[0031] Figure 2 The example illustrates two enhancement layer images 220 and 230, each corresponding to one of the Cr and Cb chromaticity residuals generated by the residual generator 140. It is not required that two enhancement layer images 220 and 230 be generated in all cases. The principles of this disclosure apply to specific implementations that generate enhancement layer data for a single color component. System designers are expected to customize system 100 of this disclosure (…). Figure 1 ) and file format ( Figure 2 The application of these technologies can be tailored to their individual needs.

[0032] The principles of this disclosure can be applied to provide multiple levels of scalability that may be desired. Figure 2 The image 200 is illustrated with dashed lines as an example of such an implementation, wherein image 200 comprises a plurality of Cr enhancement layer images 220, 240 and a plurality of Cb enhancement layer images 230, 250. For example, it may be desirable to provide image content in a base layer representation (e.g., 4:2:0), image content in an intermediate representation (e.g., 4:2:2), and image content in a full-resolution representation (e.g., 4:4:4). In this application, the first pair of residual enhancement layer images 240, 250 may provide residual information of reduced-resolution image data (Cr and Cb chromaticity in this example) in an intermediate-resolution representation. The first residual enhancement layer images 240, 250 may be generated by a first set of residual generators 140 ( Figure 1 The residual enhancement layer images 220 and 230 are differentially derived from the base layer representation. The second pair of residual enhancement layer images 220 and 230 can provide residual information from the reduced-resolution image data (in this example, Cr and Cb chromaticity) at a higher resolution. The second residual enhancement layer images 220 and 230 can be generated by another set of residual generators ( Figure 1 (Not shown) These relationships are differentially derived from the first residual enhancement layer images 240 and 250. The relationships between the base layer representation of the reduced-resolution color information and the increasingly higher-resolution representation of the enhancement layer images can be repeated for as many resolutions as possible.

[0033] As another example of multi-level scalability, one or more enhancement layer images can support increased resolution of regions of interest (ROIs) within an image. A ROI can be a spatial region of the source image that is determined to contain image data that a human viewer is likely to be interested in. Figure 2 An example of an exemplary region of interest 260 in a brightness plane 212 may be associated with a corresponding location in a source image (not shown).

[0034] In such applications, a first scalability layer may be provided by enhancement layer images 220 and 230, which provide enhancement information for the entire spatial region of the source image, and a second scalability layer may be provided by enhancement layer images 240 and 250 corresponding to the spatial region of the region of interest 260. Enhancement layer images 220 and 230 provide enhancement information for the entire spatial region of the source image, which increases the resolution of the resulting image (e.g., from 4:2:0 to 4:2:2) when decoded together with the content of the base layer 210. ROI enhancement layer images 240 and 250 provide enhancement information that increases the resolution of the region corresponding to the region of interest 260 to a maximum resolution (e.g., from 4:2:0 to 4:4:4) when decoded together with the content of the base layer image 210. ROI enhancement layer images 240 and 250 may differentially encode the spatially overlapping content relative to enhancement layer images 220 and 230 (which themselves may be differentially encoded relative to the base layer image 210). In another specific implementation, the ROI enhancement layer images 240 and 250 can be differentially encoded directly from the base layer image 210 (e.g., without considering the enhancement layer images 220 and 230).

[0035] The principles of this disclosure apply to systems in which the base layer encoder 180 supports monochrome encoding, such as by using an HEVC monochrome profile or by using an HEVC master / master 10 profile in which the chroma plane is discarded during encoding. In such an application, the Cr chroma plane 214 and Cb chroma plane 216 of the base layer image 210 will be empty. The Cr residual enhancement layer image 220 and Cb residual enhancement layer image 230 will contain a full-resolution representation of the chroma information (e.g., they are not residuals).

[0036] The principles of this disclosure also apply to systems in which the Cr residual enhancement layer 220 and the Cb residual enhancement layer 230 are not differentially encoded relative to the upsampled Cr and Cb chromaticity information from the base layer image 210. For example, the upsampler 130 and the residual generator 140 can be omitted. Figure 1 The enhancement layer buffer 150 may, as appropriate, obtain its Cr and Cb chroma inputs from the source image or color plane converter 160. Compared to implementations where the Cr residual enhancement layer 220 and Cb residual enhancement layer 230 contain residual information, this implementation may result in lower compression efficiency, but it may benefit from reduced complexity during decoding. Furthermore, the system implementation can switch between encoding in which the upsampler 130 and residual generator 140 are enabled and encoding in which the residual generator 140 is disabled; encoding system 100 ( Figure 1 It can provide metadata to identify the status of the residual generator 140 used for encoding.

[0037] In one implementation, image file 200 may be represented using a High Efficiency Image File (HEIF) format defined by MPEG, for example, in ISO / IEC 23008-12 (MPEG-H Part 12). Specifically, implementations may use the concept of derived image items and alternative groups in this format. For example, base layer image 210 may be stored as the primary item in the HEIF file. Enhancement layer images 220, 230, 240, and 250 may be placed in the "altr" alternative group, indicating that base layer image 210 (e.g., having a 4:2:0 format) and enhancement layer images 220, 230, 240, and 250 are alternative forms of each other.

[0038] Figure 3 The processing flow 300 between the components of the base layer image 310 and the enhancement layer images 320, 325 according to one embodiment of the present disclosure is illustrated. Figure 3 The process for generating enhancement layer images 320 and 325 from base layer image components and source chroma data is illustrated. As discussed, base layer image 310 may include a luminance plane 312, a Cr plane 314, and a Cb plane 316, and enhancement layer images 320 and 325 may include Cr chroma residuals and Cb chroma residuals, respectively. Corresponding upsamplers 330 and 335 upsample component data from Cr plane 314 and Cb plane 318. A pair of comparators 340 and 345 generate Cr and Cb chroma residual signals from the upsampled chroma data. Specifically, the first comparator 340 compares the upsampled Cr chroma data generated by upsampler 330 with the source Cr chroma data, and the second comparator 345 compares the upsampled Cb chroma data generated by upsampler 335 with the source Cb chroma data. These chromaticity residual signals can be used as the basis for the Cr chromaticity residual in enhancement layer image 320 and the Cb chromaticity residual in enhancement layer image 325.

[0039] Figure 4 A process flow 400 between a base layer image 410 and enhancement layer images 420, 425 according to one embodiment of the present disclosure is illustrated. Figure 4An example of a process for recovering source chromaticity data from a base layer image 410 and an enhancement layer image 420 is illustrated. As discussed, the base layer image 410 may include a luminance plane 412, a Cr plane 414, and a Cb plane 416, and the enhancement layer images 420, 425 may include Cr chromaticity residuals and Cb chromaticity residuals. Corresponding upsamplers 430 and 435 may upsample the component data from the Cr plane 414 and the Cb plane 418. A pair of adders 440 and 445 may generate recovered Cr chromaticity signals 440 and 412 from the upsampled Cr and Cb chromaticity data and the Cr chromaticity residuals 420 and 425. Specifically, the first adder 440 adds the upsampled Cr chromaticity data generated by the upsampler 430 to the Cr chromaticity residual 420, and the second comparator 445 adds the upsampled Cb chromaticity data generated by the upsampler 435 to the Cb chromaticity residual 425. Therefore, full-resolution Cr and Cb chromaticity components can be recovered from the base layer image 410 and the enhancement layer image 420.

[0040] return Figure 1 The enhancement layer data stored in buffer 150 can be compressed by encoder 180, which may, but does not necessarily, operate according to the same encoding protocol as encoder 170. For example, the base layer image may have been encoded using HEVC Master or Master 10 profile or even 8 or 10-bit AV1 Master profile, and the enhancement layer image may use AVC Non-Interleaved High profile or HEVC Master only (8-bit) or even JPEG. Other codecs / encoding specifications may be used. Metadata information may be signaled along with the encoded image, informing the decoder how each layer is encoded and how each layer can be combined with the base layer.

[0041] In applications, the enhancement layer image will likely primarily contain residual data that is typically shifted to the center of the bit-depth representation (i.e., if the encoded representation is 8 bits, a value of 128 is added to the residual signal and its amplitude is limited to 0 to 255). Therefore, for encoder 180, reducing the dynamic range of the signal and encoding it with a lower bit depth may be appropriate. Different bit depths can be used for different layers, and scaling of samples can also be used to increase accuracy, and this can be different for each layer.

[0042] In one implementation, encoder 180 can operate on the chroma component data after it has been packaged into a virtual image format by image enhancement unit 190. Image enhancement unit 190 can arrange the Cr and Cb chroma component data in a format that presents the component data to encoder 180 as if it were luminance data. Encoder 180 can apply an encoding protocol to the virtual image presented by image enhancement unit 190, which can result in the generation of a file with an enhancement layer image in an alternative format. Two examples of alternative formats are provided below. Figure 3 and Figure 4 As shown in the image.

[0043] exist Figure 5 In the example, the enhancement layer image 520 contains several elements: an encoded luminance element 522, a dummy Cr chrominance element 526, and a dummy Cb chrominance element 528. The encoded luminance element 522 may contain elements generated by the residual generator 140 ( Figure 1 The generated Cr chromaticity residuals 523 and Cb chromaticity residuals 524 are packaged into virtual luminance elements and encoded by encoder 180 to produce luminance element 522. In this application, encoder 180 can apply its encoding protocol to the Cr chromaticity residuals 523 and Cb chromaticity residuals 524 as if these chromaticity residuals constitute an image of luminance information. Figure 5 In the example, the Cr chromaticity residual 523 and the Cb chromaticity residual 524 are stacked perpendicularly to each other; therefore, the virtual luminance element can have twice the image height of the content constituting the luminance plane 512.

[0044] exist Figure 5 In a specific implementation, the dummy Cr chromaticity image 526 and Cb chromaticity image 528 may contain empty data. It is expected that when the dummy Cr chromaticity image 526 and Cb chromaticity image 528 are encoded by the encoder 180, they will have a very small bit size.

[0045] Metadata field 530 may identify the processing performed by enhancement layer packing unit 190 and / or encoder 180. For example, metadata 530 may identify the packing relationship between Cr chromaticity residual 523 and Cb chromaticity residual 524 within a virtual luminance image. In a specific implementation where system 100 can dynamically select the encoding type to be applied, metadata 530 may also identify the encoding type applied by encoder 180.

[0046] exist Figure 6 In the example, the enhancement layer image 620 also includes encoded luminance elements 622, dummy Cr chrominance elements 626, and dummy Cb chrominance elements 628. The encoded luminance element 622 may contain elements generated by the residual generator 140 (…). Figure 1The generated Cr chromaticity residuals 623 and Cb chromaticity residuals 624 are packaged into virtual luminance elements and encoded by encoder 180 to produce luminance element 622. Figure 5 As in the implementation plan, encoder 180 can apply its encoding protocol to Figure 6 The Cr chromaticity residual 623 and Cb chromaticity residual 624 in the implementation scheme are as if these chromaticity residuals constitute an image of luminance information. Figure 6 In the example, the Cr and Cb chromaticity residuals 623 and 624 are placed horizontally adjacent to each other; therefore, the luminance element can have twice the width of the image as the content constituting the luminance plane 612.

[0047] Metadata field 630 may identify the processing performed by enhancement layer packing unit 190 and / or encoder 180. For example, metadata 630 may identify the packing relationship between Cr chromaticity residual 623 and Cb chromaticity residual 624 within a virtual luminance image (e.g., their horizontal placement relative to each other). In a specific implementation where system 100 can dynamically select the encoding type to be applied, metadata 630 may also identify the encoding type applied by encoder 180.

[0048] exist Figure 6 In a specific implementation, the dummy Cr chromaticity image 626 and Cb chromaticity image 628 may contain empty data. It is expected that when the dummy Cr chromaticity image 626 and Cb chromaticity image 628 are encoded by the encoder 180, they will have a very small bit size.

[0049] Figure 7 An example is shown that can be reinforced by packaging unit 190 ( Figure 1 Another packing relationship is formed by residual generator 140. In this example, it can be achieved through residual generator 140. Figure 1 The spatial interleaving of the Cr and Cb chromaticity residuals generated by the encoder 180 is used to form virtual luminance elements. On one hand, the spatial elements can be selected to match the encoding of the virtual luminance elements by the encoder 180. Figure 1 The spatial granularity used. For example, Cr and Cb chromaticity residuals can be interleaved at the superblock granularity, coding unit granularity, or other granularities that facilitate efficient encoding by encoder 180.

[0050] Metadata field 730 may identify the processing performed by enhancement layer packing unit 190 and / or encoder 180. For example, metadata 730 may identify the packing relationship between Cr chromaticity residual 723 and Cb chromaticity residual 724 within a virtual luminance image (e.g., their interleaving relationship relative to each other and the granularity at which they are selected). In a specific implementation where system 100 can dynamically select the encoding type to be applied, metadata 730 may also identify the encoding type applied by encoder 180.

[0051] As in other embodiments, the dummy Cr chroma image 726 and Cb chroma image 728 may contain empty data. It is expected that when the dummy Cr chroma image 726 and Cb chroma image 728 are encoded by encoder 180, they will have a very small bit size.

[0052] Figure 8 This is a data flow diagram illustrating the data flow between a base layer image 810 and enhancement layer images 820, 830 according to an embodiment of the present disclosure. In this embodiment, the base layer image 810 includes a luma plane 812, a chromaticity plane 814, and a chromaticity plane 816 arranged in a conventional representation such as 4:2:0 or 4:2:2. The first chromaticity enhancement layer image 820 may include a virtual luma plane 822, a virtual chromaticity plane 824, and a virtual chromaticity plane 826. The second chromaticity enhancement layer image 830 may include a virtual luma plane 832, a virtual chromaticity plane 834, and a virtual chromaticity plane 836. Each of the base layer image 810, the chromaticity enhancement layer image 820, and the chromaticity enhancement layer image 830 may be encoded by a corresponding encoder 840, 850, 860.

[0053] exist Figure 8 In the example, the Cr chromaticity data (not shown) of the source image is packed into the luminance plane 822 of the Cr enhancement layer image 820. The dummy Cr chromaticity field 824 and dummy Cb chromaticity field 826 of the Cr enhancement layer image 820 may contain dummy data. The Cr enhancement layer image 820 can be presented to the first enhancement layer encoder 850 in a conventional representation (e.g., 4:2:0 or 4:2:2), which allows the Cr enhancement layer image 820 to be encoded by a conventional encoder used in conventional consumer electronics devices.

[0054] Similarly, the Cb chromaticity data (also not shown) of the source image can be packed into the luminance plane 832 of the Cb enhancement layer image 830. The dummy Cr chromaticity field 834 and dummy Cb chromaticity field 836 of the Cb enhancement layer image 830 may contain dummy data. The Cb enhancement layer image 830 can be presented to the second enhancement layer encoder 860 in a conventional representation (e.g., 4:2:0 or 4:2:2), which allows the Cb enhancement layer image 830 to be encoded by a conventional encoder used in conventional consumer electronics devices.

[0055] Figure 8 Also shown is a multiplexer 870, which can organize encoded data from encoders 840, 850, and 860 according to the requirements of file storage protocols such as HEIF.

[0056] Figure 9 This is an example of... Figure 8The data flow diagram illustrates the decoding process of the base layer image and enhancement layer image generated from the data stream. In this embodiment, the demultiplexer 910 can parse the data from the file into encoded components representing the encoded base layer image and enhancement layer image, and forward these components to the corresponding decoders 920, 930, and 940. The base layer decoder 920 can reverse the data generated by the base layer encoder 840 (… Figure 8 The base layer decoder 950 performs an encoding process and generates a recovered base layer image 950 from it. The base layer decoder 950 can operate according to a conventional decoding process, in which case it can produce a decoded image 950 in a conventional representation such as 4:2:0 or 4:2:2.

[0057] The first enhancement layer decoder 930 is reversible by the first enhancement layer encoder 850 ( Figure 8 The first enhancement layer decoder 930 performs an encoding process and generates a recovered Cr enhancement layer image 960 from it. The first enhancement layer decoder 930 can also operate according to a conventional decoding process, in which case it can produce a decoded image 960 in a conventional representation such as 4:2:0 or 4:2:2. Due to the Cr enhancement layer image 820 ( Figure 8 The Cr enhancement layer image 960 contains Cr chromaticity data of the source image packed into a virtual luminance plane 822, and therefore the recovered Cr enhancement layer image 960 may also contain recovered Cr chromaticity data packed into a virtual luminance plane 962 of the Cr enhancement layer image 960. The virtual Cr chromaticity plane 964 and Cb chromaticity plane 966 of the Cr enhancement layer image 960 may contain dummy data.

[0058] The second enhancement layer decoder 940 is reversible by the second enhancement layer encoder 860 ( Figure 8 The second enhancement layer decoder 940 performs an encoding process and generates a recovered Cb enhancement layer image 970 from it. The second enhancement layer decoder 940 can also operate according to a conventional decoding process, in which case it can produce a decoded image 970 in a conventional representation such as 4:2:0 or 4:2:2. Due to the Cb enhancement layer image 830 ( Figure 8 The Cb enhancement layer image 970 contains Cb chromaticity data of the source image packed into the virtual luminance plane 832 of the Cb enhancement layer image 970, and therefore the recovered Cb enhancement layer image 970 may also contain recovered Cb chromaticity data packed into the virtual luminance plane 972 of the Cb enhancement layer image 970. The virtual Cr chromaticity plane 974 and Cb chromaticity plane 976 of the Cb enhancement layer image 970 may contain dummy data.

[0059] Image reconstructor 980 generates reconstructed image 990 from the recovered base layer image 950 and enhancement layer images 960, 970. As discussed, virtual luminance planes 962, 972 of the Cr enhancement layer image 960 and the Cb enhancement layer image 970 may contain recovered Cr chromaticity data and Cb chromaticity data, respectively. The data in those luminance planes 962, 972 can represent the Cr and Cb components at full resolution (e.g., matching the resolution of the luminance data contained in the luminance plane 952 of the recovered base layer image 950). Image reconstructor can derive reconstructed image 990 at full resolution (e.g., 4:4:4) from the full-resolution luminance representation contained in the luminance plane 952 of the base layer image 950, the full-resolution Cr chromaticity representation contained in the luminance plane 962 of the first enhancement layer image 960, and the Cb chromaticity representation contained in the luminance plane 972 of the second enhancement layer image 970. Of course, in processing applications where lower resolution image information is appropriate (e.g., a 4:2:0 representation of the source image is sufficient), the decoder can decode the base layer image 950 without processing any encoded enhancement layer images from the file.

[0060] Figure 9 and Figure 8 The data flow diagram illustrates the enhancement layer images 820 and 830 ( Figure 9 ) and 960, 970 ( Figure 10 The Cr and Cb chromaticity information of the base layer image 810 is not differentially formed with respect to the Cr chromaticity information 814 and Cb chromaticity information 816 of the base layer image 810.

[0061] Figure 1 An encoding system 1000 according to another embodiment of this disclosure is illustrated. System 1000 may include a downsampler 1010, a base layer buffer 1020, an encoder / decoder 1030, an upsampler 1040, a residual generator 1050, an enhancement layer buffer 1060, and an encoder 1070. Figure 2 As in the implementation scheme, system 1000 can accept image data represented in multiple colors; for images received in a non-luminance / chrominance format, they can be converted to a luminance / chrominance color format. Therefore, system 1000 is shown to process source images in which the luminance, Cr, and Cb color components have the same resolution as each other.

[0062] Downsampler 1010 downsamples the resolution of the chroma components to a lower resolution that conforms to the color format followed by the base layer image. Therefore, in an implementation using the 4:2:2 format, downsampler 1010 downsamples the chroma components (Cr, Cb) such that each chroma component has half the resolution of its corresponding luminance component in the horizontal direction. Similarly, in an implementation using the 4:2:0 format, downsampler 1010 downsamples the chroma components (Cr, Cb) such that each chroma component has half the resolution of its corresponding luminance component in both the horizontal and vertical directions. Downsampler 1010 outputs the downsampled Cr and Cb chroma data to the base layer buffer 1020.

[0063] The base layer buffer 1020 can store luminance component data and downsampled chrominance component data until they need to be transferred to a file. The data stored in the base layer buffer 1020 can form the base layer image of the image to be generated by the system 1000. Figure 2 In this implementation, the base layer data is compressed by encoder 1070 before being stored in a file. Compression may occur according to an interoperability coding standard such as HEVC or AV1. In such cases, the downsampling provided by downsampler 1010 may conform to a resolution suitable for the image data used by encoder 1070 (e.g., 4:2:2, 4:2:0, or another resolution). In practice, encoder 1070 may be an encoding system provided by a processing device on which system 1000 operates.

[0064] The encoder / decoder 1030 encodes the downsampled Cr and Cb chromaticity signals according to the encoding algorithm applied by the encoder 1070, and decodes the encoded signals. Many encoding algorithms are lossy encoding processes, which cause encoding loss during signal encoding and decoding. Therefore, the encoder / decoder 1030 can output Cr and Cb chromaticity signals that represent the Cr and Cb chromaticity signals input to the encoder / decoder 1030 but exhibiting some encoding errors. The encoding errors introduced by the encoder / decoder 1030 are likely similar to those caused by the decoding system (not shown) when the image file is decoded.

[0065] Upsampler 1040 can upsample the downsampled Cr and Cb chromaticity signals input from encoder / decoder 1030 to the upsampler to a higher resolution. For example, Cr and Cb chromaticity data can be upsampled from a 4:2:2 or 4:2:0 resolution existing in base layer buffer 1020 to a full-resolution format (e.g., 4:4:4). Similarly, upsampler 1040 can operate according to predefined scaling techniques (such as Lanczos 5, bilinear, bicubic, or some other upsampler). The upsampling technique considers the chromaticity position type compared to the luminance chromaticity position type, i.e., whether the chromaticity position type is equal to 0, 1, 2, etc., which can affect the phase of the amplifier used.

[0066] The residual generator 1050 generates residual signals of Cr and Cb chromaticity data based on a comparison between the upsampled Cr and Cb chromaticity signals and the source Cr and Cb chromaticity signals at the system input. These Cr and Cb chromaticity residual signals can be input to the corresponding enhancement layer buffer 1060. These Cr and Cb chromaticity residual signals form the basis of the enhancement layer image of the source image.

[0067] The enhancement layer data stored in enhancement layer buffer 1060 can be compressed by encoder 1090, which can (but does not necessarily) operate according to the same encoding protocol as in encoder 1070. Encoder 1090 can operate directly on the Cr and Cb chromaticity components, in which case the enhancement layer image 220 ( Figure 2 It can have, for example Figure 5 to Figure 7 The illustrated object is for the Cr plane 214 and the Cb plane 216. Alternatively, the encoder 1090 may be as described regarding Figure 11 Either of the discussed methods uses a virtual image generated by the enhancement layer packing unit 1080 for operation.

[0068] In an alternative embodiment, encoder / decoder 1030 may be implemented simply as a decoder that reverses the encoding operation performed by encoder 1070. In this embodiment, encoded Cr and Cb chromaticity data may be input from encoder 1070 to decoder 1030 (the path shown by the dashed line).

[0069] Figure 11 The processing flow 1100 between the components of the base layer image 1110 and the enhancement layer images 1120, 1125 according to one embodiment of the present disclosure is illustrated. Figure 12The process for generating enhancement layer images 1120 and 1125 from base layer image components and source chroma data is illustrated. As discussed, base layer image 1110 may include a luma plane 1112, a chroma plane 1114, and a Cb plane 1116; first enhancement layer image 1120 may include chroma residuals; and second enhancement layer image 1125 may include Cb chroma residuals. Corresponding encoder / decoder units 1130 and 1135 may encode and then decode the chroma signals 1114 and 1116 from base layer image 1110. Corresponding upsamplers 1140 and 1145 may upsample the decoded component data from chroma plane 1114 and Cb plane 1118. A pair of comparators 1150 and 1155 may generate chroma residual signals chroma and Cb from the upsampled chroma data. Specifically, the first comparator 1150 compares the upsampled Cr chromaticity data generated by the upsampler 1140 with the source Cr chromaticity data, and the second comparator 1155 compares the upsampled Cb chromaticity data generated by the upsampler 1145 with the source Cb chromaticity data. These chromaticity residual signals form the basis of the Cr chromaticity residual enhancement layer image 1120 and the Cb chromaticity residual enhancement layer image 1125.

[0070] Figure 12 A process flow 1200 between a base layer image 1210 and enhancement layer images 1220, 1225 according to one embodiment of the present disclosure is illustrated. Figure 13 A process for recovering source chroma data from a base layer image 1210 and enhancement layer images 1220, 1225 is illustrated. As discussed, the base layer image 1210 may include a luminance plane 1212, a chroma plane 1214, and a Cb plane 1216; the first enhancement layer image 1220 may include a chroma residual, and the second enhancement layer image 1225 may include a Cb chroma residual. Corresponding decoders 1230, 1235 may decode the chroma signals 1214 and 1216 from the base layer image 1210. Corresponding upsamplers 1240, 1245 may upsample the decoded chroma data from the decoders 1230, 1235. A pair of adders 1250 and 1255 can generate recovered Cr chromaticity signals 1260 and 1265 from upsampled Cr and Cb chromaticity data and Cr chromaticity residuals 1220 and 1225. Specifically, the first adder 1250 can add the upsampled Cr chromaticity data generated by the upsampler 1240 to the Cr chromaticity residual 1220, and the second comparator 1255 can add the upsampled Cb chromaticity data generated by the upsampler 1245 to the Cb chromaticity residual 1225. Therefore, full-resolution Cr and Cb chromaticity components can be recovered from the base layer image 1210 and the enhancement layer images 1220 and 1225.

[0071] Figure 1 An example of a decoding system 1300 for generating a restored image with full-resolution chroma according to one embodiment of the present disclosure is illustrated. System 1300 may include a base layer decoder 1310, which can be used to invert the encoder of the system that created the base layer image (…). Figure 10 , Figure 2 The encoding process applied to the base layer image ( Figure 5 to Figure 7 , Figure 1 Therefore, for a base layer image encoded according to HEVC technology, the base layer image can also be decoded according to HEVC technology. Furthermore, for a base layer image encoded according to AV1 technology, the base layer image can be decoded according to AV1 technology. The base layer decoder 1310 can output the recovered luminance, Cr, and Cb chrominance data. This data is compared with the source image from which the recovered image data is generated (…). Figure 10 , Figure 2 In contrast, the recovered image data output by the base layer decoder 1310 may have reduced resolution Cr and Cb chromaticity. For example, the recovered image data from the base layer decoder 1310 may have a 4:2:2 or 4:2:0 format suitable for the base layer decoder 1310.

[0072] System 1300 may also include an enhancement layer decoder 1320, an image repackaging unit 1330, an adder 1340, and a unit for processing the enhancement layer image. Figure 5 to Figure 7 , Figure 1 The upsampler 1350. The enhancement layer decoder 1320 can decode the enhancement layer image to invert the image generated by the enhancement image encoder (such as...). Figure 10 encoder 180 or Figure 1 The image repackaging unit 1330 can reformat the decoded data output from the enhancement layer decoder 1320 to reverse the processing techniques performed by the enhancement image packing unit 190. Figure 10 ) or 1080 ( Figure 5 to Figure 7 The formatting operation performed. Therefore, in a specific implementation where the virtual luminance image is formed and encoded from the Cr and Cb chromaticity residuals, such as... Figure 1 As shown in the example, the image repackaging unit 1330 can reformat a virtual luminance image into Cr and Cb chromaticity residuals.

[0073] Upsampler 1350 performs upsampling operations on the Cr and Cb chromaticity signals output from base layer decoder 1310. The upsampling operation can simulate... Figure 3 to Figure 4 , Figure 10 and / or Figure 4The upsampling operation performed by the upsampler shown is illustrated. As a result of the upsampling operation, the Cr and Cb chromaticity signals output from the base layer decoder 1310 can be converted to a higher resolution representing the Cr and Cb residual signals from the enhancement layer decoder 1320. The adder 1340 can generate a full-resolution restored Cr and Cb chromaticity signal from the upsampled Cr and Cb chromaticity signal from the upsampler 1350 and the Cr and Cb residual signal from the enhancement layer decoder 1320. Therefore, the system 1300 can output the luminance signal from the base layer decoder 1310 and the restored Cr and Cb chromaticity signal from the adder as a full-resolution representation of the source image from which the generated file is derived. Therefore, when the base layer image can represent the source image in a 4:2:2 or 4:2:0 format, the system 1300 can output the restored image in a 4:4:4 format when using full-resolution Cr or Cb chromaticity data such as that restored by the adder 1340. The data stream of the system 1300 can be, for example, Figure 12 or Figure 14 As shown.

[0074] Figure 1 An encoding system 1400 according to another embodiment of this disclosure is illustrated. System 1400 may include a downsampler 1410, a base layer buffer 1420, a color converter 1430, one or more component processors 1440.1-1440.n, an enhancement layer buffer 1450, and a base layer encoder 1460. Figure 2 As in the implementation scheme, system 1400 can accept image data represented in multiple colors; for images received in a non-luminance / chrominance format, they can be converted to a luminance / chrominance color format. Therefore, system 1400 is shown to process source images in which the luminance, Cr, and Cb color components have the same resolution as each other.

[0075] Downsampler 1410 downsamples the resolution of the chroma components to a lower resolution that conforms to the color format followed by the base layer image. Therefore, in an implementation using the 4:2:2 format, downsampler 1410 downsamples the chroma components (Cr, Cb) such that each chroma component has half the resolution of its corresponding luminance component in the horizontal direction. Similarly, in an implementation using the 4:2:0 format, downsampler 1410 downsamples the chroma components (Cr, Cb) such that each chroma component has half the resolution of its corresponding luminance component in both the horizontal and vertical directions. Downsampler 1410 outputs the downsampled Cr and Cb chroma data to the base layer buffer 1420.

[0076] The base layer buffer 1420 stores luminance component data and downsampled chrominance component data until it needs to be transferred to a file. The data stored in the base layer buffer 1420 forms the base layer image of the image to be generated by the system 1400. Figure 10 In this implementation, the base layer data is compressed by encoder 1470.

[0077] Encoder 1460 can encode luminance data and downsampled Cr and Cb chromaticity data before system 1400 stores the base layer image in a file. As in previous embodiments, compression can occur according to an interoperability coding standard such as HEVC or AV1. In such cases, the downsampling provided by downsampler 1410 can conform to a resolution suitable for the image data used by encoder 1470 (e.g., 4:2:2, 4:2:0, or another resolution). In practice, encoder 1460 can be an encoding system provided by a processing device on which system 1400 operates.

[0078] Color converter 1430 converts the source luminance component and downsampled Cr and Cb chrominance components from a luminance / chrominance color space to an alternative color space, such as the red / green / blue color space or the Y'UV color space. Other representations can be used, such as different RGB representations with different primary colors and different transfer characteristics (e.g., from YCbCr BT.709 to RGB BT.2100 PQ). Color converter 1430 can output component data to component processors 1440.1-1440.n. When color converter 1430 has the ability to convert input image data to multiple color spaces, color converter 1430 can output metadata identifying the selected color space.

[0079] System 1400 may have multiple component processors 1440.1-1440.n, providing a component processor for each color component of the color space to which color converter 1430 converts its input data. Each component processor 1440.1, ..., 1440.n may have an upsampler 1442 and a filter system 1444. As in previous embodiments, upsampler 1442 may upsample its corresponding color component to the full resolution of the source image. Upsampler 1440 may upsample the downsampled Cr and Cb chromaticity signals input from encoder / decoder 1430 to the upsampler to a higher resolution. For example, Cr and Cb chromaticity data may be upsampled from a 4:2:2 or 4:2:0 resolution existing in base layer buffer 1420 to a full-resolution format (e.g., 4:4:4). Filter system 1444 may apply filtering operations to the upsampled color components. The filtered output from filter 1444 may be stored in enhancement layer buffer 1450 and may form the basis of the enhancement layer image.

[0080] System 1400 may also have an image packing unit 1470 that arranges component data stored in enhancement layer buffer 1450 into a virtual image, which can be encoded by encoder 1480. As in previous embodiments, encoder 1480 may (but does not have to) operate according to a compression standard different from that of encoder 1460.

[0081] In one implementation, system 1400 may further include, for example: Figure 14 The encoder / decoder described in the implementation scheme (not shown). In this specific implementation, the encoder / decoder can operate on the data from the base layer buffer 1420 before it is input to the color converter 1430.

[0082] because Figure 1 The implementation scheme transforms the enhanced color information into an alternative color space, so the Cr and Cb components are likely to contribute to each component in the alternative color space. In such applications, the decoder (not shown) can select which decoding layers to decode based on its local computing resources. For example, in some cases, the decoder may choose to decode only the base layer and skip decoding all enhancement layers (e.g., when a user is viewing an image and scrolling or zooming very quickly), or decode only one component from the enhancement layers instead of all components.

[0083] In the aforementioned embodiments, the base layer image and enhancement layer image can be stored in a container format such as an HEIF container. In the case of an HEIF container, for backward compatibility and to allow older players to render "traditional" 4:2:0 images, the base layer image (e.g., a 4:2:0 image) can be stored as the primary item in the HEIF file, and the enhancement layer image can be stored in an alternative group. Alternatively, both image items (the base layer image and the enhancement layer image) can be placed in an "altr" alternative group, indicating that the images are alternative forms of each other. Placing the base layer image as the first image in the altr group and the enhancement layer image as the secondary image facilitates backward compatibility with decoders not programmed to recognize enhancement layer images, as decoders are typically programmed to ignore data representations they do not recognize. However, systems programmed to understand enhancement layer images can access, decode, and display larger resolutions (e.g., 4:4:4).

[0084] In another embodiment, the system according to the aforementioned embodiment can cascade operations to generate a series of enhancement layer images. For example, the "altr" group may have an image at 4:4:4 resolution, another image at 4:2:2 resolution, and another image at 4:2:0 resolution. In the cascaded operation, the 4:4:4 version can be derived from the 4:2:2 version, and the 4:2:2 version can be derived from the 4:2:0 version. Figure 1 In the context of the implementation, the first-level upsampler 130 may be programmed to scale the input image data to an intermediate resolution (e.g., from 4:2:0 to 4:2:2), and the second-level upsampler (not shown) may scale the data stored in the enhancement layer buffer 150 (e.g., from 4:2:2 to 4:4:4). For this purpose, a second sequence of upsamplers, residual generators, and enhancement layer buffers may be provided. Figures 1 to 14 (Not shown in the image).

[0085] In another implementation, additional features such as HDR may be included. For example, the "altr" group may have 4:4:4HDR, 4:2:0 HDR, and 4:2:0 SDR (primarily), or 4:4:4 HDR, 4:4:4 SDR, and 4:2:0 SDR. In the first case, the HDR-enhanced image is exported first, followed by 4:4:4 generation, while in the second case, the 4:4:4 SDR-enhanced image is exported first, followed by the HDR-enhanced image. For all these cases, a closed-loop conversion can be employed at the encoder end to achieve improved performance.

[0086] The foregoing discussion describes various embodiments of this disclosure within the context of encoding systems, decoding systems, and the process flows they employ. In practice, these systems can be applied to a variety of devices, such as mobile devices with integrated video cameras (e.g., telephones, entertainment systems, and computers with camera functionality) and / or wired communication systems such as video conferencing equipment and camera-enabled desktop computers. In some applications, the functional blocks described above can be provided as elements of an integrated software system, where the blocks can be provided as elements of a computer program, stored in memory as program instructions, and executed by a general-purpose processing system. In other applications, the functional blocks can be provided as discrete circuit components of a processing system, such as functional units within a digital signal processor or application-specific integrated circuit. Other applications of the invention can be embodied as hybrid systems of dedicated hardware and software components. Furthermore, it is not necessary to provide the functional blocks described herein as separate components. For example, although... ​The illustrations depict components of a video encoder and decoder as separate units; however, in one or more embodiments, some or all of them may be integrated, and they need not be separate units. Unless otherwise stated, these implementation details are not essential to the operation of the invention.

[0087] Furthermore, the accompanying drawings illustrated herein provide only the necessary details to illustrate the subject matter of the invention. In practice, video encoders and decoders will typically include functional units other than those described herein, including buffers for storing data in the illustrated encoding pipeline and communication transceivers for managing communication with the communication network and the corresponding encoder / decoder devices. For clarity, such elements have been omitted from the above discussion.

[0088] This document specifically illustrates and / or describes several embodiments of the invention. However, it should be understood that modifications and variations of the invention are covered by the foregoing teachings and are within the scope of the appended claims without departing from the spirit and intended scope of the invention.

Claims

1. An encoding method, the encoding method comprising: Encoding a multi-color source image using scalable coding representation, wherein the scalable coding representation includes: A base layer image, comprising an encoded representation of each of a plurality of color components, wherein at least the first color component has a reduced resolution compared to the resolution of the second color component. An enhancement layer image, the enhancement layer image containing an encoded representation of the first color component, the encoded representation being at an increased resolution relative to the reduced resolution of the first color component.

2. The method of claim 1, wherein the encoded representation of the first color component is at a resolution that matches the resolution of the second color component in the base layer image.

3. The method of claim 1, wherein the encoded representation of the first color component is an intermediate resolution between the resolution of the second color component in the base layer image and the reduced resolution of the first color component in the base layer image.

4. The method of claim 1, wherein the encoded representation of the first color component in the enhancement layer is differentially encoded relative to the encoded representation in the base layer image.

5. The method of claim 1, wherein the encoded representations of the first color components in the enhancement layer image are all derived non-differentially from the source image relative to the first color components in the base layer image.

6. The method of claim 1, further comprising a second enhancement layer image, the second enhancement layer image containing an encoded representation of the first color component, the encoded representation of the first color component being at an intermediate resolution between the resolution of the first color component in the base layer image and the resolution of the first color component in the first enhancement layer image.

7. The method of claim 1, further comprising a second enhancement layer image, the second enhancement layer image containing an encoded representation of the first color component, the encoded representation of the first color component being located at a spatial position corresponding to a region of interest in the source image.

8. The method of claim 1, further comprising forming the enhancement layer image by: The first color component is upsampled from the reduced resolution of the base layer image. A residual is generated by comparing the first color component upsampled from the base layer image with the resolution of the source image.

9. The method of claim 8, further comprising, prior to the upsampling, encoding the reduced resolution first color component data by a compression technique matching the compression technique used to encode the base layer image, and then performing decision processing.

10. The method of claim 1, further comprising forming the enhancement layer image by: The first color component is converted from the color space of the source image to an alternative color space. The converted color component data in the enhancement layer image is encoded.

11. The method of claim 6, further comprising bandwidth compression of the residual.

12. The method according to claim 1, wherein: The encoded representation of the base layer image is formed by bandwidth compression of the color components of the base layer image, and The encoded representation of the enhancement layer image is formed by bandwidth compression of the color components of the enhancement layer image.

13. The method of claim 12, wherein the bandwidth compression of the base layer image color components and the bandwidth compression of the enhancement layer image color components are performed according to the same operability criterion.

14. The method of claim 1, wherein the base layer image and the enhancement layer image are each represented in a corresponding HEIF alternative group.

15. The method of claim 1, wherein the first color component and the second color component are members of a luminance-chrominance color space.

16. The method of claim 1, wherein the base layer image represents the source image in a 4:2:0 format.

17. The method of claim 1, wherein the base layer image represents the source image in a 4:2:2 format.

18. The method of claim 1, wherein the base layer image and the enhancement layer image represent the source image in a 4:4:4 format.

19. An encoding method, the encoding method comprising: Encoding a multi-color source image using scalable coding representation, wherein the scalable coding representation includes: A base layer image, comprising an encoded representation of each of a plurality of color components, wherein the first and second color components each have a reduced resolution compared to the resolution of the third color component. An enhancement layer image, the enhancement layer image comprising encoded representations of the first color component and the second color component, the encoded representations being at an increased resolution relative to the reduced resolution of the base layer image.

20. The method of claim 19, further comprising forming the enhancement layer image by: Upsampling is performed on the reduced-resolution representation of the first color component from the base layer image. The first color component residual is generated by comparing the upsampled first color component of the base layer image with the resolution of the source image. The reduced-resolution representation of the second color component from the base layer image is upsampled. A second color component residual is generated by comparing the upsampled second color component of the base layer image with the resolution of the source image; and A virtual image is formed by spatially merging the residuals of the first color component and the second color component, and The virtual image is encoded into the enhancement layer image.

21. The method of claim 20, wherein the formation places the first color component residual and the second color component residual horizontally adjacent to each other.

22. The method of claim 20, wherein the formation places the first color component residual and the second color component residual perpendicularly adjacent to each other.

23. The method of claim 20, wherein the formation causes the first color component residual and the second color component residual to be spatially interleaved.

24. An encoding method, the encoding method comprising: Encoding a multi-color source image using scalable coding representation, wherein the scalable coding representation includes: A base layer image, the base layer image comprising an encoded representation of the first color component of the source image according to a monochrome encoding algorithm. An enhancement layer image, which contains encoded representations of different second color components of the source image. The base layer image and the enhancement layer image provide recovered source image data when decoded, in which the first color component and the second color component have a common resolution.

25. A decoding method, the decoding method comprising: In response to determining that the image to be recovered at a basic resolution level is to be obtained, a base layer image is decoded to obtain the recovered image, the base layer image containing an encoded representation of each of a plurality of color components of the source image, at least the first color component having a reduced resolution compared to the resolution of the second color component. In response to determining that the restored image should be obtained at an enhanced resolution level: The base layer image, representing the encoded representation of each of the plurality of color components containing the source image, is decoded. Decoding the enhancement layer image containing the encoded representation of the first color component yields the residual of the first color component, the encoded representation being at a resolution increased relative to the reduced resolution of the first color component, and The decoded base layer image and the enhancement layer image are merged to obtain the recovered image.

26. An encoder, the encoder comprising: A base layer encoder having inputs representing color component data of a source image. Enhancement layer coding system, the enhancement layer coding system comprising: An upsampler having an input for data corresponding to a first color component and an output for upsampling the first color component data, the first color component being at a resolution corresponding to the base layer resolution of the color component, and A residual generator having a first input coupled to the output of the upsampler, a second input for first color component data from the source image, and an output for residual first color component data.

27. The encoder of claim 26, further comprising an encoder / decoder having an input for data corresponding to a first color component and an output coupled to the upsampler input, the first color component being at a resolution corresponding to the base layer resolution of the color component.

28. The encoder of claim 26, further comprising a second upsampler and a second residual generator for data of the second color component.

29. The encoder of claim 28, further comprising a virtual image generator having an input coupled to the output of the residual generator for the first color component and the second color component.

Images