Encoding single channel into multi-channel container followed by image compression

Abstract

Encoding methods and apparatus for packing single-channel data into a multi-channel container (e.g., MP4, TIFF or JPEG container) to at least achieve good utilization of container data capacity. In some examples, a method of encoding includes converting a plurality of scalar values of a received data stream to a corresponding plurality of n-dimensional values, the converting being performed using a mapper; assigning each of the n-dimensional values as a pixel value to a respective pixel of the virtual image frame, where n is an integer greater than one; and compressing the virtual image frame according to the type of the image data container. The mapper is configured to map scalar values to corresponding n-dimensional values based on a relationship represented by an n-dimensional curve or by a plurality of 2n-way tree partitions of an n-dimensional space.

Description

[0001] 1. Cross - reference to related applications

[0002] This application claims the priority benefits of U.S. Provisional Application No. 63 / 407,885, filed on September 19, 2022, and European Application No. 23151686.5, filed on January 16, 2023, and each of these applications is hereby incorporated by reference in its entirety. 2. Technical Field

[0003] Various example embodiments generally relate to video / image compression, and more specifically but not exclusively to video / image encoding and decoding. 3. Background Art

[0004] Compression can reduce the amount of memory required to store video / images and the amount of bandwidth required to transmit video / images. The Moving Picture Experts Group (MPEG) codec based on the H.264 compression standard is an example codec for this purpose. Other codecs are also available on the market. Many codecs are designed to be compatible with various professional, consumer, and mobile cameras. Summary of the Invention

[0005] Disclosed herein are various methods and apparatuses for packing single - channel data into a multi - channel container (e.g., an MP4, TIFF, or JPEG container) to at least achieve good utilization of the container data capacity. In various embodiments, the packing is performed based on a mapping relationship represented by an n - dimensional curve or a multi - 2 n way tree (e.g., an octree when n = 3) partitioning of an n - dimensional space. At least some embodiments are compatible with existing hardware of a traditional video / image transmission pipeline, i.e., do not inherently rely on modifying any of its hardware / infrastructure. Instead, such embodiments can be advantageously implemented in software and / or firmware, e.g., by interfacing corresponding additional data - processing modules with existing codecs without changing the (multiple) native container formats of the codecs.

[0006] According to an example embodiment, there is provided an encoding method, the method comprising: converting, using a processor, a plurality of scalar values of a received data stream into corresponding plurality of n - dimensional values, the conversion being performed using a mapper; assigning, using the processor, each of the n - dimensional values as a pixel value to a corresponding pixel of a virtual image frame, where n is an integer greater than one; and compressing, using the processor, the virtual image frame according to the type of an image data container; and wherein the mapper is configured to map scalar values to corresponding n - dimensional values based on a relationship represented by an n - dimensional curve or a multi - 2 n way tree partitioning.

[0007] According to another exemplary embodiment, a non-transitory computer-readable medium storing instructions is provided, which when executed by an electronic processor cause the electronic processor to perform operations including the above method.

[0008] According to yet another exemplary embodiment, an apparatus for encoding image data is provided, the apparatus including: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus to at least: convert a plurality of scalar values of a received data stream into corresponding plurality of n-dimensional values using an electronic mapper; assign each of the n-dimensional values as a pixel value to a corresponding pixel of a virtual image frame, where n is an integer greater than one; and compress the virtual image frame according to a type of an image data container; and wherein the electronic mapper is configured to map the scalar values to the corresponding n-dimensional values based on a relationship represented by an n-dimensional curve or a plurality of 2 n way tree partitions of an n-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By way of example, according to the following detailed description and the drawings, other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent, in which:

[0010] Figure 1 An example process of a video / image transmission pipeline is depicted;

[0011] Figure 2 is a flowchart of an encoding method that can be used in the Figure 1 video / image transmission pipeline according to an embodiment;

[0012] Figure 3 Graphically illustrates the operating principle of a 1D to 3D mapper that can be used in the Figure 2 encoding method according to an embodiment;

[0013] Figure 4 Graphically illustrates the operating principle of a 1D to 3D mapper that can be used in the Figure 2 encoding method according to another embodiment;

[0014] Figures 5A to 5B Graphically illustrates the operating principle of a 1D to 3D mapper that can be used in the Figure 2 encoding method according to yet another embodiment;

[0015] Figure 6 is a flowchart of a decoding method that can be used in the Figure 1 video / image transmission pipeline according to an embodiment; and

[0016] Figure 7 FIG. illustrates a block diagram of a computing device according to an embodiment. DETAILED DESCRIPTION

[0017] The present disclosure and aspects thereof may be embodied in various forms, including: hardware, devices, or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; and hardware-implemented methods, signal processing circuits, memory arrays, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. The foregoing is only intended to give a general idea of the various aspects of the present disclosure and does not limit the scope of the present disclosure in any way.

[0018] Figure 1 FIG. depicts an example process of a video / image transmission pipeline 100 according to an embodiment, which shows the various stages from video / image capture to video / image content display. An image generation block 105 may be used to capture or generate a sequence of video / image frames 102. The frames 102 may be digitally captured (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide video and / or image data 107. Alternatively, the frames 102 may be captured on film by a film camera. The film may then be scanned and converted into a digital format to provide video / image data 107.

[0019] During the production stage 110, the data 107 may be edited to provide a video / image production stream 112. The data of the video / image production stream 112 may be provided to a processor (e.g., one or more processors, such as a central processing unit CPU, etc.) at a post-production block 115 for post-production editing. The post-production editing at block 115 may include, for example, adjusting or modifying the color or brightness in a specific area of the image to enhance the image quality or achieve a specific look of the image according to the creative intent of the video creator. This part of the post-production editing is sometimes referred to as "color timing" or "color grading". Other edits (e.g., scene selection and sequencing, image cropping, adding computer-generated visual effects, removing artifacts, etc.) may be performed at block 115 to produce a "final" version 117 of the work for distribution. During the post-production editing 115, the video and / or image may be optimized for viewing on a reference display 125.

[0020] After post-production 115, the data of the final version 117 can be transferred to the encoding block 120 for further downstream transfer to decoding and playback devices such as televisions, set-top boxes, and movie theaters. In some embodiments, the encoding block 120 can include audio and video encoders (such as audio and video encoders defined by ATSC, DVB, DVD, Blu-ray, and other transport formats) to generate an encoded bitstream 122. In the receiver, the encoded bitstream 122 is decoded by the decoding unit 130 to generate a corresponding decoded signal 132 that represents a copy or a nearly approximate version of the signal 117. The receiver can be attached to a target display 140, which can have slightly different or completely different characteristics from the reference display 125. In this case, the display management (DM) block 135 can be used to map the decoded signal 132 to the characteristics of the target display 140 by generating a display mapping signal 137. Depending on the embodiment, the decoding unit 130 and the display management block 135 can include separate processors or can be based on a single integrated processing unit. The various embodiments disclosed below can be used to implement the encoding block 120 and / or the decoding unit 130.

[0021] The codecs used in the encoding block 120 and / or the decoding unit 130 are capable of performing video / image data processing and compression / decompression. Compression is used in the encoding block 120 to make the corresponding file(s) smaller. The decoding process performed by the decoding unit 130 generally includes decompressing the received video / image data file(s) into a form that can be used for playback and / or further editing. Example codecs that can be used in the encoding block 120 and the decoding unit 130 include, but are not limited to, the XviD / DivX codec, the MPEG codec, and the H.264 codec.

[0022] A container is a digital file that contains video / audio data and any corresponding metadata, which are organized into a single package. The metadata can include subtitles, resolution information, creation date, device type, language information, etc. The container file interleaves different data types in a way that enables its components to be easily accessed by the decoding unit 130. Different types of containers are typically identified by their respective file extensions, which include MP4, WAV, AIFF, AVI, MOV, WMV, MKV, TIFF, JPEG, HEVC, FLV, F4V, SWF, etc.

[0023] For example, the JPEG image file format is a commonly used choice for storing and transmitting photographic images (e.g., still images and individual video frames). Many operating systems have viewers that support the visualization of JPEG image files, which are stored with the JPG or JPEG extension. Many web browsers also support the visualization of JPEG image files. JPEG encoding generally includes the following operations:

[0024] (1) Transformation: Transform the color image from the RGB (Red, Green, Blue) color space to the luminance / chrominance space.

[0025] (2) Downsampling: Downsampling is usually performed on the chrominance components, but not on the luminance component. For example, the image frame can be downsampled at a ratio of 2:1 horizontally and 1:1 vertically (2h 1v).

[0026] (3) Grouping: Organize the pixels of each color component into groups of 8×8 pixels, commonly referred to as "data units". If the number of rows is not an integer multiple of 8, the bottom row will be copied once or more times to achieve the required multiplicity. A similar copy can also be applied to the rightmost column.

[0027] (4) Discrete Cosine Transform (DCT): Apply the DCT to each data unit to create an 8×8 transformed component map. Due to the limited precision of machine-based arithmetic operations, the DCT usually causes some information loss.

[0028] (5) Quantization: Each of the 64 transformed components in the data unit is divided by a separate number called the Quantization Coefficient (QC), and then rounded to an integer. This operation usually results in additional information loss. Larger QC values tend to cause more loss. Many encoders rely on the QC tables recommended in the JPEG standard for quantization.

[0029] (6) Encoding: Encode the 64 quantized transform coefficients (now integers) of each data unit using a combination of run-length encoding (RLE) and Huffman encoding.

[0030] (7) Header: The final operation is to add a header listing all relevant JPEG parameters.

[0031] The corresponding JPEG decoder uses inverse operations to generate an image very close to the original encoded image.

[0032] As another example, a TIFF image frame is composed of a rectangular grid of pixels. The two axes of this geometry are called the horizontal axis (or X, or width) and the vertical axis (or Y, or length). The horizontal and vertical resolutions do not need to be equal. A baseline TIFF image divides the vertical extent of the image into one or more strips that are independently (separately) encoded and compressed. The TIFF format is an alternative to the tiled image format, where both the horizontal and vertical extents of the image are divided into smaller units. The data for a pixel includes one or more samples. For example, an RGB image typically has one red sample, one green sample, and one blue sample per pixel, while a grayscale image has only one sample per pixel. The TIFF format can be used with additive (e.g., RGB) and subtractive (e.g., cyan, magenta, yellow, black, or CMYK) color models. In at least some examples, the interpretation of the channel data is performed outside of the TIFF container. This interpretation can be assisted by metadata (such as, for example, an International Color Consortium (ICC) profile). The TIFF format does not limit the number of samples per pixel, nor the number of bits used to encode each sample. For example, three samples per pixel are at the low end of the multi-spectral imaging supported by TIFF, while hyperspectral imaging (also supported by TIFF) may use one hundred or more samples per pixel. In at least some embodiments disclosed herein, the functionality to support custom selection of the number of samples per pixel in the TIFF format is utilized. A TIFF image can be uncompressed, compressed using a lossless compression scheme, or compressed using a lossy compression scheme. An example of a lossless compression scheme compatible with the TIFF format is the LZW (Lempel-Ziv-Welch) compression scheme.

[0033] Augmented reality (AR) applications, virtual reality (VR) applications, and various applications involving the rendering of three-dimensional (3D) scenes can generally generate additional image data streams, each of which can take the form of a corresponding data sequence transmitted via a corresponding dedicated data channel. Examples of such data streams include, but are not limited to, depth data, vertex data, and index data. However, many of the above types of containers (file formats) and the corresponding conventional hardware of the encoding block 120 and the decoding unit 130 do not themselves support single-channel transmission, which disadvantageously poses difficulties for the efficient encoding / decoding and compression / decompression of the above additional data streams.

[0034] Various embodiments disclosed herein solve at least some of the above problems in the prior art by providing various schemes for packing single-channel data into a multi-channel container to achieve one or more of the following: (i) efficient utilization of the data capacity of the multi-channel container; (ii) maintaining the spatial and / or temporal consistency of the single-channel data stream in the packed form; and (iii) suppressing or avoiding excessive compression artifacts in the single-channel data decompressed at the decoder unit 130. At least some embodiments are fully compatible with the existing hardware of the corresponding video / image transmission pipeline (such as Figure 1 100), i.e., essentially do not depend on modifying any of its hardware / infrastructure. Alternatively, such embodiments can be advantageously implemented in software and / or firmware, e.g., by interfacing a corresponding relatively small data processing module with the existing codec without changing the (multiple) native container formats of the codec.

[0035] Figure 2 FIG. 6 is a flowchart of an encoding method 200 that can be used in the encoding block 120 according to an embodiment. The encoding method 200 is capable of encoding a single-channel data stream (e.g., a data sequence) into an n-channel container, where n is an integer greater than one. The single-channel data stream can be, for example, a depth data stream corresponding to a 3D scene. The n-channel container can be, for example, one of the above multi-channel containers.

[0036] The encoding method 200 includes receiving the next value of the single-channel data stream (at block 202). For the first instance of block 202, the received next value is the first value of the stream. For any subsequent instance of block 202, the received next value is the value in the stream that comes after the previously received value.

[0037] The encoding method 200 further includes selecting the next pixel of a virtual image frame (at block 204). Herein, a "virtual image frame" refers to an image frame similar to a conventional image frame. However, unlike a conventional image frame, a virtual image frame does not represent a conventional image. Alternatively, any desired pixel value can be assigned to different pixels of the virtual image frame, e.g., values generated by a suitable mapper (e.g., see block 206 and Figure 3 to FIG. 5). The pixels in such a virtual image frame can be spatially arranged in the same manner as a conventional image frame, e.g., in a rectangular array where the pixels are in rows and columns.

[0038] In some examples, in the first instance of block 204, the top left pixel of the frame is selected. In any subsequent instance of block 204, the selection process may follow a raster pattern or any other suitable predetermined pattern. The processing of the entire virtual image frame in encoding method 200 generally ends after each pixel of the frame has been selected once. For example, for a raster pattern, the pixel selected in the last instance of block 204 may be the bottom right pixel of the frame.

[0039] In other examples, a pixel processing order other than the above sequential order may also be used. In one example, random pixel selection is implemented. In various additional examples, the pixels of the virtual image frame are processed in parallel in the following modes: single instruction, multiple data (SIMD) vectorization; multiple instruction, multiple data (MIMD) multithreading; or a combination thereof. This alternative can also be implemented in various examples of decoding method 600 (see Figure 6 ).

[0040] Encoding method 200 also includes converting (in block 206) the 1D (1-dimensional scalar) value received in block 202 into a corresponding n-dimensional (nD) value using the selected 1D to nD mapper. In some specific examples, the nD value may be represented by a vector in nD space. In a Cartesian coordinate system, an origin-based vector v in nD space is represented by a string of values (x1, x2, …, x n ), where x i is the length of the projection of vector v onto the coordinate axis corresponding to the i-th dimension of the nD space. The number n is an algorithm parameter depending on the container used in encoding method 200. For example, for JPEG and MP4 containers, the number n is n = 3. For TIFF containers, the number n is the number of samples per pixel, which may be n > 3 in at least some examples, as explained above. Several non-limiting examples of the 1D to nD mapper that can be used in block 206 are described in more detail below with reference to Figure 3 to FIG. 5.

[0041] Encoding method 200 also includes assigning (in block 208) the nD value generated in block 206 to the pixel selected in block 204. For example, for n = 3 and a container supporting an RGB color scheme, the x1, x2, and x3 components of the corresponding 3D vector are assigned to the selected pixel as the R, G, and B values of the pixel (in block 208). As another example, for n = 16 and a TIFF container, the x1, x2, …, x 16 components of the corresponding 16D vector are assigned to the pixel as the corresponding 16 samples corresponding to multispectral imaging (in block 208). Other suitable assignment schemes may also be used in other examples of block 208.

[0042] Encoding method 200 further includes determining (in decision block 210) whether the end of the corresponding virtual image frame has been reached. When it is determined that the end of the frame has not been reached (being "No" at decision block 210), the operation of encoding method 200 loops back to block 202. Otherwise (being "Yes" at decision block 210), the virtual image frame (for which all pixels have been assigned) is compressed in a conventional manner according to the container format (in block 212). Then, the container can be directed from encoding block 120 to decoding unit 130 as described above (see also Figure 1 ). After the operation of block 212 is completed, encoding method 200 terminates.

[0043] Figure 3 Figs. 6 to 5 illustrate several non - limiting examples of 1D to nD mappers that can be used in block 206 of encoding method 200 according to various embodiments. For ease of presentation and without any implied limitation, the examples shown correspond to the number n = 3. Based on the provided description, a person of ordinary skill in the art will be able to manufacture and use various 1D to nD mappers corresponding to other n values (e.g., n = 2 and n>3) without any unnecessary experimentation. In at least some examples, the corresponding 1D to nD mapper is implemented using one or more lookup tables (LUTs).

[0044] Figure 3 The operating principle of a 1D to 3D mapper that can be used in block 206 of encoding method 200 according to an embodiment is graphically illustrated. More specifically, Figure 3 a 3D logarithmic spiral 302 in a Cartesian coordinate system is shown, and the coordinate axes of this coordinate system are respectively labeled as X1, X2, and X3. The spiral 302 can be used to map a scalar value d in the range [0, D] to a 3D vector (Y, Cb, Cr), where Y, Cb, and Cr respectively represent luminance, blue - difference chrominance component, and red - difference chrominance component.

[0045] In some examples, the scalar value d (D≥d≥0) is mapped to the spiral 302 by finding the point on the spiral whose distance from the origin O of the spiral is d. Then, the corresponding values of Y, Cb, and Cr are respectively determined using the Cartesian coordinates (x3, x2, x1) of the found point. In some specific cases, the processor of encoding block 120 is programmed with the following equations (1) - (3) to dynamically perform the corresponding calculations in block 206 of encoding method 200:

[0046] Cr = e (b×d) cos(a×d)-1 - Cr0 (1)

[0047] Cb = e (b×d) sin(α×d)-Cb0 (2)

[0048] Y = F(d) (3)

[0049] where a and b are parameters that determine how the radius of the helix 302 grows; Cr0 and Cb0 are (offset) constants; and F(d) is a function of d that determines the distance scale along the Y (luminance) axis. In some specific examples, the function F(d) is a quadratic function of d. In some other specific cases, equations (1)-(3) are used to pre-compute the corresponding LUT, and then the processor accesses the LUT to perform the corresponding operations in block 206.

[0050] When the mapping performed in block 206 is implemented based on the helix 302, this mapping is consistent in space and time. In particular, this mapping is distance-preserving. For any two points located on the helix 302, the mapping is also unique. Additionally, for any three points located on the helix 302, the distance relationship in 3D space is the same as the distance relationship along the helix 302 (i.e., in 1D space). For example, when point B is located between point A and point C on the helix 302, the mapping causes the distance between point A and point C in 3D space to be greater than the distance between point A and point B, and also greater than the distance between point B and point C. These properties resulting from the spatial and temporal consistency of the mapping are generally beneficial for implementing efficient compression in, for example, JPEG, HEVC, MP4, and many other formats, and for suppressing compression artifacts that appear in the decompressed single-channel data computed by the decoder unit 130.

[0051] Figure 4 The operating principle of a 1D to 3D mapper that can be used in block 206 of the encoding method 200 is graphically illustrated according to another embodiment. More specifically, Figure 4 a 3D Peano curve 402 in a Cartesian coordinate system is shown, and the axes of this coordinate system are labeled X1, X2, and X3 respectively. The granularity of the Peano curve 402 is 2 bits per dimension. In additional embodiments, Peano curves with other granularities can be used similarly. In other additional embodiments, various nD Peano curves can be used to implement corresponding 1D to nD mappers with various granularities, where n = 2 or n > 3.

[0052] The Peano curve 402 is used to map a scalar value d in the range [0, D] to 64 different 3D vectors (Y, Cb, Cr) or (R, G, B). In some examples, the scalar value d (D ≥ d ≥ 0) is mapped to the curve 402 by finding the point on the curve whose distance from the origin O of the curve, rounded to d. Then the Cartesian coordinates (x3, x2, x1) of the found point are used to determine the corresponding values of Y, Cb, and Cr or R, G, and B for the relevant pixel respectively.

[0053] A Peano curve (such as Peano curve 402) is an example of a space-filling curve (with endpoints) that covers the entire extent corresponding to an n-dimensional hypercube. Other examples of space-filling curves include, but are not limited to, Hilbert curves and Morton curves. Space-filling curves are a special case of fractal curves. Various suitable space-filling and / or fractal curves (with endpoints) can be used to construct various 1D to nD mappers suitable for implementing block 206 of encoding method 200 in a manner similar to that described above with reference to Peano curve 402 and Figure 4 the manner described above.

[0054] Figures 5A to 5B The operating principle of a 1D to 3D mapper that can be used in block 206 of encoding method 200 is graphically illustrated in accordance with yet another embodiment. More specifically, Figure 5A is a diagram illustrating an octree-partitioned 3D cube 500. Figure 5B is a diagram illustrating an octree 510 for partitioning the 3D cube 500.

[0055] Generally speaking, an octree (such as Figure 5B octree 510) is a tree-shaped data structure in which each internal node has exactly eight children. Octrees can be used to partition a bounded three-dimensional space (such as 3D cube 500) by recursively subdividing it into eight octants. In a point region (PR) octree, a node stores an explicit three-dimensional point as the “center” for the subdivision of that node. This center point defines one of the corresponding corners of each of the eight children. In a matrix (MX)-based octree, the subdivision point is implicitly the center of the space represented by the node. The root node of a PR octree can represent infinite space. The root node of an MX octree represents a finite bounded space, so the implicit center is well-defined. For example, the root node R of octree 510 (see Figure 5B ) represents 3D cube 500. Octree 510 is an example of an n = 3 2 n -way tree. One of ordinary skill in the relevant art will readily understand how to construct various 2 n -way trees for other n values without performing any undue experimentation. In additional examples, the value of n is 2, 4, 5, and so on.

[0056] The eight children of the root node R of octree 510 are nodes 0 through 7 (see Figure 5B ). In Figure 5A , the sub-cubes (octants) of 3D cube 500 corresponding to child nodes 0 - 7 are also labeled 0 through 7. In the view shown in Figure 5A , sub-cube 7 is not directly visible. For ease of illustration, Figure 5B only the children of child node 1 are explicitly shown. These grandchild nodes are shown inFigure 5B are labeled 10 through 17 in Figure 5A , the sub-cubes (octants) of sub-cube 1 corresponding to the grand-nodes 10 - 17 are also labeled 10 through 17. In Figure 5A the view shown, sub-cube 17 is not directly visible. For purposes of illustration, Figure 5B only the progeny of grand-node 11 are explicitly shown in Figure 5B and are labeled 110 through 117 in Figure 5A , the sub-cubes (octants) of sub-cube 11 corresponding to the great-grand-nodes 110 - 117 are also labeled 110 through 117. In Figure 5A the view shown, sub-cube 117 is not directly visible. A person of ordinary skill in the relevant art will readily understand that each of the child-nodes 0 and 2 - 7 similarly has grand-nodes, and the grand-nodes similarly have great-grand-nodes ( Figure 5A not explicitly shown in Figure 5A ). A person of ordinary skill in the relevant art will further understand that each of the grand-nodes 10 and 12 - 17 similarly has great-grand-nodes ( Figure 5A not explicitly shown in

[0057] There are a total of 256 great-grand-nodes in the octree 510. Similarly, there are a total of 256 corresponding great-grand-sub-cubes in the 3D cube 500. The 256 great-grand-sub-cubes partition the 3D cube 500 into 256 non-overlapping parts, each having the shape of a cube.

[0058] In some specific examples, to implement the 1D-to-3D mapper of block 206 of the encoding method 200, the three dimensions X1, X2, and X3 of the 3D cube 500 are respectively assigned to represent R, G, and B pixel values or Y, Cb, and Cr pixel values. The range [0, D] of the scalar d values is divided into 256 intervals. Each interval is assigned to a corresponding one of the 256 great-grand-sub-cubes of the 3D cube 500. Then the R, G, B (or Y, Cb, Cr) pixel values of the mapped scalar d value are determined using the Cartesian coordinates (x3, x2, x1) of the center of the corresponding great-grand-sub-cube. In some specific cases, the above relationship between the scalar d value and the coordinates (x3, x2, x1) of the sub-cube center is pre-computed and listed in a corresponding LUT, and then the encoder accesses the LUT to perform the relevant operations in block 206.

[0058] In various examples, the octree coding granularity (i.e., the number of sub-cubes in the 3D cube 500) is selected based on the desired accuracy and compression ratio required by the encoder. The finer the subdivision of the cube, the more likely it is that the lossy compression performed in block 212 of the encoding method 200 will result in errors during the corresponding decoding performed in the decoding unit 130 (see also Figure 6)。Therefore, the choice of the octree coding granularity may also need to be based on the amount of error introduced by the lossy compression.

[0059] Figure 6 is a flowchart of a decoding method 600 that can be used in the decoding unit 130 according to an embodiment. The decoding method 600 is compatible with the encoding method 200. Therefore, the decoding method 600 can substantially recover the single-channel data stream encoded using the selected multi-channel container format and the encoding method 200 for transmission.

[0060] The decoding method 600 includes decompressing the compressed virtual image frame according to the container format (at block 602). The decompression performed at block 602 is the inverse operation of the compression performed at block 212 of the encoding method 200.

[0061] The decoding method 600 further includes selecting the next pixel of the decompressed virtual image frame and reading the nD pixel value of the selected pixel (at block 604). In a representative example, the selection operation at block 604 of the decoding method 600 is implemented in a manner similar to the selection operation at block 204 of the above-described encoding method 200. Therefore, the reader can refer to the above description of block 204 to understand the relevant details of the pixel selection process.

[0062] The decoding method 600 further includes converting (at block 606) the nD pixel values read at block 604 into corresponding scalar values using a suitably selected nD-to-1D demapper. The selected nD-to-1D demapper is such that the demapping it performs is the reverse of the mapping performed by the 1D-to-nD mapper used at block 206 of the encoding method 200. In at least some examples, the nD-to-1D demapper used at block 606 of the decoding method 600 and the 1D-to-nD mapper used at block 206 of the encoding method 200 are implemented based on the same LUT. More specifically, the nD-to-1D demapper used at block 606 is configured to look up a scalar value in the LUT based on the provided nD value, while the 1D-to-nD mapper used at block 206 of the encoding method 200 is configured to look up an nD value in the same LUT based on the provided scalar value. In various examples, the operation of the nD-to-1D demapper is used to resolve errors introduced by lossy compression. For example, when the RGB values (100, 200, 30) of a virtual frame pixel become RGB values (94, 203, 31) after decoding due to errors introduced by lossy compression, the operation of the nD-to-1D demapper is used to correct the error to return the original (100, 200, 30) RGB values. This error correction is achieved, for example, by selecting the granularity of the 1D-to-nD mapping in such a way that the typical spread of the decoded points around the original constellation point is within the range (e.g., in the sense of Euclidean distance) closest to the original constellation point, rather than within the range of some other constellation point. This feature of the demapper is generally referred to as maximum likelihood detection.

[0063] The decoding method 600 further includes outputting (at block 608) the scalar values determined at block 606 as the next values of the corresponding data stream (data sequence). In the absence of loss and / or significant compression artifacts, the output data stream is a copy or approximate copy of the data stream received at block 202 of the encoding method 200.

[0064] The decoding method 600 further includes determining (at decision block 610) whether the end of the corresponding virtual image frame has been reached. When it is determined that the end of the frame has not been reached (being "no" at decision block 610), the operation of the decoding method 600 loops back to block 604. Otherwise (being "yes" at decision block 610), the decoding method 600 terminates.

[0065] Figure 7 FIG. is a block diagram of a computing device 700 according to an embodiment. The device 700 may be used, for example, in the encoding block 120. A computing device similar to the device 700 may also be used in the decoding unit 130. Based on the following description of the device 700, those of ordinary skill in the art will readily understand how to manufacture, configure, and use a computing device similar to the decoding unit 130.

[0066] Device 700 includes an input / output (I / O) device 710, an encoding engine 720, and a memory 730. The I / O device 710 can be used to enable the device 700 to receive at least a portion of the video / image stream 117 and output at least a portion of the encoded bitstream 122. The memory 730 can have a buffer for receiving, for example, image data to be encoded and compressed via the video / image stream 117. The received image data can include, in particular, the single-channel data stream described above. Once the data is buffered, the memory 730 can provide a portion of the data to the encoding engine 720 for processing therein. The encoding engine 720 includes a processor 722 and a memory 724. The memory 724 can store program code therein, which when executed by the processor 722 enables the encoding engine 720 to perform various encoding operations, including but not limited to the various encoding operations described above with reference to Figure 2 Parts or all of FIGS. 3 to 5. The memory 724 can also store the above-described LUT therein for the processor 722 to access as needed.

[0067] Aspects of the present invention can be further understood from the following enumerated example embodiments (EEEs)

[0068] EEE(1): An encoding method, comprising: converting, by a processor, a plurality of scalar values of a received data stream into corresponding n-dimensional values, the conversion being performed using a mapper; assigning, by the processor, each of the n-dimensional values as a pixel value of a corresponding pixel of a virtual image frame, where n is an integer greater than one; and compressing, by the processor, the virtual image frame according to the type of an image data container; and wherein the mapper is configured to map scalar values to corresponding n-dimensional values based on a relationship represented by an n-dimensional curve or a multi-way tree partition of an n-dimensional space. Here, a straight n-dimensional line is not an example of the "n-dimensional curve". For example, a "curve" includes at least two parts that are not collinear with each other in the corresponding n-dimensional space. n For example, "curve" includes at least two parts that are not collinear with each other in the corresponding n-dimensional space.

[0069] EEE(2): The method according to EEE(1), wherein n is greater than three.

[0070] EEE(3): The method according to EEE(1) or EEE(2), wherein the corresponding n-dimensional values are a set including a red value, a green value, and a blue value, or a set including a cyan value, a magenta value, and a yellow value, or a set including a luminance value, a blue chrominance difference value, and a red chrominance difference value.

[0071] EEE(4): The method according to any one of EEE(1) to EEE(3), wherein the plurality of scalar values are depth data, vertex data, or index data representing a three-dimensional scene.

[0072] EEE(5): A method as described in EEE(1), EEE(3), or EEE(4), wherein the n-dimensional curve is a helix; and wherein n = 2 or n = 3.

[0073] EEE(6): A method as described in any one of EEE(1) to EEE(5), wherein the n-dimensional curve is a space-filling curve having two endpoints.

[0074] EEE(7): A method as described in EEE(6), wherein the space-filling curve is selected from the group consisting of Peano curve, Hilbert curve, Morton curve, and fractal curve.

[0075] EEE(8): A method as described in any one of EEE(1) to EEE(6), wherein the mapper is configured to determine the corresponding n-dimensional value by: finding a position on the n-dimensional curve that has a distance representing the scalar value from its endpoints, the distance being along the n-dimensional curve; and representing the corresponding components of the n-dimensional value by a set of coordinates of the position in the n-dimensional space.

[0076] EEE(9): A method as described in any one of EEE(1) to EEE(5) and EEE(8), wherein the mapper is configured to determine the corresponding n-dimensional value by: identifying a partition in the plurality of 2 n way tree partitions that represents the scalar value; and representing the corresponding components of the n-dimensional value by a set of coordinates of the one partition in the n-dimensional space.

[0077] EEE(10): A method as described in any one of EEE(1) to EEE(9), wherein the mapper is configured to use a lookup table pre-computed based on the n-dimensional curve or the plurality of 2 n way tree partitions of the n-dimensional space.

[0078] EEE(11): A method as described in any one of EEE(1) to EEE(10), further comprising: decompressing the compressed image frame using the processor or another processor to generate a decompressed image frame, wherein the decompression is performed according to the type of the container, the compressed image frame having been generated by the compression; and transforming the plurality of n-dimensional pixel values of the decompressed image frame into another plurality of scalar values using the processor or the other processor, the transformation being performed using an inverse mapper; and wherein the inverse mapper is configured to perform an inverse mapping operation opposite to the corresponding mapping operation of the mapper.

[0079] EEE(12): The method as described in EEE(11), wherein both the mapper and the demapper are configured to use the same lookup table pre-computed based on the plurality of 2-way tree partitions of the n-dimensional curve or the n-dimensional space. n The same lookup table pre-computed based on the 2-way tree partitions.

[0080] EEE(13): A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including any one of EEE(1) to EEE(12).

[0081] EEE(14): An apparatus for encoding image data, the apparatus comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, together with the at least one processor, cause the apparatus to at least: convert a plurality of scalar values of a received data stream into corresponding n-dimensional values using an electronic mapper; assign each of the n-dimensional values as a pixel value of a corresponding pixel of a virtual image frame, where n is an integer greater than one; and compress the virtual image frame according to the type of the image data container; and wherein the electronic mapper is configured to map the scalar values to the corresponding n-dimensional values based on the relationship represented by the plurality of 2-way tree partitions of the n-dimensional curve or the n-dimensional space. n The relationship represented by the 2-way tree partitions of the n-dimensional curve or the n-dimensional space.

[0082] EEE(15): The apparatus as described in EEE(14), wherein the electronic mapper is configured to: find a position on the n-dimensional curve having a distance representing the scalar value from its end point, the distance being along the n-dimensional curve; and represent the corresponding components of the corresponding n-dimensional value by a set of coordinates of the position in the n-dimensional space.

[0083] EEE(16): The apparatus as described in EEE(14), wherein the electronic mapper is configured to: identify a partition representing the scalar value in the plurality of 2-way tree partitions; and represent the corresponding components of the corresponding n-dimensional value by a set of coordinates of the partition in the n-dimensional space. n The relationship represented by the 2-way tree partitions of the n-dimensional curve or the n-dimensional space.

[0084] EEE(17): The apparatus as described in any one of EEE(14) to EEE(16), wherein the electronic mapper is configured to use a lookup table pre-computed based on the plurality of 2-way tree partitions of the n-dimensional curve or the n-dimensional space. n The same lookup table pre-computed based on the 2-way tree partitions.

[0085] EEE(18): A device as described in any one of EEE(14) to EEE(17), wherein the at least one memory and the program code are further configured, together with the at least one processor, to cause the device to: decompress a compressed image frame according to the type of the container to generate a decompressed image frame; and transform a plurality of n-dimensional pixel values of the decompressed image frame into another plurality of scalar values by using an electronic demapper; and wherein the electronic demapper is configured to perform a demapping operation opposite to the corresponding mapping operation of the electronic mapper.

[0086] EEE(19): A device as described in EEE(18), wherein both the electronic mapper and the electronic demapper are configured to use a common look-up table (e.g., corresponding copies of the same look-up table) pre-computed based on the plurality of 2-way tree partitions of the n-dimensional curve or the n-dimensional space. n way tree partitions pre-computed common look-up table (e.g., corresponding copies of the same look-up table).

[0087] EEE(20): A device as described in any one of EEE(14) to EEE(19), wherein the plurality of scalar values are depth data, vertex data, or index data representing a 3D scene.

[0088] Regarding the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of these processes, etc. have been described as being performed in a specific ordered sequence, these processes can be practiced using the described steps executed in an order different from that described herein. Further, it should be understood that certain steps can be performed simultaneously, additional steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0089] Accordingly, it should be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided will be apparent upon reading the above description. The scope should not be determined with reference to the above description, but rather should be determined with reference to the appended claims and the full scope of equivalents to which those claims are entitled. It is expected and hoped that the technologies discussed herein will be developed in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In summary, it should be understood that this application is capable of modification and change.

[0090] All terms used in the claims are intended to be given the broadest reasonable interpretation and ordinary meaning as understood by those who are knowledgeable about the technologies described herein, unless an express contrary indication appears herein. In particular, the use of singular articles such as "a," "the," "said," etc. should be understood to recite one or more of the indicated elements unless the claim recites an express contrary limitation.

[0091] A summary of the present disclosure is provided to enable a reader to quickly ascertain the nature of the technical disclosure. This summary is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing detailed description, it can be seen that various features are grouped together in various embodiments for the purpose of presenting the disclosure as a unified whole. The methods of the present disclosure should not be construed as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as reflected in the appended claims, the inventive subject matter lies in less than all of the features of a single disclosed embodiment. Accordingly, the appended claims are hereby incorporated into the detailed description, with each claim standing on its own as a separately claimed subject matter.

[0092] Although this disclosure includes references to illustrative embodiments, this specification is not to be construed as limiting. Various modifications to the described embodiments, as well as other embodiments within the scope of the disclosure, which are obvious to those skilled in the art related to this disclosure, are considered to be within the principles and scope of the disclosure, e.g., as expressed in the claims.

[0093] Some embodiments may be implemented as circuit-based processes, including possible implementations on a single integrated circuit.

[0094] Some embodiments may be embodied in the form of methods and apparatuses for practicing these methods. Some embodiments may also be embodied in the form of program code recorded in a tangible medium, such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy disk, a CD-ROM, a hard disk drive, or any other non-transitory machine-readable storage medium, wherein when the program code is loaded into and executed by a machine (such as a computer, etc.), the machine becomes an apparatus for practicing the various embodiments described herein. Some embodiments may also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium (including being loaded into and / or executed by a machine), wherein when the program code is loaded into and executed by a machine (such as a computer or a processor, etc.), the machine becomes an apparatus for practicing the various embodiments described herein. When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates similarly to specific logic circuits.

[0095] Unless otherwise expressly stated, each numerical value and range should be interpreted as approximate as if the value or range were preceded by the word "about" or "approximately".

[0096] The use of figure numbers and / or reference numerals in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate the interpretation of the claims. Such use should not be construed as necessarily limiting the scope of these claims to the embodiments shown in the corresponding figures.

[0097] Although elements in the method claims (if any) are recited in a specific order with corresponding labels, these elements are not necessarily intended to be limited to being implemented in the specific order unless the claim recitation otherwise implies a specific order for implementing some or all of these elements.

[0098] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present disclosure. In this specification, the appearance of the phrase "in one embodiment" in various places does not necessarily refer to the same embodiment, nor are separate or additional embodiments mutually exclusive of other embodiments. The same applies to the term "implementation".

[0099] Unless otherwise specified herein, the use of ordinal adjectives "first," "second," "third," etc. to refer to an object among multiple similar objects merely indicates that different instances of such similar objects are being referred to, and is not intended to imply that the similar objects so referred to must be in a corresponding order or sequence, whether temporally, spatially, in ranking, or in any other manner.

[0100] Unless otherwise specified herein, in addition to its plain meaning, the conjunction "if" may also or alternatively be interpreted to mean "when" or "at" or "in response to determining" or "in response to detecting", which interpretation may depend on the corresponding specific context. For example, the phrase "if it is determined that..." or "if [the stated condition] is detected" may be interpreted to mean "upon determining that..." or "in response to determining that..." or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]".

[0101] Also for the purposes of this description, the terms "couple", "coupling", "coupled", "connect", "connecting", or "connected" refer to any means known in the art or later developed to allow energy to be transferred between two or more elements, and the insertion of one or more additional elements is contemplated, although this is not required. In contrast, the terms "directly coupled", "directly connected", etc. imply the absence of such additional elements.

[0102] As used herein with respect to components and standards, the terms "compatible" and "compliant" mean that the component communicates with other components in a manner specified, in whole or in part, by the standard and will be recognized by other components as being capable of communicating with other components in a manner specified by the standard. A compatible component need not operate internally in a manner specified by the standard.

[0103] The functions of the various components shown in the figures, including any functional blocks labeled "processor" and / or "controller", can be provided by using dedicated hardware as well as hardware capable of executing software associated with appropriate software. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. In addition, the explicit use of the term "processor" or "controller" should not be construed as referring exclusively to hardware capable of executing software and can implicitly include, but is not limited to, digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read only memories (ROMs) for storing software, random access memories (RAMs), and non-volatile storage devices. Other conventional and / or custom hardware may also be included. Similarly, any switches shown in the figures are conceptual only. Their functions can be performed by the operation of program logic, by dedicated logic, by the interaction of program control and dedicated logic, or even manually, with the particular technique being selectable by the implementer, as more specifically understood from the context.

[0104] As used in this application, the terms "circuit" and "circuitry" can refer to one or more or all of the following: (a) a pure hardware circuit implementation (such as an implementation in a pure analog circuit and / or digital circuit); (b) a combination of hardware circuitry and software, such as (if applicable): (i) a combination of analog hardware circuitry and / or digital hardware circuitry with software / firmware, and (ii) any part of a hardware processor with software (including (a) digital signal processor(s)), software, and (a) memory(ies), which together cause a device such as a mobile phone or a server to perform various functions); and (c) (a) hardware circuit(s) and / or (a) processor(s), such as (a) microprocessor(s) or a part of (a) microprocessor(s), which require software (e.g., firmware) to operate but may not have software present when not needed. This definition of circuitry applies to all uses of the term in this application (including all claims). As a further example, as used in this application, the term "circuit" also encompasses an implementation of only one hardware circuit or one processor (or processors) or a part of a hardware circuit or processor together with its accompanying software and / or firmware. The term "circuit" also encompasses, for example, a baseband integrated circuit or a processor integrated circuit for a mobile device or a similar integrated circuit in a server, a cellular network device, or other computing or network device when applicable to a particular claim element.

[0105] One of ordinary skill in the art will recognize that any block diagrams and flowcharts herein represent illustrative conceptual views of circuits embodying the principles of the present disclosure. Similarly, it will be recognized that any flowchart, flow diagram, state transition diagram, pseudocode, etc. represent various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not the computer or processor is explicitly shown.

[0106] The "Summary of the Invention" in this specification is intended to introduce some example embodiments, and additional embodiments are described in the "Detailed Description" and / or with reference to one or more of the drawings. The "Summary of the Invention" is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

1. An encoding method for encoding a single-channel data stream into an n-channel image container, the method comprising: Using a processor to convert a plurality of scalar values of a received single-channel data stream into corresponding n-dimensional values, where n is an integer greater than one and depends on the format of the n-channel image container used for the encoding, and the conversion is performed using a mapper; Using the processor to assign each of the n-dimensional values as a pixel value to a corresponding pixel of a virtual image frame; and Using the processor to compress the virtual image frame according to the n-channel image container format; and Wherein, the mapper is configured to map each of the scalar values to a corresponding n-dimensional value based on a mapping relationship represented by an n-dimensional curve or a plurality of 2 n way tree partitions, wherein the mapping relationship includes, for each of the scalar values, finding a position on the n-dimensional curve that has a distance representing the scalar value from its end point or the plurality of 2 n way tree partitions that represent the scalar value, the distance being measured along the n-dimensional curve; and representing the corresponding components of the n-dimensional value by a set of coordinates in the n-dimensional space through the position or the partition.

2. The method according to claim 1, wherein, n is greater than three.

3. The method according to claim 1 or 2, wherein, The corresponding n-dimensional values are a set including a red value, a green value, and a blue value, or a set including a cyan value, a magenta value, and a yellow value, or a set including a luminance value, a blue chrominance difference value, and a red chrominance difference value.

4. The method according to any one of claims 1 to 3, wherein, The plurality of scalar values represent depth data, vertex data, or index data of a 3D scene.

5. The method according to claim 1, 3 or 4, wherein, The n-dimensional curve is a helix, and n = 2 or n = 3.

6. The method according to any one of the preceding claims, wherein, The n-dimensional curve is a space-filling curve with two endpoints.

7. The method according to claim 6, wherein, The space-filling curve is selected from the group consisting of a Peano curve, a Hilbert curve, a Morton curve, and a fractal curve.

8. The method according to any one of the preceding claims, wherein, The mapper is configured to use a lookup table pre-computed based on the plurality of 2-way tree partitions of the n-dimensional curve or the n-dimensional space. n way tree partitions.

9. The method according to any one of the preceding claims, further comprising: Using the processor or another processor to decompress the compressed image frame to generate a decompressed image frame, where the decompression is performed according to the container format, and the compressed image frame has been generated through the compression; and Using the processor or the another processor to transform the plurality of n-dimensional pixel values of the decompressed image frame into another plurality of scalar values, and the transformation is performed using a demapper; and wherein the demapper is configured to perform a demapping operation opposite to the corresponding mapping operation of the mapper.

10. The method according to claim 9, wherein, Both the mapper and the demapper are configured to use a common lookup table pre-computed based on the plurality of 2-way tree partitions of the n-dimensional curve or the n-dimensional space. n way tree partitions.

11. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform the operations of any one of the preceding claims.

12. An apparatus for encoding image data, the apparatus comprising: At least one processor; and At least one memory including program code; wherein the at least one memory and the program code are configured to, together with the at least one processor, cause the device to at least: Using an electronic mapper to convert a plurality of scalar values of a received single-channel data stream into corresponding n-dimensional values, where n is an integer greater than one and depends on the format of the n-channel image container used for the encoding; Assigning each of the n-dimensional values as a pixel value to a corresponding pixel of a virtual image frame; and Compressing the virtual image frame according to the n-channel image container format; and Wherein, the electronic mapper is configured to map each of the scalar values to a corresponding n-dimensional value based on a mapping relationship represented by an n-dimensional curve or a plurality of 2 n way tree partitions, wherein the mapping relationship includes, for each of the scalar values, finding a position on the n-dimensional curve that has a distance representing the scalar value from its end point or the plurality of 2 n way tree partitions that represent the scalar value, the distance being measured along the n-dimensional curve; and representing the corresponding components of the n-dimensional value by a set of coordinates in the n-dimensional space of the position or the partition.

13. The apparatus according to claim 12, wherein, The electronic mapper is configured to use a lookup table pre-computed using the plurality of 2-way tree partitions based on the n-dimensional curve or the n-dimensional space. n way tree partitions.

14. The device according to claim 12 or 13, wherein, The at least one memory and the program code are configured to, together with the at least one processor, further cause the device to: Generate a decompressed image frame by decompressing the compressed image frame according to the container format; and Using an electronic demapper to transform the plurality of n-dimensional pixel values of the decompressed image frame into another plurality of scalar values; and wherein the electronic demapper is configured to perform a demapping operation opposite to the corresponding mapping operation of the electronic mapper.

15. The device according to claim 14, wherein, Both the electronic mapper and the electronic demapper are configured to use a common lookup table pre-computed based on the plurality of 2-way tree partitions of the n-dimensional curve or the n-dimensional space. n way tree partitions.

16. The device according to any one of claims 12 to 15, wherein, The plurality of scalar values represent depth data, vertex data, or index data of a 3D scene.

Images