Encoding and decoding method of multi-planar image representation and corresponding apparatus

Abstract

A decoding method is disclosed. At least one picture of a packed layer comprising a multi-plane image representation is decoded, wherein the size of the at least one layer is reduced with respect to a reference size. Information describing each layer in terms of size and position in the at least one picture is decoded; and based on the decoded at least one picture and the decoded information, a multi-plane image representation comprising a plurality of layers is reconstructed.

Description

[0001] Cross-reference to related applications This application claims the benefit of European application number 24305409.5, filed on 19 March 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] At least one of these embodiments generally relates to methods and apparatus for encoding and decoding multiplanar image representations. Background Technology

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transform to utilize spatial and temporal redundancy in video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame image correlations, and then the differences between the original and predicted blocks (typically represented as prediction error or prediction residuals) are transformed, quantized, and entropy encoded / decoded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy encoding / decoding, quantization, transform, and prediction. Summary of the Invention

[0004] In one implementation, an MPI representation comprising multiple layers is obtained. Multiple layers can be tiled (also called packed) into at least one image (e.g., within a single image), where the size of at least one layer is reduced relative to a reference size. The tiled image is encoded. Information describing each layer in terms of size and position within the tiled image is also encoded. In one example, at least one layer of the multiplanar image (MPI) representation is subsampled to reduce the overall size of the encoded tiled image. In another example, at least one layer of the MPI representation is cropped to reduce the overall size of the tiled image. In yet another example, at least one layer is cropped, and at least one layer is subsampled to reduce the overall size of the tiled image. In yet another example, at least one layer is cropped (e.g., into regions) and may be reassembled (e.g., the regions are repositioned). For a single layer, at least two distinct regions may be retained after cropping. In yet another example, blank areas of the tiled image are implicitly or explicitly filled, for example, by modifying the subsampling factor or the cropped region definition. In another example, at least one layer is subsampled, and a layer is clipped (e.g., into multiple regions) and may be reassembled. For a single layer, at least two distinct regions may be retained after clipping. Attached Figure Description

[0005] Figure 1 The diagram illustrates a block diagram of a system in which various aspects of this embodiment can be implemented; Figure 2 A block diagram illustrating an embodiment of a video encoder is shown; Figure 3A block diagram illustrating an embodiment of a video decoder is shown; Figure 4 Perspective projection and orthographic projection are depicted; Figure 5 Two tiled images were depicted, one for texture and one for opacity map, each with 16 layers; Figure 6 This represents the combination of textures and opacities in a side-by-side packaged image. Figure 7 This represents the texture and opacity that make up the image when arranged in a top-to-bottom pack; Figure 8 The image is composed of textures and opacities that are interwoven over time; Figure 9 A flowchart illustrating the encoding method based on the example is provided. Figure 10 A flowchart illustrating the decoding method based on the example is provided. Figure 11 This represents a viewpoint of the MPI content referred to as the "train." Figure 12 This represents an example of side-by-side images that include several subsampled layers; Figure 13 The four layers representing the MPI content, referred to as the "train"; Figure 14 The four clipping layers represent the MPI content referred to as the "train"; Figure 15A This shows an example of side-by-side images that include several cropped layers; Figure 15B The location of the clipping layer in the original layer i is depicted; Figure 15C The location of the clipping layer in the tiled image is depicted; Figure 16A The four layers representing the MPI content, referred to as "trains," which are cut into several regions; Figure 16B The initial position of the clipping region in the layer and the position of the clipping region in the reconstructed layer are depicted; Figure 16C The four layers representing the cropped and subsampled MPI content, known as the "train," are: Figure 17 This represents an example of side-by-side images that include some subsampling layers and implicit subsampling to remove blank space; and Figure 18 The illustration shows adding layers on the borders of images in a top-bottom stacked arrangement. Detailed Implementation

[0006] This application describes a wide variety of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail, and are generally described in a manner that may sound restrictive, at least for the purpose of showing individual features. However, this is for the purpose of clarity and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in earlier applications.

[0007] The aspects described and envisioned in this application can be realized in many different forms. The following... Figure 1 , Figure 2 and Figure 3 Some embodiments have been provided, but other embodiments are contemplated, and Figure 1 , Figure 2 and Figure 3 The discussion does not limit the scope of the implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to the transmission of a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media (e.g., non-transitory computer-readable storage media) having instructions thereon stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon.

[0008] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "encoding" and "encoding / decoding," "pixel" and "sample," and "image," "picture," and "frame." Generally, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0009] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first" and "second" can be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of these terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0010] For clarity, throughout the embodiments described herein, satisfied and unsatisfied conditions and configuration condition(s) are described relative to (e.g., greater than or less than) a threshold, (e.g., a threshold) value, a configuration (e.g., a threshold) value, etc.). For example, satisfied conditions may be described as being above (e.g., a threshold) value, and unsatisfied conditions (e.g., performance criteria) may be described as being below (e.g., a threshold) value. The embodiments described herein are not limited to threshold-based conditions. Any other kind of conditions and (e.g., belonging to or not belonging to a range of values) may be applied to the embodiments described herein.

[0011] This aspect is not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether existing or future-developed, as well as any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application may be used alone or in combination.

[0012] Figure 1 The illustration shows a block diagram of an example system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100 may be embodied individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0013] System 100 includes at least one processor 110 configured to execute instructions loaded thereon for implementing various aspects, such as those described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 140 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0014] System 100 includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents one or more modules that may be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 130 may be implemented as a separate element of system 100, or it may be incorporated within processor 110 as a combination of hardware and software known to those skilled in the art.

[0015] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0016] In some embodiments, the memory within processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., processor 110 or encoder / decoder module 130) is used for one or more of these functions. External memory may be memory 120 and / or storage device 140, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Codec, also known as H.265 and MPEG-H Part 2), or VVC (Various Video Codec, a new standard being developed by JVET (Joint Video Experts Group)).

[0017] As shown in block 105, inputs can be provided to the components of system 100 through various input devices. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcasting company, (ii) component (COMP) input terminals (or a set of COMP input terminals), (iii) universal serial bus (USB) input terminals, and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 1 Other examples not shown include composite video.

[0018] In various embodiments, the input device of block 105 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a frequency band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower frequency band to select, for example, a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0019] Furthermore, the USB and / or HDMI terminals may include their respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, in a separate input processing IC or within processor 110. Similarly, as needed, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within processor 110. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0020] Various components of system 100 can be provided within an integrated housing, in which various components can be interconnected and transmit data therebetween using a suitable connection arrangement 115, such as an internal bus known in the art, including an I2C bus, wiring, and printed circuit board.

[0021] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, in a wired and / or wireless medium.

[0022] In various embodiments, data is streamed to system 100 using a Wi-Fi network such as IEEE 802.11 (IEEE stands for Institute of Electrical and Electronics Engineers). The Wi-Fi signals in these embodiments are received via a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 100, delivering data via an HDMI connection to input box 105. Still other embodiments use an RF connection to input box 105 to provide streaming data to system 100. As indicated above, various embodiments provide data in a non-streaming manner. Furthermore, various embodiments also use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0023] System 100 can provide output signals to various output devices, including a display 165, a speaker 175, and other peripheral devices 185. The display 165 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be used in a television, tablet, laptop, mobile phone, or other device. The display 165 can also be integrated with other components (e.g., as in a smartphone) or exist independently (e.g., as in an external display of a laptop). In various examples of embodiments, other peripheral devices 185 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, both abbreviated as DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide functionality based on the output of system 100. For example, a disc player performs the function of playing the output of system 100.

[0024] In various embodiments, signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention are used to transmit control signals between system 100 and display 165, speaker 175, or other peripheral devices 185. Output devices can be communicatively coupled to system 100 via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speaker 175 can be integrated into a single unit with other components of system 100 in electronic devices such as televisions. In various embodiments, display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0025] Display 165 and speaker 175 may alternatively be separate from one or more other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments where display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0026] The embodiments can be implemented by computer software implemented by processor 110, by hardware, or by a combination of software and hardware. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. Memory 120 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. Processor 110 can be of any type suitable for the technical environment and, as a non-limiting example, can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0027] Figure 2 The illustration shows an example video encoder 200, such as a VVC (Various Video Codec) encoder. Figure 2 The diagram can also show an encoder that has been improved upon the VVC standard or an encoder that uses a technology similar to VVC.

[0028] Before encoding, the video sequence may undergo pre-coding (201), for example, applying color transformations to the input color image (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping on the input image components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization on one of the color components). Metadata may be associated with pre-processing and attached to the bitstream.

[0029] In encoder 200, the image is encoded by encoder elements as described below. The image to be encoded is partitioned (202) and processed in units such as CTU (code-decode tree unit) and CU (code-decode unit). Each unit is encoded using, for example, an intra-frame or inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (260), for example using an intra-frame prediction tool such as decoder-side intra-frame mode derivation (DIMD). In inter-frame mode, motion estimation (275) and compensation (270) are performed. The encoder determines (205) which of the intra-frame or inter-frame modes to use to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (210) the predicted block (also called the prediction block) from the original image block.

[0030] The prediction residuals are then transformed (225) into transformation coefficients c (also known as prediction residual transformation coefficients), and the transformation coefficients c are quantized (230) into quantization indices. (Also known as the encoder-side transform coefficient level or quantized transform coefficient). Quantization level (also known as the quantization index). Motion vectors and other syntax elements (such as image partitioning information) are entropy encoded / decoded (245) to output a bitstream. The encoder can skip the transform and directly quantize the untransformed residual signal. The encoder can bypass the transform and quantization, i.e., the residual is directly encoded / decoded without applying the transform or quantization process.

[0031] The encoder decodes the coded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) (also known as scaling) and inverse transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. An in-loop filter (265) is applied to the reconstructed image to perform filtering such as deblocking / SAO (Sample Adaptive Offset) / ALF (Adaptive Loop Filter) to reduce coding artifacts. The filtered image is stored in a reference image buffer (280). Thus, an in-loop filter (265) is used to enhance the reconstructed image before storing it in the reference image buffer (280). In-loop filters form a complete series. Among them, the deblocking filter (DBF) is designed to reduce blocking artifacts that appear along block boundaries. Deblocking filters are generally designed to improve subjective quality, i.e., the perceptibility of such encoding / decoding errors to the human psychological visual system. In typical video codec standards such as HEVC and VVC, deblocking filters are pre-determined based on codec information (such as prediction modes, motion vectors, and transform coefficients) and local variations across block boundaries. On the other hand, an adaptive loop filter (ALF) is learned on the encoder side to minimize the mean square error relative to the source image, and the learned filter weights are then encoded into the bitstream. The adaptive loop filter is typically applied at the CTU level, while the deblocking filter is applied along block boundaries.

[0032] Figure 3 A block diagram of an example video decoder 300 is shown. In decoder 300, the bitstream is decoded by decoder elements, as described below. Video decoder 300 typically performs operations similar to... Figure 2 The encoding process described herein is the opposite of the decoding process. Encoder 200 typically also performs video decoding as part of the encoded video data.

[0033] Specifically, the input to the decoder includes a video bitstream, which can be generated by the video encoder 200. The bitstream is first entropy-decoded (330) to obtain the quantization level. (Also known as the transform coefficient level or quantization level on the decoder side), prediction mode, motion vectors, and other encoded information. Image partitioning information indicates how the image is partitioned. Therefore, the decoder can partition (335) images based on the decoded image partitioning information. Quantization level The dequantized (340) transformation coefficients are converted into the reconstructed coefficients. Dequantization is also known as scaling. The reconstructed transform coefficients... The image is inversely transformed (350) to obtain the prediction residual. The prediction residual and the predicted block (also called the prediction block) are combined (355) to reconstruct the image block. The prediction block can be obtained from intra-frame prediction (360) or motion-compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference image buffer (380). Note that for a given image, the contents of the reference image buffer 380 on the decoder 300 side are the same as the contents of the reference image buffer 280 on the encoder 200 side for the same image.

[0034] The decoded image can undergo further post-decoding processing (385), such as inverse color transformation (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (201). The post-decoding process can use metadata derived in the pre-encoding process and signaled in the bitstream.

[0035] Figure 4 Perspective and orthographic projections are depicted. A multi-plane image (MPI) is a layer-based description (also called a representation) of a 3D scene, such as that generated from a multi-view scene. An MPI comprises a set of images (also called image planes or layers, such as those including textures and opacity / transparency maps) at fixed resolutions (e.g., in pixels) at different depth locations in space (e.g., relative to the camera position) and a set of metadata collection parameters (such as the depth of each image). The difference between an MPI and a classic slice of a 3D scene is that an MPI is not an orthographic projection (or cross-section) of the scene, but a perspective projection. With orthographic projection, layers in the background have the same pixel pitch and number of pixels as layers in the foreground, while with perspective projection, the number of pixels is the same, but the pixel pitch of the layers increases linearly with depth. Based on such an MPI representation, a renderer can synthesize any viewpoint of a 3D scene. The renderer can apply a simple accumulation of MPI information in the direction of the view to be interpolated. Opacity / transparency maps are well-suited for handling occlusion.

[0036] Encoding such an MPI representation is of particular interest. To this end, the MPI layer can be packed (also known as tiled) into at least one image (e.g., a single large image) that can be decoded by (e.g., a single) decoder (e.g., a conventional 2D video decoder). Figure 5 Two tiled images were depicted, one for texture and one for opacity map, each with 16 layers.

[0037] Multiplanar Image Information (MPII) SEI messages have been further defined for implementing virtual reality / immersive experiences on existing consumer devices, such as using existing video codecs, such as HEVC and VVC developed for regular 2D video. Bitstreams can be decoded by regular 2D video decoders, and the resulting image from the ordinary decoding process can be reinterpreted using SEI messages as a packed representation of texture and opacity layers. MPII SEI messages specify the multiplanar image (MPI) scene representation information that can be used for view composition. In other words, MPII SEI messages convey information that a decoder can use to reconstruct an MPI image from the decoded output image containing packed MPI texture and opacity layers. MPII SEI messages support side-by-side, top-bottom, and temporally interleaved packing arrangements. MPII SEI messages can also convey depth information corresponding to each MPI layer. Examples of MPII SEI message syntax are provided in Table 1. Table 1.

[0038] The increment of 1 in mpii_num_layers_minus1 specifies the number of texture and opacity layers represented by MPI.

[0039] An mpii_layer_depth_equal_distance_flag of 1 indicates that MPI layers are generated using equal distances, and the depth parameter Z[i] of each layer can be derived using the nearest depth value ZNear and the farthest depth value ZFar.

[0040] The depth value of the i-th MPI layer Z[ i ] = i * (ZFar – Znear) / (mpi_num_layers_minus1) + Znear.

[0041] The value of mpii_layer_depth_equal_distance_flag being 0 indicates that the depth information for each layer follows the SEI in scientific form with sign / exponent / mantissa.

[0042] In some applications, disparity is used instead of depth (the relationship between disparity value D and depth value Z is D = 1 / Z). When mpii_layer_depth_equal_distance_flag is equal to 1, instead of the derived depth value, the disparity value of the i-th MPI layer is D[i] = i * (DFar – Dnear) / (mpi_num_layers_minus1) + Dnear.

[0043] An mpii_texture_opacity_arrangement_flag value of 0 indicates that the decoded output image is represented as follows: Figure 7 The image is composed of textures and opacities arranged in a top-bottom pack.

[0044] An mpii_texture_opacity_arrangement_flag value of 1 indicates that the decoded output image is represented as follows: Figure 6 The images are composed of textures and opacities arranged side-by-side.

[0045] mpii_texture_opacity_interleave_flag equal to 1 indicates that the decoded output image corresponds to, for example, Figure 8 The output sequence shown comprises temporally interleaved textures and opacities that make up the image. `mpii_texture_opacity_interleave_flag` equal to 0 indicates that the decoded output image corresponds to... Figure 6 and Figure 7 The image is composed of spatially packed textures and opacities.

[0046] `mpii_picture_num_layers_in_height_minus1` plus 1 specifies the number of spatially packed layers in the height, for example, for images 0 and 1. The variable `hLayers` is set to equal to `mpii_picture_num_layers_in_height_minus1+1`, while the variable `wLayers` is set to equal to `(mpii_num_layers_minus1+1) / hLayers`. Therefore, `hLayers` represents the number of layers in the height, and `wLayers` represents the number of layers in the width.

[0047] The layer size is fixed and depends on the values ​​of hLayers and wLayers. Given a complete image size of W * H, the layer sizes can be obtained as follows: Wlayer = W / wLayers and Hlayer = H / hLayers.

[0048] Without constraints on hLayers and the number of layers (denoted as Nlayers = mpii_num_layers_minus1+1), it is not guaranteed that (Wlayer * wLayers) equals W, and (Hlayer * hLayers) equals H. Therefore, it is possible to end with a tiled image with a width of (W - (Wlayer * wLayers)) and a height of (H - (Hlayer * hLayers)) and blank right and bottom boundaries. It can also be noted that, depending on the value of hLayers and the number of layers, there may be fewer layers than the actual number represented by the MPI. For example, if the number of layers represented by the MPI is 17 and hLayers is 4, then wLayers equals 4 (e.g., a rounded integer value of 4.25), which results in hLayers * wLayers = 16, which is less than the number of layers, i.e., 17.

[0049] The SEI messages in Table 1 indicate the number of layers and describe how different layers of MPI content are tiled (e.g., packaged) together into a single image or two temporally interleaved images. Having a single (e.g., large) image or interleaved image containing all layers of MPI content makes it possible to use an already deployed 2D video decoder. However, the above methods are limited by the maximum size of images that the deployed decoder can handle (e.g., process). For example, a 4K image (e.g., an image of 3860 × 2160 pixels) is the upper limit that most currently available AVC, HEVC, or VVC devices can support. Under this limitation, 16 layers of 1K images (e.g., 960 × 540 pixels) of MPI is the maximum that can be processed. The number of layers and the resolution per layer are key parameters for the MPI renderer to correctly render 3D scenes. To improve the user experience provided by such MPI content, increasing the number of layers and / or increasing the resolution of each layer image is of interest.

[0050] Therefore, a method is disclosed that makes it possible to enhance the user experience by increasing the number of layers while maintaining (or even increasing) the resolution of each layer's image as well as the resolution of the entire image.

[0051] Figure 9 A flowchart illustrating the encoding method 900 based on the example is provided.

[0052] In S900, a multi-plane image (MPI) representation including multiple layers is obtained.

[0053] In S902, multiple layers are packaged (i.e., combined) into at least one image (e.g., in a single image), wherein the size of at least one layer is reduced relative to a reference size. The reduced size can be obtained by subsampling at least one layer and / or, for example, cropping at least one layer based on active pixels.

[0054] In S904, for example via Figure 2 The encoder shown encodes at least one image.

[0055] In S906, information (e.g., multi-planar image information) is encoded, which describes each layer in terms of size and position in the tiled image.

[0056] In the example, the information includes the reference size, the number of layers, and for each layer, at least one subsampling factor and information indicating the position (e.g., location) of that layer in the tiled image.

[0057] In another example, the information includes a reference size, the number of layers, and for each layer, a subsampling factor in the vertical direction and a subsampling factor in the horizontal direction, as well as information indicating the position (e.g., location) of the layer in the tiled image.

[0058] In another example, the information includes the number of layers, and for each layer, the information includes the size of the cropping layer and information indicating the position (e.g., location) of the cropping layer in the original layer and its position in the tiled image. In yet another example, the information includes the number of layers, and for each layer, it includes one or two subsampling factors (e.g., one for both directions, or one for the vertical direction and one for the horizontal direction), the size of the cropping layer, and information indicating the position (e.g., location) of the cropping layer in the original layer and its position in the tiled image.

[0059] In another example, the information includes the number of layers, and for each layer, it includes the number of cropped regions, the size of each region, and information indicating the initial and repositioning positions of each region within that layer, as well as information indicating the position (e.g., location) of that layer in the tiled image.

[0060] In another example, the information includes the number of layers, and for each layer, one or two subsampling factors for that layer, the number of cropped (e.g., selected) regions, the size of each region, and information indicating the initial and reassembled positions of each region within that layer, as well as information indicating the layer's position (e.g., localization) within the tiled image. The reassembled position is the location of the region within the reassembled layer.

[0061] SEI messages can be used to encode additional information. Various examples of such messages are provided below.

[0062] Figure 10 A flowchart of decoding method 1000 based on the example is depicted.

[0063] In S1000, for example via Figure 3 The decoder decodes at least one image (e.g., a tiled image). The at least one image includes a packing layer of a multi-planar image representation, wherein the size of at least one layer is reduced relative to a reference size.

[0064] In S1002, the information is decoded, which describes each layer in terms of size and position in at least one image.

[0065] In S1004, a multi-plane image representation comprising multiple layers is reconstructed based on at least one decoded image and the decoded information. For example, if subsampling is performed on the encoder side, the layers are upsampled using the subsampling factor mentioned in the information decoded in S1002. In the case of cropping, the cropping layer can be repositioned within the layers of the multi-plane image representation according to the position mentioned in the information decoded in S1002.

[0066] In all examples, the position of the layer or clipping region can be the position of the top-left pixel of that layer or clipping region. However, this principle is not limited to the top-left pixel. In fact, the position can be defined relative to another pixel, such as the center pixel, the bottom-right pixel, etc. The position is defined, for example, in pixels. However, the position can be defined in another unit.

[0067] In one example, at least one layer is subsampled to reduce the overall size of the tiled image. One possible criterion for deciding to subsample a layer is when that layer contains a small number of effective pixels. Ineffective pixels are pixels with an opacity of 0 (e.g., pixel opacity is derived from an opacity map). For most layers in MPI content, there are regions with zero opacity (transparent regions). For example, if the effective pixels are less than a given percentage, one or more layers can be defined as less important. Other criteria can be defined to signal that a layer is less important, such as whether the effective pixels are in non-salient regions (based on saliency map information), or whether the effective pixels correspond to regions of high-speed motion, or whether they correspond to low-frequency textures. For such layers, subsampling can reduce the corresponding size of that layer in the tiled image with interest.

[0068] Figure 11 This refers to a viewpoint of the MPI content, often referred to as the "train." By constructing MPI content, some layers may have very few effective pixels. Certain layers can be easily subsampled without compromising the quality of future rendering. Figure 12 This example shows a side-by-side image that includes some subsampled layers. In this example, up to 21 layers are tiled, compared to 16 layers without subsampling.

[0069] Therefore, the SEI message in Table 1 can be modified to include information about the subsampled layers. In the example, the SEI message may include the reference size of the layer (e.g., defined in pixels), the number of layers, and the subsampling factor for each layer (e.g., relative to the reference size) and the location (e.g., localization) of that layer (e.g., the subsampled layer) in the tiled image. The reference size can be the size of a layer without any subsampling. Table 2 provides an example. Differences from the messages in Table 1 are indicated in italics. Table 2.

[0070] A value of 0 for mpii_layer subsampling_flag indicates that no layer is subsampled.

[0071] The mpii_layer subsampling_flag being equal to 1 indicates that at least one layer is subsampled by a factor defined for each layer.

[0072] mpii_reference_size_layer_x specifies the pixel size of the reference layer in the horizontal dimension, which is the reference size in the horizontal dimension.

[0073] mpii_reference_size_layer_y specifies the pixel size of the reference layer in the vertical dimension, which is the reference size in the vertical dimension.

[0074] mpii_subsampling_factor[i] specifies the subsampling factor for the i-th layer. Therefore, the subsampling factor is defined relative to the reference size.

[0075] mpii_layer_position_intiled_picture_x[i] specifies the horizontal position of the i-th layer in the tiled image (e.g., defined in pixels) (e.g., the horizontal position of the top-left corner of the i-th layer in the tiled image).

[0076] mpii_layer_position_intiled_picture_y[i] specifies the vertical position of the i-th layer in the tiled image (e.g., defined in pixels) (e.g., the vertical position of the top-left corner of the i-th layer in the tiled image).

[0077] In one variation, subsampling factors can be indicated for each horizontal and vertical direction; for example, they can be different. In this configuration, the SEI message can include the layer reference size in pixels, the number of layers, and for each layer, the subsampling factors in the horizontal direction, the subsampling factors in the vertical direction, and the position (e.g., location) of the layer (e.g., subsampled layer) in the tiled image. Examples are provided in Tables 3 and 4. Differences from the messages in Table 1 are indicated in italics. Table 3.

[0078] mpii_subsampling_factor_x[i] specifies the subsampling factor of the i-th layer in the horizontal direction.

[0079] mpii_subsampling_factor_y[i] specifies the subsampling factor of the i-th layer in the vertical direction.

[0080] The subsampling factor can be defined relative to the reference size.

[0081] In another example, add a syntax element (e.g., mpii_layer_subsampling_horizontalflag) to indicate whether to encode a single subsampling factor for both directions or to encode a subsampling factor for each direction. Table 4.

[0082] An mpii_layer subsampling_horizontal_flag value of 0 indicates that the horizontal subsampling factor is the same as the vertical subsampling factor.

[0083] The mpii_layer subsampling_horizontal_flag being equal to 1 indicates the existence of two subsampling factors, one for each direction, meaning the horizontal subsampling factor can be different from the vertical subsampling factor.

[0084] In another example, at least one layer is cropped to reduce the overall size of the tiled image. Layers can be cropped by retaining only the valid portions / regions of the layer. Most layers in MPI content include regions without valid pixels, e.g., corresponding to missing information at that depth in the scene. For a given MPI content "train", Figure 13 The diagram illustrates four layers, where the meaningful portion of the scene is not expanded across the entire image. It is limited to certain areas. These layers can then be cropped, for example, by retaining only the areas containing the valid pixels. Figure 14The illustration shows the possible clipping of these layers, corresponding to rectangular shapes that can be defined using two dimensions, X and Y. The area is significantly reduced, and these clipped layers can be tiled as follows: Figure 15A The tiled (also known as packed) image is shown. In the example, the size of the region or cropping layer is a multiple of 16 or 32.

[0085] The SEI message can therefore be modified to include information about the clipping layers. In the example, the SEI message may include the number of layers, and for each layer, the size of the clipping layer, its position (e.g., location) within the original layer, and its position within the tiled image. Table 5 provides an example. Differences from the messages in Table 1 are indicated in italics. Table 5.

[0086] A value of 0 for mpii_layer cropping_flag indicates that no layers are cropped.

[0087] The mpii_layer cropping_flag setting of 1 indicates that at least one layer is cropped, and defines the size and positioning of each layer.

[0088] mpii_cropped_position_inlayer_x[i] specifies the horizontal position of the i-th cropped layer in the original layer (e.g., the top-left pixel of the i-th cropped layer). This position can be indicated in pixels.

[0089] mpii_cropped_position_inlayer_y[i] specifies the vertical position of the i-th cropped layer in the original layer (e.g., the top-left pixel of the i-th cropped layer). This position can be indicated in pixels.

[0090] The original layer is the layer before any cropping.

[0091] mpii_cropped_size_layer_x[i] specifies the pixel size of the i-th cropped layer in the horizontal dimension.

[0092] mpii_cropped_size_layer_y[i] specifies the pixel size of the i-th cropped layer in the vertical dimension.

[0093] mpii_layer_position_intiled_picture_x[i] specifies the horizontal position of the i-th cropping layer (e.g., the top left corner) in the tiled image.

[0094] mpii_layer_position_intiled_picture_y[i] specifies the vertical position of the i-th cropping layer (e.g., the top left corner) in the tiled image.

[0095] Figure 15B It depicts the position of the clipping layer within the original layer, while Figure 15C The diagram illustrates the location of the clipping layer in the tiled image.

[0096] In another example, at least one layer is cropped, and at least one layer is subsampled to reduce the overall size of the tiled image. A method to reduce the size of less interesting layers by applying a subsampling factor has been proposed. A method to reduce the size of some layers by applying cropping to eliminate unimportant parts of these layers has also been proposed. Both cropping and subsampling can be applied to the same layer.

[0097] The SEI message can therefore be modified to include information about cropped and / or subsampled layers. In the example, the SEI message may include the number of layers, and for each layer, it may include the subsampling factor for that layer (or two subsampling factors, such as one in each direction), the cropping size of that layer, and the layer's position (e.g., location) within the original layer, as well as the position of the cropped layer within the tiled image. Table 6 provides examples. Differences from the messages in Table 1 are indicated in italics. Table 6.

[0098] A value of 0 for mpii_layer subsampling_flag indicates that no layer is subsampled.

[0099] The mpii_layer subsampling_flag being equal to 1 indicates that at least one layer is subsampled by a factor defined for each layer.

[0100] A value of 0 for mpii_layer cropping_flag indicates that no layers are cropped.

[0101] The mpii_layer cropping_flag setting of 1 indicates that at least one layer is cropped, and defines the size and positioning of each layer.

[0102] mpii_subsampling_factor[i] specifies the subsampling factor for the i-th layer.

[0103] mpii_cropped_position_inlayer_x[i] specifies the pixel position of the i-th cropped layer in the original layer in the horizontal direction.

[0104] mpii_cropped_position_inlayer_y[i] specifies the vertical pixel position of the i-th cropped layer in the original layer.

[0105] mpii_cropped_size_layer_x[i] specifies the pixel size of the i-th cropped layer in the horizontal dimension.

[0106] mpii_cropped_size_layer_y[i] specifies the pixel size of the i-th cropped layer in the vertical dimension.

[0107] mpii_layer_position_intiled_picture_x[i] specifies the horizontal position of the top-left corner of the i-th layer in the tiled image.

[0108] mpii_layer_position_intiled_picture_y[i] specifies the vertical position of the top-left corner of the i-th layer in the tiled image.

[0109] The examples in Table 6 can be combined with the examples in Tables 3 and 4, where subsampling factors can be indicated for each horizontal and vertical direction.

[0110] In another example, at least one layer is cropped (e.g., into multiple regions) and may be reassembled (e.g., the regions are repositioned). In this example, at least two distinct regions are retained after cropping a layer. At least two regions are packed together, and the newly aggregated layer is then considered to be packed within the tiled image. For the MPI content "train," the layer can be divided into several cropped regions, such as... Figure 16A As shown in Figure 16, the four layers of MPI content, referred to as the "train," are clipped into several regions that can be reassembled or aggregated in a packaged manner (the clipped regions are represented by gray areas). Figure 16B The illustration shows the reorganization of a layer with five regions. The reorganization was performed by limiting the size of the reorganized regions. The reorganized regions are described as rectangular in shape.

[0111] The SEI message can therefore be modified to include information about cropped and / or subsampling layers. In the example, the SEI message may include the number of layers, and for each layer, it includes the number of selected (e.g., cropped) regions, the size and initial position of each region in that layer, the reassembly position, and the layer's position (e.g., localization) within the tiled image. The reassembly position of a region is its location within the reassembly layer. Table 7 provides examples. Differences from the messages in Table 1 are indicated in italics. Table 7.

[0112] A value of 0 for mpii_layer cropping_flag indicates that no layers are cropped.

[0113] The mpii_layer cropping_flag setting of 1 indicates that at least one layer is cropped, and defines the size and positioning of each layer.

[0114] A value of 0 for mpii_area_layer cropping_flag indicates that no layer is cropped into regions.

[0115] The mpii_area_layer cropping_flag setting of 1 indicates that at least one layer is cropped into several regions, and defines the size and positioning of each layer.

[0116] mpii_num_areas_layers_minus1[i] +1 specifies the number of regions used for MPI representation in the i-th layer.

[0117] mpii_cropped_areas_initialposition_inlayer_x[i][j] specifies the initial position (e.g., in pixels) of the j-th region in the i-th layer in the horizontal dimension.

[0118] mpii_cropped_areas_initialposition_inlayer_y[i][j] specifies the initial position (e.g., in pixels) of the j-th region in the i-th layer in the vertical dimension.

[0119] mpii_cropped_areas_organized position_inlayer_x[i][j] specifies the position (e.g., in pixels) of the j-th region in the i-th layer after layer reorganization.

[0120] mpii_cropped_areas_organized position_inlayer_y[i][j] specifies the position (e.g., in pixels) of the j-th region in the i-th layer after layer reorganization.

[0121] mpii_cropped_size_areas_layer_x[i][j] specifies the pixel size of the j-th region in the i-th layer in the horizontal dimension.

[0122] mpii_cropped_size_areas_layer_y[i][j] specifies the pixel size of the j-th region in the i-th layer in the vertical dimension.

[0123] mpii_cropped_size_layer_x[i] specifies the pixel size of the i-th layer in the horizontal dimension.

[0124] mpii_cropped_size_layer_y[i] specifies the pixel size of the i-th layer in the vertical dimension.

[0125] mpii_cropped_position_inlayer_x[i] specifies the pixel position of the i-th cropped layer in the original layer in the horizontal direction.

[0126] mpii_cropped_position_inlayer_y[i] specifies the vertical pixel position of the i-th cropped layer in the original layer.

[0127] mpii_layer_position_intiled_picture_x[i] specifies the horizontal position of the top-left corner of the i-th layer in the tiled image.

[0128] mpii_layer_position_intiled_picture_y[i] specifies the vertical position of the top-left corner of the i-th layer in the tiled image.

[0129] In another example, at least one layer is subsampled, and at least one layer is cropped and reassembled. It is proposed to reduce the size of a layer by applying a subsampling factor to at least one layer (e.g., the less interesting layer). It is also proposed to consider retaining at least two distinct regions for a layer after cropping. The at least two regions are packed together, and the new size of the aggregated layer is then considered to be packed within the tiled image. For a given MPI content, such as... Figure 16C The content shown is divided into several clipping regions for each layer, and the first layer is also subsampled in both the horizontal and vertical directions.

[0130] The SEI message can therefore be modified to include information about cropped and / or subsampled layers. In the example, the SEI message may include the number of layers, and for each layer, it includes: the subsampling factor of the layer, the number of selected (e.g., cropped) regions per layer, the size of each region in the layer, the initial and reassembly locations, and the layer's location (e.g., positioning) within the tiled image. Table 8 provides an example. Differences from the messages in Table 1 are indicated in italics. Table 8.

[0131] A value of 0 for mpii_layer subsampling_flag indicates that no layer is subsampled.

[0132] The mpii_layer subsampling_flag being equal to 1 indicates that at least one layer is subsampled by a factor defined for each layer.

[0133] mpii_subsampling_factor[i] specifies the subsampling factor for the i-th layer.

[0134] A value of 0 for mpii_layer cropping_flag indicates that no layers are cropped.

[0135] The mpii_layer cropping_flag setting of 1 indicates that at least one layer is cropped, and defines the size and positioning of each layer.

[0136] A value of 0 for mpii_area_layer cropping_flag indicates that no layer is cropped into regions.

[0137] The mpii_area_layer cropping_flag setting of 1 indicates that at least one layer is cropped into several regions, and defines the size and position of each layer.

[0138] mpii_num_areas_layers_minus1[i] +1 specifies the number of regions used for MPI representation in the i-th layer.

[0139] mpii_cropped_areas_initialposition_inlayer_x[i][j] specifies the pixel position of the j-th region in the i-th layer in the horizontal dimension.

[0140] mpii_cropped_areas_initialposition_inlayer_y[i][j] specifies the pixel position of the j-th region in the i-th layer in the vertical dimension.

[0141] mpii_cropped_size_areas_layer_x[i][j] specifies the pixel size of the j-th region in the i-th layer in the horizontal dimension.

[0142] mpii_cropped_size_areas_layer_y[i][j] specifies the pixel size of the j-th region in the i-th layer in the vertical dimension.

[0143] mpii_cropped_areas_organized position_inlayer_x[i][j] specifies the position (e.g., in pixels) of the j-th region in the i-th layer after layer reorganization.

[0144] mpii_cropped_areas_organized position_inlayer_y[i][j] specifies the position (e.g., in pixels) of the j-th region in the i-th layer after layer reorganization.

[0145] mpii_cropped_size_layer_x[i] specifies the pixel size of the i-th layer in the horizontal dimension.

[0146] mpii_cropped_size_layer_y[i] specifies the pixel size of the i-th layer in the vertical dimension.

[0147] mpii_cropped_position_inlayer_x[i] specifies the pixel position of the i-th cropped layer in the original layer in the horizontal direction.

[0148] mpii_cropped_position_inlayer_y[i] specifies the vertical pixel position of the i-th cropped layer in the original layer.

[0149] mpii_layer_position_intiled_picture_x[i] specifies the horizontal position of the top-left corner of the i-th layer in the tiled image.

[0150] mpii_layer_position_intiled_picture_y[i] specifies the vertical position of the top-left corner of the i-th layer in the tiled image.

[0151] As mentioned above, it is possible to use, for example Figure 12 and Figure 15A The tiled image with blank spaces is shown at the end. In one example, the blank spaces in the tiled image can be implicitly filled, for example, through adaptation of the subsampling factor. For example, in Figure 12 In the case shown, layer 16 can be accessed through, as follows: Figure 17 The shown modifies only the horizontal subsampling. Layer 16 is implicitly selected based on criteria such as effective pixel rate.

[0152] In another variation, not only is the subsampling factor implicitly modified, but the cropping operation used to fill the blank space is also implicitly modified. For example, in the case shown in Figure 15, layers 16, 11, and 13 can be modified to fill the blank space. Layers 16, 11, and 13 are selected based on criteria such as the effective pixel rate in the cropped region.

[0153] In another variation, it might end with a tiled image having blank spaces at the right and bottom boundaries of width (W - (Wlayer * wLayers)) and height (H - (Hlayer * hLayers)), where Wlayer and Hlayer are the layer sizes in X and Y, respectively, and wLayers and hLayers are the number of layers in the horizontal and vertical directions, respectively. In one variation, the blank spaces are filled with layers of a shape (size) that are no larger than (W - (Wlayer * wLayers)) and no higher than (H - (Hlayer * hLayers)). This is preferably applicable when mpii_texture_opacity_arrangement_flag equals 1 (top / bottom mode of texture and opacity map). Figure 18 The concept is illustrated, in which layer 16 is inserted as a recombination layer.

[0154] With a fixed layer size, or when the reference size of a layer is defined in pixels (e.g., by default, i.e., the layer has this size), depending on the value of hLayers and the number of layers Nlayers, there may be fewer layers than the actual number of layers represented by the MPI. For example, if the number of layers represented by the MPI is 17 and hLayers is 4, then wLayers equals 4, which results in hLayers * wLayers = 16 layers, which is smaller than the number of layers equal to 17. To avoid this problem, several solutions can be considered.

[0155] In an example that can be combined with any previous example, the calculation of wLayers is modified as follows to ensure that wLayers * hLayers ≥ mpii_num_layers_minus1. For example, wLayers is set to be equal to (mpii_num_layers_minus1 + 1) * ((hLayers – 1) / hLayers).

[0156] In another example that can be combined with any previous example, hLayers and wLayers are defined in the syntax, and Nlayers is derived as (hLayers * wLayers), where Nlayers >= mpii_num_layers_minus1. Therefore, the SEI message can be modified as shown in Table 9. Differences from the message in Table 1 are indicated in italics. Table 9.

[0157] New syntax elements mpii_picture_num_layers_in_width_minus1 and mpii_picture_num_layers_in_height_minus1 have been added.

[0158] `mpii_picture_num_layers_in_width_minus1` plus 1 specifies the number of spatially packed layers in width. `mpii_picture_num_layers_in_height_minus1` plus 1 specifies the number of spatially packed layers in height. In this example, the previous syntax element `mpii_num_layers_minus1` is no longer encoded. `hLayers` is calculated as `(mpii_picture_num_layers_in_height_minus1+1)`, and `wLayers` is calculated as `(mpii_picture_num_layers_in_width_minus1+1)`.

[0159] Furthermore, this aspect is not limited to ECM, VVC, or HEVC, but can be applied to, for example, other standards and recommendations, as well as any extensions of such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described in this application may be used alone or in combination.

[0160] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0161] Note that the syntax elements used in this article, such as terms in equations and algorithms, signal (e.g., flags, patterns) labels / names, such as mpii_subsampling_factor[i], mpii_layer_position_x[i], etc., are descriptive terms. Therefore, they do not preclude the use of other syntax element names.

[0162] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of a process performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various implementations described herein, such as decoding a packed multiplane image and information describing the size and position of each layer of the packed multiplane image in relation to the packed image.

[0163] As further examples, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding; and in yet another embodiment, "decoding" refers to the entire image reconstruction process, including entropy decoding. It will be clear, and those skilled in the art, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to the broader decoding process, based on the specific context of the description.

[0164] Various implementations involve encoding. Similar to the discussion of "decoding" above, the term "encoding" as used herein can encompass all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as segmentation, differential coding, transform, quantization, and entropy coding. In various embodiments, such a process also includes, or alternatively includes, processes performed by encoders of the various implementations described herein, such as encoding a packed multiplane image and information describing the size and position of each layer of the packed multiplane image in relation to the packed image.

[0165] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. It will be clear, and those skilled in the art, whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process, based on the context of the specific description.

[0166] This disclosure describes a wide variety of information, such as, for example, syntax, which can be transmitted or stored. This information can be packaged or arranged in a variety of ways, including, for example, those common in video standards, such as placing the information in SPS, PPS, NAL units, headers (e.g., NAL unit headers or slice headers), or SEI messages. Other methods are also available, including, for example, those common in system-level or application-level standards, such as placing the information in one or more of the following: a. SDP (Session Description Protocol), a format for describing multimedia communication sessions for the purposes of session announcement and session invitation, such as as described in the RFC, and used in conjunction with RTP (Real-Time Transport Protocol) transmission.

[0167] b. DASH MPD (Media Presentation Description) descriptors, such as those used in DASH and transmitted via HTTP, are associated with a representation or set of representations to provide additional features for content representation.

[0168] c. RTP header extensions, such as those used during RTP streaming.

[0169] d. ISO-based media file formats, such as those used in OMAF, and using boxes (also called “atoms” in some specifications) as object-oriented building blocks defined by unique type identifiers and lengths.

[0170] e. An HLS (HTTP Live Stream) manifest transmitted over HTTP. A manifest can be associated with, for example, a version of content or a collection of versions to provide characteristics of that version or collection of versions.

[0171] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0172] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in a single implementation context (e.g., discussed only as a method), the features under discussion can be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, in appropriate hardware, software, and firmware. These methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0173] The reference to "an embodiment," "an embodiment," "an implementation," or "an implementation mode," as well as other variations thereof, means that the specific features, structures, characteristics, etc., described in connection with that embodiment are included in at least one embodiment. Therefore, the phrases "in an embodiment," "in an embodiment," "in an implementation mode," or "in an implementation mode," and any other variations appearing throughout this application, do not necessarily all refer to the same embodiment.

[0174] Furthermore, this application may involve "determining" various pieces of information. Determining information may include, for example, one or more of estimated information, calculated information, predicted information, or information retrieved from memory.

[0175] Furthermore, this application may involve "accessing" various information fragments. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0176] Furthermore, this application may involve "receiving" various pieces of information. Like "access," receiving is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) or one or more of them. Further, "receiving" is typically involved in one or more ways during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0177] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “…at least one of…” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” this wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as are listed.

[0178] Furthermore, as used herein, the term "signal" refers, among other things, to indicating something to the corresponding decoder. For example, in some embodiments, the encoder signals a specific one of a plurality of subsampling factors, the size and position of the clipping layer. In this way, in embodiments, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder may transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has that specific parameter as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select that specific parameter. In various embodiments, bit savings are achieved by avoiding the transmission of any actual function. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. While the foregoing refers to the verb form of the term "signal," the term "signal" may also be used as a noun herein.

[0179] It will be apparent to those skilled in the art that implementations can generate a wide variety of signals formatted to carry, for example, information that can be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a wide variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0180] This document describes numerous examples. Features of the examples may be provided individually or in any combination across various claim classes and types. Furthermore, examples may include one or more of the features, devices, or aspects described herein, individually or in any combination across various claim classes and types. For example, the features described herein may be implemented in a bitstream or signal that includes information generated as described herein. According to any of the described embodiments, the information may allow a decoder to decode a bitstream, an encoder, a bitstream, and / or a decoder. For example, the features described herein may be implemented by creating and / or transmitting and / or receiving and / or decoding a bitstream or signal. For example, the features described herein may be implemented as a method, process, apparatus, medium storing instructions, medium storing data, or signal. For example, the features described herein may be implemented by a TV, set-top box, mobile phone, tablet computer, or other electronic device performing decoding. The TV, set-top box, mobile phone, tablet computer, or other electronic device may display (e.g., using a monitor, screen, or other type of display) the obtained image (e.g., an image reconstructed from a residual of a video bitstream). The TV, set-top box, mobile phone, tablet computer, or other electronic device may receive a signal including an encoded image and perform decoding.

[0181] Several embodiments have been described above. These embodiments may be provided individually or in any combination, spanning various claim classes and types.

[0182] A decoding method is disclosed, including: Decoding at least one image (e.g., a tiled image) comprising a packing layer of multiplanar image representation, wherein the size of at least one layer is reduced relative to a reference size. Decoding information describing each layer in terms of size and position within at least one image; and Based on at least one decoded image and the decoded information, a multi-plane image representation comprising multiple layers is reconstructed.

[0183] In the example, decoding the information describing each layer in terms of size and position in at least one image includes decoding a supplemental enhancement information message that includes a reference size, the number of layers, and for each layer, at least one subsampling factor and information indicating the position of that layer in the tiled image.

[0184] In the example, the supplemental enhancement message includes the horizontal subsampling factor and the vertical subsampling factor.

[0185] In the example, decoding the information describing each layer in terms of size and position in at least one image includes decoding a supplemental enhancement message that includes the number of layers and, for each layer, information about the size of the cropped layer, the position of the cropped layer in the original layer, and the position of the cropped layer in the tiled image.

[0186] In the example, decoding the information describing each layer in terms of size and position in at least one image includes decoding a supplemental enhancement message that includes the number of layers, and for each layer, one or two subsampling factors, the size of the cropped layer, and information indicating the position of the cropped layer in the original layer and the position of the cropped layer in the tiled image.

[0187] In the example, decoding the information describing each layer in terms of size and position in at least one image includes decoding a supplemental enhancement information message that includes the number of layers, and for each layer, the number of cropped regions, the size of each region and information indicating the position of each region in the initial layer, information indicating the position of each region in the reconstructed layer, and information indicating the position of the layer in the tiled image.

[0188] In the example, the supplemental enhancement message includes syntax elements indicating the number of space packing layers in height and syntax elements indicating the number of space packing layers in width.

[0189] An encoding method is disclosed, including: Obtain a multi-planar image representation including multiple layers; Pack multiple layers into at least one image (e.g., a tiled image), wherein the size of at least one layer is reduced relative to a reference size; Encode at least one image; and Information describing each layer is encoded in terms of size and position within at least one image.

[0190] In the example, encoding information describing each layer in terms of size and position in at least one image includes encoding supplemental enhancement information messages, which include a reference size, number of layers, and for each layer, at least one subsampling factor and information indicating the position of that layer in the tiled image.

[0191] In the example, the supplemental enhancement message includes the horizontal subsampling factor and the vertical subsampling factor.

[0192] In the example, encoding information describing the size and position of each layer in at least one image includes encoding supplemental enhancement information messages that include the number of layers and, for each layer, information about the size of the cropped layer, the position of the cropped layer in the original layer, and the position of the cropped layer in the tiled image.

[0193] In the example, encoding information describing each layer in terms of size and position in at least one image includes encoding supplemental enhancement information messages that include the number of layers, and for each layer, one or two subsampling factors, the size of the cropped layer, and information indicating the position of the cropped layer in the original layer and the position of the cropped layer in the tiled image.

[0194] In the example, encoding information describing each layer in terms of size and position in at least one image includes encoding supplemental enhancement information messages that include the number of layers, and for each layer, the number of cropped regions, the size of each region and information indicating the position of each region in the initial layer, information indicating the position of each region in the reconstructed layer, and information indicating the position of the layer in the tiled image.

[0195] In the example, the supplemental enhancement message includes syntax elements indicating the number of space packing layers in height and syntax elements indicating the number of space packing layers in width.

[0196] In the example, where at least one layer includes a number of valid pixels below a certain value, the size of at least one layer is reduced relative to the reference size by subsampling at least one layer.

[0197] In the example, the size of at least one layer is reduced relative to the reference size by cropping at least one layer to retain only a portion of that layer with valid pixels.

[0198] In the example, at least one blank portion of the tiled image is filled by implicitly or explicitly modifying the subsampling factor or the cropped region.

[0199] A decoding apparatus is disclosed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform a decoding method.

[0200] An encoding apparatus is disclosed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform an encoding method.

Claims

1. A decoding method, comprising: Decoding at least one tiled image comprising a packing layer of multiplanar image representation, wherein the size of at least one layer is reduced relative to a reference size. Decode information describing the size and position of each layer in at least one tiled image; as well as Based on at least one tiled image decoded and the decoded information, a multi-plane image representation comprising multiple layers is reconstructed.

2. The method according to claim 1, wherein, Decoding information describing the size and position of each layer in at least one tiled image includes decoding supplemental enhancement information messages, which include a reference size, the number of layers, and for each layer, at least one subsampling factor and information indicating the position of the layer in the tiled image.

3. The method according to claim 2, wherein, The supplemental enhancement information message includes a horizontal subsampling factor and a vertical subsampling factor.

4. The method according to claim 1, wherein, Decoding information describing the size and position of each layer in at least one tiled image includes decoding supplemental enhancement information messages, which include the number of layers and, for each layer, information about the size of the cropped layer, the position of the cropped layer in the original layer, and the position of the cropped layer in the tiled image.

5. The method according to claim 1, wherein, Decoding information describing the size and position of each layer in at least one tiled image includes decoding supplemental enhancement information messages, which include the number of layers and, for each layer, one or two subsampling factors, the size of the cropped layer, and information indicating the position of the cropped layer in the original layer and the position of the cropped layer in the tiled image.

6. The method according to claim 1, wherein, Decoding information describing the size and position of each layer in at least one tiled image includes decoding supplemental enhancement information messages, which include the number of layers, and for each layer, the number of cropped regions, the size of each region, information indicating the position of each region in the initial layer, information indicating the position of each region in the reconstructed layer, and information indicating the position of the layer in the tiled image.

7. The method according to any one of claims 2-6, wherein, The supplemental enhancement information message includes syntax elements indicating the number of spatial packing layers in height and syntax elements indicating the number of spatial packing layers in width.

8. An encoding method, comprising: Obtain a multi-planar image representation including multiple layers; Pack multiple layers into at least one tiled image, wherein the size of at least one layer is reduced relative to a reference size; Encode at least one image; as well as Information describing each layer in terms of size and position in at least one image is encoded.

9. The method according to claim 8, wherein, Encoding information describing each layer in terms of size and position in at least one image includes encoding supplemental enhancement information messages, which include a reference size, number of layers, and for each layer, at least one subsampling factor and information indicating the position of the layer in the tiled image.

10. The method according to claim 9, wherein, The supplemental enhancement information message includes a horizontal subsampling factor and a vertical subsampling factor.

11. The method according to claim 8, wherein, Encoding information describing the size and position of each layer in at least one image includes encoding supplemental enhancement information messages, which include the number of layers and, for each layer, information about the size of the cropped layer, the position of the cropped layer in the original layer, and the position of the cropped layer in the tiled image.

12. The method according to claim 8, wherein, Encoding information describing each layer in terms of size and position in at least one image includes encoding supplemental enhancement information messages, which include the number of layers and, for each layer, one or two subsampling factors, the size of the cropped layer, and information indicating the position of the cropped layer in the original layer and the position of the cropped layer in the tiled image.

13. The method according to claim 8, wherein, Encoding information describing the size and position of each layer in at least one image includes encoding supplemental enhancement information messages, which include the number of layers, and for each layer, the number of cropped regions, the size of each region and information indicating the position of each region in the initial layer, information indicating the position of each region in the reconstructed layer, and information indicating the position of the layer in the tiled image.

14. The method according to claim 9 or 11, wherein, In cases where at least one tiled image includes blank areas, the blank areas are filled by modifying the subsampling factor or the size of the cropping layer.

15. A decoding apparatus comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform the method of any one of claims 1-7.

16. An encoding apparatus comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform the method of any one of claims 8-14.

Images