Multi-direction 2d snapshot image tracks for v3c content

Abstract

Embodiments of the device relate to improvements to dynamic mesh coding for multi-directional image tracks. The device obtains a track comprising one or more samples, wherein a corresponding one of the one or more samples comprises a plurality of coded two-dimensional projection images of a coded volume frame as a plurality of sub-samples. The device decodes at least one of the plurality of coded two-dimensional projection images to generate at least one of the plurality of two-dimensional projection images. The device renders the at least one two-dimensional projection image.

Description

Technical Field

[0001] This disclosure relates to dynamic mesh coding, and more specifically, to multi-directional 2D snapshot image trajectories for, for example, but not limited to, coding content based on visual volumetric video (V3C). Background Technology

[0002] Currently, the Moving Picture Experts Group (MPEG) is researching volumetric content compression. Video-based point cloud compression (V3C) and video-based dynamic mesh coding (V-DMC) both utilize various techniques, including traditional video compression techniques, to compress volumetric content such as point clouds and meshes. These techniques include: ISO / IEC 23090-5 video-based point cloud compression, ISO / IEC 23090-29 video-based dynamic mesh coding (V-DMC), and ISO / IEC 23090-10 video-based point cloud compression data carriers.

[0003] Both V3C and V-DMC use a variety of techniques, including traditional video compression techniques, to compress volumetric content such as point clouds and meshes. Compressing volumetric content offers significant benefits in terms of saving resources used for storing and delivering content. However, it presents challenges for quick previewing or playing back effects, similar to those faced by any other compressed video data. Since unidirectional or multidirectional correlation coding can be applied to further improve compression efficiency, multiple video frames must be decoded to obtain the specific frames of the volumetric content. If the random access points of the components are misaligned or the frame rates of the components differ from each other, then a larger number of video frames should be decoded to obtain the result. Therefore, while quick previewing or playing back effects of uncompressed volumetric content is a simple task, when the content is compressed, the process becomes quite complex, resource-intensive, and time-consuming.

[0004] The descriptions set forth in the Background section should not be construed as prior art simply because they are set forth therein. The Background section may describe aspects or embodiments of this disclosure. Summary of the Invention

[0005] Technical solution One aspect of this disclosure provides an apparatus including a communication interface and a processor operatively coupled to the communication interface. The communication interface is configured to acquire a track. The track includes one or more samples. A corresponding sample among the one or more samples includes a plurality of coded two-dimensional projection images of coded volumetric frames as a plurality of subsamples. The processor is configured to decode at least one of the plurality of coded two-dimensional projection images to generate at least one two-dimensional projection image. The processor is further configured to render the at least one two-dimensional projection image.

[0006] One aspect of this disclosure provides an apparatus including a communication interface and a processor operatively coupled to the communication interface. The processor is configured to encode a plurality of two-dimensional projection images of one or more volumetric frames to generate a plurality of encoded two-dimensional projection images. The processor is also configured to generate a track comprising one or more samples. Each of the one or more samples is associated with a corresponding volumetric frame among the one or more volumetric frames and includes at least two encoded two-dimensional projection images associated with the volumetric frame as a plurality of subsamples. The processor is further configured to transmit the track.

[0007] One aspect of this disclosure provides a method performed by a device. The method includes obtaining a track. The track includes one or more samples. A corresponding sample among the one or more samples includes multiple coded two-dimensional projection images of coded volumetric frames as multiple sub-samples. The method further includes decoding at least one of the multiple coded two-dimensional projection images to generate at least one two-dimensional projection image. The method further includes presenting the at least one two-dimensional projection image. Attached Figure Description

[0008] Figure 1 An example communication system 100 according to an embodiment of the present disclosure is shown.

[0009] Figure 2 An example electronic device according to an embodiment of the present disclosure is shown.

[0010] Figure 3 An example electronic device according to an embodiment of the present disclosure is shown.

[0011] Figure 4 A block diagram of an encoder for encoding intra-frames according to an embodiment is shown.

[0012] Figure 5 A block diagram for a decoder according to an embodiment is shown.

[0013] Figure 6A basic block diagram of a 2D snapshot image SEI message processor according to an embodiment of the present disclosure is shown.

[0014] Figure 7 An example of the content of a V3C bitstream according to an embodiment is shown.

[0015] Figure 8 This is a flowchart illustrating the operation of an encoder according to an embodiment.

[0016] Figure 9 This is a flowchart illustrating the operation of the decoder according to an embodiment.

[0017] In one or more embodiments, not all of the components depicted in each figure may be required, and one or more embodiments may include additional components not shown in the figures. The arrangement and type of components may be changed without departing from the scope of this disclosure. Within the scope of this disclosure, additional components, different components, or fewer components may be utilized. Detailed Implementation

[0018] The detailed description set forth below with reference to the accompanying drawings is intended to describe various embodiments and not to represent the only possible implementation of the subject matter. Rather, the detailed description includes specific details for the purpose of providing a thorough understanding of the subject matter of the invention. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the scope of this disclosure. Therefore, the drawings and description are to be considered illustrative rather than restrictive in nature. The same reference numerals denote the same elements.

[0019] 360° video and 3D volumetric video are emerging as new ways to experience immersive content due to the readily available availability of powerful handheld devices such as smartphones. While 360° video provides consumers with an immersive, “real-life,” “being there” experience by capturing a 360° outside-in view of the world, 3D volumetric video offers a full 6DoF experience of being present within and moving within the content. Users can interactively change their viewpoint and dynamically view any part of the captured scene or object they desire. Display and navigation sensors track the user’s head movements in real time to determine the area of ​​the 360° video or volumetric content the user wants to view or interact with. Multimedia data that is inherently three-dimensional (3D), such as point clouds or 3D polygon meshes, can be used in immersive environments.

[0020] A point cloud is a collection of 3D points representing the surface or volume of an object, along with attributes such as color, normals, reflectivity, and point size. Point clouds are widely used in various applications, including games, 3D mapping, visualization, medical applications, augmented reality, virtual reality, autonomous driving, multi-view replay, and 6DoF immersive media, to name a few. If point clouds are not compressed, they typically require significant bandwidth for transmission. Due to high bit rate requirements, point clouds are usually compressed before transmission. Dedicated hardware is typically required to compress 3D objects such as point clouds. To avoid the need for dedicated hardware to compress 3D point clouds, they can be transformed into traditional 2D frames, which can then be compressed and subsequently reconstructed for the user's viewing.

[0021] Polygonal 3D meshes (especially triangular meshes) are another common format for representing 3D objects. A mesh typically includes a set of vertices, edges, and faces used to represent the surfaces of a 3D object. A triangular mesh is a simple polygonal mesh where faces are simple triangles covering the surfaces of a 3D object. Typically, one or more attributes can be associated with a mesh. In one scene, one or more attributes can be associated with each vertex in the mesh. For example, texture attributes (RGB) can be associated with each vertex. In another scene, each vertex can be associated with a pair of coordinates (u, v). The (u, v) coordinates can point to the location in the texture map associated with the mesh. For example, the (u, v) coordinates can refer to the row index and column index in the texture map, respectively. A mesh can be thought of as a point cloud with additional connectivity information.

[0022] Point clouds or meshes can be dynamic, meaning they can change over time. In these cases, a point cloud or mesh at a specific moment can be referred to as a point cloud frame or a mesh frame, respectively.

[0023] Because point clouds and meshes contain a large amount of data, they need to be compressed for efficient storage and transmission. This is especially true for dynamic point clouds and meshes, which may contain 60 or more frames per second.

[0024] The accompanying drawings and various embodiments used to describe the principles of this disclosure are for illustrative purposes only and should not be construed as limiting the scope of this disclosure in any way. Those skilled in the art will understand that the principles of this disclosure can be implemented in any suitably arranged system or apparatus.

[0025] Figure 1 An example communication system 100 according to an embodiment of the present disclosure is shown. Figure 1 The embodiment of the communication system 100 shown is for illustrative purposes only. Other embodiments of the communication system 100 may be used without departing from the scope of this disclosure.

[0026] Communication system 100 includes a network 102 that facilitates communication between various components within the communication system 100. For example, network 102 can communicate between network addresses to transmit IP packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, or other information. Network 102 includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or part of a global network such as the Internet, or one or more other communication systems located in one or more locations.

[0027] In this example, network 102 facilitates communication between server 104 and various client devices 106 to 116. Client devices 106 to 116 may be, for example, smartphones, tablets, laptops, personal computers, TVs, interactive displays, wearable devices, HMDs, etc. Server 104 may represent one or more servers. Each server 104 includes any suitable computing or processing device that can provide computing services to one or more client devices, such as client devices 106 to 116. Each server 104 may, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication via network 102. As described in more detail below, server 104 may transmit compressed bitstreams representing point clouds or meshes to one or more display devices, such as client devices 106 to 116. In some embodiments, each server 104 may include an encoder.

[0028] Each client device 106 to 116 represents any suitable computing or processing device that interacts with at least one server (such as server 104) or one or more other computing devices via network 102. Client devices 106 to 116 include desktop computers 106, mobile phones or mobile devices 108 (such as smartphones), PDAs 110, laptop computers 112, tablet computers 114, and HMDs 116. However, any other or additional client devices may be used in the communication system 100. A smartphone represents a class of mobile devices 108 that are handheld devices with a mobile operating system and integrated mobile broadband cellular network connectivity for voice, short message service (SMS), and Internet data communication. An HMD 116 may display a 360° scene including one or more dynamic or static 3D point clouds. In some embodiments, any client device 106 to 116 may include an encoder, a decoder, or both. For example, mobile device 108 can record 3D volumetric video and then encode the video so that it can be transmitted to one of client devices 106 to 116. In another example, laptop computer 112 can be used to generate a 3D point cloud or mesh, which is then encoded and transmitted to one of client devices 106 to 116.

[0029] In this example, some client devices 108 to 116 communicate indirectly with network 102. For example, mobile device 108 and PDA 110 communicate via one or more base stations (such as cellular base stations or eNodeBs (eNBs)) 118. Additionally, laptop computer 112, tablet computer 114, and HMD 116 communicate via one or more wireless access points 120 (such as IEEE 802.11 wireless access points). Note that these are for illustrative purposes only, and each client device 106 to 116 may communicate directly with network 102 or indirectly with network 102 via any one or more suitable intermediaries or networks. In some embodiments, server 104 or any client device 106 to 116 may be used to compress point clouds or meshes, generate bitstreams representing point clouds or meshes, and transmit the bitstreams to another client device, such as any client device 106 to 116.

[0030] In some embodiments, any of client devices 106 to 114 securely and efficiently transmits information to another device, such as, for example, server 104. Furthermore, any of client devices 106 to 116 can trigger information transmission between itself and server 104. Any of client devices 106 to 114 can function as a VR display when attached to headphones via a bracket, and functions similarly to HMD 116. For example, mobile device 108, when attached to a bracket system and worn on a user's eyes, can function similarly to HMD 116. Mobile device 108 (or any other client device 106 to 116) can trigger information transmission between itself and server 104.

[0031] In some embodiments, any of client devices 106-116 or server 104 may create, compress, transmit, receive, decode, render, or a combination thereof 3D point clouds or meshes. For example, server 104 may then compress the 3D point cloud or mesh to generate a bitstream, and then transmit the bitstream to one or more of client devices 106-116. As another example, one of client devices 106-116 may compress the 3D point cloud or mesh to generate a bitstream, and then transmit the bitstream to another client device among client devices 106-116 or server 104.

[0032] although Figure 1 An example of a communication system 100 is shown, but it is possible to modify it. Figure 1 Various changes can be made. For example, communication system 100 can include any number of each component in any suitable arrangement. Typically, computing and communication systems have a wide variety of configurations, and Figure 1 This disclosure is not intended to limit the scope to any particular configuration. Although Figure 1 This paper illustrates an operating environment in which the various features disclosed in this patent document can be used, but these features can be used in any other suitable system.

[0033] Figure 2 and Figure 3 An example electronic device is shown according to an embodiment of the present disclosure.

[0034] In particular, Figure 2 Example server 200 is shown, and server 200 can represent Figure 1 Server 104 in the context of server 200. Server 200 may represent one or more encoders, decoders, local servers, remote servers, cluster computers, and components acting as a single seamless resource pool, cloud-based servers, etc. Server 200 may be... Figure 1One or more client devices from client devices 106 to 116 or another server can access the server.

[0035] Server 200 may represent one or more local servers, one or more compression servers, or one or more encoding servers, such as encoders. In some embodiments, the encoder may perform decoding. Figure 2 As shown, server 200 includes bus system 205, wherein bus system 205 supports communication between at least one processing device (e.g., processor 210), at least one storage device 215, at least one communication interface 220 and at least one input / output (I / O) unit 225.

[0036] Processor 210 executes instructions that can be stored in memory 230. Processor 210 may include one or more processors or other devices of any suitable number and type in any suitable arrangement. Example types of processor 210 include microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays, application-specific integrated circuits (ASICs), and discrete circuits.

[0037] In some embodiments, processor 210 may encode a 3D point cloud or mesh stored in storage device 215. In some embodiments, the operation of encoding the 3D point cloud also includes decoding the 3D point cloud or mesh to ensure that when the point cloud or mesh is reconstructed, the reconstructed 3D point cloud or mesh matches the 3D point cloud or mesh before encoding.

[0038] Memory 230 and persistent storage 235 are examples of storage device 215, representing one or more of any structures capable of storing and facilitating the retrieval of information such as temporary or permanent data, program code, or other suitable information. Memory 230 may represent random access memory or one or more other suitable volatile or non-volatile storage devices. For example, instructions stored in memory 230 may include instructions for decomposing a point cloud into patches, instructions for packing patches onto 2D frames, instructions for compressing 2D frames, and instructions for encoding 2D frames in a specific order to generate a bitstream. Instructions stored in memory 230 may also include instructions for use in 360° omnidirectional scenes (such as through VR headsets, such as...) Figure 1 Instructions for rendering point clouds on the HMD (116) being viewed. Persistent storage 235 may contain one or more components or devices that support long-term storage of data, such as read-only memory, hard disk drive, flash memory, or optical disk.

[0039] Communication interface 220 supports communication with other systems or devices. For example, communication interface 220 may include features that facilitate communication via... Figure 1 The communication interface 220 is a network interface card or wireless transceiver for communication on network 102. The communication interface 220 can support communication via one or more suitable physical or wireless communication links. For example, the communication interface 220 can transmit a bitstream containing a 3D point cloud to another device, such as one of client devices 106 to 116.

[0040] I / O unit 225 allows for data input and output. For example, I / O unit 225 can provide connectivity for user input via a keyboard, mouse, keypad, touchscreen, or other suitable input device. I / O unit 225 can also transmit output to a display, printer, or other suitable output device. However, note that I / O unit 225 can be omitted, such as when I / O interaction with server 200 occurs via a network connection.

[0041] Note that, although Figure 2 Described as representing Figure 1 The server 104 may be used, but the same or similar architecture may be used in one or more client devices 106 to 116. For example, desktop computer 106 or laptop computer 112 may have the same architecture as the server 104. Figure 2 The structures shown are the same or similar.

[0042] Figure 3 An example electronic device 300 is shown, and the electronic device 300 can represent Figure 1 One or more of the client devices 106 to 116. Electronic device 300 may be a mobile communication device, such as, for example, a mobile station, a user station, a wireless terminal, or a desktop computer (similar to...). Figure 1 Desktop computers 106), portable electronic devices (similar to) Figure 1 Mobile devices 108, PDA 110, laptop computer 112, tablet computer 114, or HMD 116, etc. In some embodiments, Figure 1 One or more of the client devices 106 to 116 may include configurations identical or similar to those of electronic device 300. In some embodiments, electronic device 300 is an encoder, a decoder, or both. For example, electronic device 300 may be used for data transmission, image or video compression, image or video decompression, encoding, decoding, and media rendering applications.

[0043] like Figure 3 As shown, the electronic device 300 includes an antenna 305, a radio frequency (RF) transceiver 310, a transmit (TX) processing circuit 315, a microphone 320, and a receive (RX) processing circuit 325.

[0044] RF transceiver 310 may include, for example, RF transceivers, Bluetooth transceivers, Wi-Fi transceivers, ZigBee transceivers, infrared transceivers, and various other wireless communication signals. Electronic device 300 also includes a speaker 330, a processor 340, an input / output (I / O) interface (IF) 345, an input device 350, a display 355, a memory 360, and one or more sensors 365. The memory 360 includes an operating system (OS) 361 and one or more applications 362.

[0045] RF transceiver 310 receives incoming RF signals from an access point (such as a base station, Wi-Fi router, or BLUETOOTH device) or other devices on network 102 (such as Wi-Fi, BLUETOOTH, cellular, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network) from antenna 305. RF transceiver 310 down-converts the incoming RF signals to generate an intermediate frequency (IF) or baseband signal. The IF or baseband signal is sent to RX processing circuitry 325, which generates a processed baseband signal by filtering, decoding, and / or digitizing the baseband or IF signal. RX processing circuitry 325 transmits the processed baseband signal to speaker 330 (e.g., for voice data) or to processor 340 for further processing (e.g., for web browsing data).

[0046] The TX processing circuit 315 receives analog or digital voice data from the microphone 320, or other outgoing baseband data from the processor 340. Outgoing baseband data may include web data, email, or interactive video game data. The TX processing circuit 315 encodes, multiplexes, and / or digitizes the outgoing baseband data to generate a processed baseband or intermediate frequency (IF) signal. The RF transceiver 310 receives the outgoing processed baseband or IF signal from the TX processing circuit 315 and up-converts the baseband or IF signal into an RF signal transmitted via the antenna 305.

[0047] Processor 340 may include one or more processors or other processing devices. Processor 340 may execute instructions stored in memory 360 (such as OS 361) to control the overall operation of electronic device 300. For example, processor 340 may control the reception of forward channel signals and the transmission of reverse channel signals by RF transceiver 310, RX processing circuitry 325, and TX processing circuitry 315 according to known principles. Processor 340 may include one or more processors or other devices of any suitable number and type in any suitable arrangement. For example, in some embodiments, processor 340 includes at least one microprocessor or microcontroller. Example types of processor 340 include microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays, application-specific integrated circuits, and discrete circuits.

[0048] Processor 340 is also capable of executing other processes and programs residing in memory 360, such as operations for receiving and storing data. Processor 340 may move data into or out of memory 360 as needed by the executing process. In some embodiments, processor 340 is configured to execute one or more applications 362 based on OS 361 or in response to signals received from one or more external sources or operators. For example, applications 362 may include encoders, decoders, VR or AR applications, camera applications (for still images and video), video call applications, email clients, social media clients, SMS messaging clients, virtual assistants, etc. In some embodiments, processor 340 is configured to receive and transmit media content.

[0049] The processor 340 is also integrated with an I / O interface 345, which provides the electronic device 300 with the ability to connect to other devices, such as client devices 106 to 114. The I / O interface 345 is the communication path between these accessories and the processor 340.

[0050] Processor 340 is also integrated with input device 350 and display 355. An operator of electronic device 300 can use input device 350 to input data or information into electronic device 300. Input device 350 may be a keyboard, touchscreen, mouse, trackball, voice input device, or other device capable of serving as a user interface allowing the user to interact with electronic device 300. For example, input device 350 may include voice recognition processing, allowing the user to input voice commands. In another example, input device 350 may include a touch panel, (digital) pen sensor, key, or ultrasonic input device. Touch panel may recognize touch input using at least one scheme, such as capacitive, pressure-sensitive, infrared, or ultrasonic. By providing additional input to processor 340, input device 350 may be associated with one or more sensors 365 and / or cameras. In some embodiments, sensors 365 include one or more inertial measurement units (IMUs) (such as accelerometers, gyroscopes, and magnetometers), motion sensors, optical sensors, cameras, pressure sensors, heart rate sensors, altimeters, etc. The input device 350 may also include control circuitry. In a capacitive scheme, the input device 350 can detect touch or proximity.

[0051] Display 355 may be a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED), an active-matrix OLED (AMOLED), or other display capable of displaying text and / or graphics such as those from websites, videos, games, images, etc. The size of display 355 may be adjusted to fit an HMD. Display 355 may be a single display or multiple displays capable of creating stereoscopic displays. In some embodiments, display 355 is a head-up display (HUD). Display 355 may display 3D objects, such as 3D point clouds or meshes.

[0052] Memory 360 is integrated with processor 340. A portion of memory 360 may include RAM, and another portion of memory 360 may include flash memory or other ROM. Memory 360 may include persistent storage (not shown), wherein the persistent storage represents one or more structures capable of storing and facilitating the retrieval of information such as data, program code, and / or other suitable information. Memory 360 may include one or more components or devices supporting long-term storage of data, such as read-only memory, hard disk drive, flash memory, or optical disk. Memory 360 may also contain media content. Media content may include various types of media, such as images, videos, 3D content, VR content, AR content, 3D point clouds, meshes, etc.

[0053] The electronic device 300 also includes one or more sensors 365, wherein the one or more sensors 365 can measure physical quantities or detect the activation state of the electronic device 300 and convert the measured or detected information into electrical signals. For example, the sensor 365 may include one or more buttons for touch input, a camera, a gesture sensor, an IMU sensor (such as a gyroscope or gyroscope sensor and accelerometer), an eye-tracking sensor, a barometric pressure sensor, a magnetic sensor or magnetometer, a grip sensor, a proximity sensor, a color sensor, a biophysical sensor, a temperature / humidity sensor, an illuminance sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalography (EEG) sensor, an electrocardiography (ECG) sensor, an IR sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, and a color sensor (such as a red-green-blue (RGB) sensor), etc. The sensor 365 may also include control circuitry for controlling any of the included sensors.

[0054] As discussed in more detail below, one or more of these sensors 365 can be used to control the user interface (UI), detect UI input, determine the user's orientation and facing direction for displaying 3D content, etc. Any of these sensors 365 may be located within the electronic device 300, within a secondary device operatively connected to the electronic device 300, within an earpiece configured to hold the electronic device 300, or in a single device including the earpiece in the electronic device 300.

[0055] Electronic device 300 can create media content, such as generating virtual objects or capturing (or recording) content via a camera. Electronic device 300 can encode the media content to generate a bitstream, allowing the bitstream to be directly transmitted to another electronic device or, for example, via... Figure 1 Network 102 is transmitted indirectly. Electronic device 300 can receive bit streams directly from another electronic device, or through, for example, via... Figure 1 Network 102 indirectly receives bit streams.

[0056] although Figure 2 and Figure 3 Examples of electronic devices are shown, but may vary. Figure 2 and Figure 3 Make various changes. For example, Figure 2 and Figure 3The various components can be combined, further subdivided, or omitted, and additional components can be added as needed. As a specific example, processor 340 can be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). Furthermore, as with computing and communication, electronic devices and servers can have a wide variety of configurations, and Figure 2 and Figure 3 This disclosure is not limited to any particular electronic device or server.

[0057] Figure 4 A block diagram of an encoder for encoding intra-frames according to an embodiment is shown.

[0058] like Figure 4 As shown, the encoder 400 that encodes intra-frames according to the embodiment may include a quantizer 401, a static mesh encoder 403, a static mesh decoder 405, a shift updater 407, a wavelet transform 409, a quantizer 411, an image packer 413, a video encoder 415, an image unpacker 417, an inverse quantizer 419, an inverse wavelet transform 421, an inverse quantizer 423, a deformable mesh reconstructor 425, an attribute transfer module 427, a fill module 429, a color space converter 431, a video encoder 433, a multiplexer 435, and a controller 437.

[0059] Quantizer 401 can quantize the base mesh m(i) to generate a quantized base mesh. In some embodiments, the base mesh may have fewer vertices than the original mesh.

[0060] The static mesh encoder 403 can encode and compress the quantized base mesh to generate a compressed base mesh bitstream. In some embodiments, the base mesh can be compressed in a lossy or lossless manner. In some embodiments, an existing mesh codec such as Draco can be used to compress the base mesh.

[0061] The static grid decoder 405 can decode the compressed underlying grid bitstream to generate a reconstructed quantized underlying grid m'(i).

[0062] The displacement updater 407 can update the displacement d(i) based on the subdivided base mesh m(i) and the reconstructed quantized base mesh m'(i) to generate the updated displacement d'(i). The reconstructed base mesh can be subdivided, and then the displacement field between the original mesh and the subdivided reconstructed base mesh can be calculated. In inter-frame coding of the mesh frame, the base mesh can be encoded by sending vertex motion instead of directly compressing the base mesh. In either case, a displacement field can be created. The displacement field, along with the modified property map, can be encoded using a video codec and can also be included as part of the V-DMC bitstream.

[0063] Wavelet transform 409 can perform wavelet transform using the updated displacement d'(i) to generate displacement wavelet coefficients e(i). The wavelet transform can include a series of prediction and update lifting steps.

[0064] The quantizer 411 can quantize the displacement wavelet coefficients e(i) to generate quantized displacement wavelet coefficients e'(i). The quantized displacement wavelet coefficients can be represented by an array dispQuantCoeffArray.

[0065] Image packer 413 can pack quantized displacement wavelet coefficients e'(i) into a 2D image including the packed quantized displacement wavelet coefficients dispQuantCoeffFrame. In this disclosure, the 2D video frame may be referred to as a displacement frame or a displacement video frame.

[0066] The video encoder 415 can encode the packed, quantized displacement wavelet coefficients dispQuantCoeffFrame to generate a compressed displacement bitstream.

[0067] Image unpacker 417 can unpack the packed quantized displacement wavelet coefficients dispQuantCoeffFrame to generate an array of quantized displacement wavelet coefficients dispQuantCoeffArray.

[0068] The dequantizer 419 can dequantize the array of quantized shift wavelet coefficients, dispQuantCoeffArray, to generate shift wavelet coefficients.

[0069] The inverse wavelet transform 421 can perform an inverse wavelet transform using the displacement wavelet coefficients to generate the reconstructed displacement d''(i).

[0070] The dequantizer 423 can dequantize the reconstructed quantized base grid m'(i) to generate the reconstructed base grid m''(i).

[0071] The deformable mesh reconstructor 425 can generate a reconstructed deformable mesh DM(i) based on the reconstructed displacement D''(i) and the reconstructed base mesh m''(i).

[0072] The attribute transfer module 427 can update the attribute graph A(i) based on the static / dynamic mesh m(i) and the reconstructed deformable mesh DM(i) to generate an updated attribute graph A'(i). The attribute graph can be a texture graph, but other attributes can also be sent.

[0073] The filling module 429 can perform filling to fill blank areas in the updated property graph A'(i) in order to remove high-frequency components.

[0074] Color space converter 431 can perform color space conversion on the filled and updated attribute graph A'(i).

[0075] The video encoder 433 can encode the output of the color space converter 431 to generate a compressed attribute bitstream.

[0076] Multiplexer 435 can multiplex the compressed base grid bitstream, the compressed shift bitstream, and the compressed attribute bitstream to generate a compressed bitstream b(i).

[0077] The controller 437 can control the module of the encoder 400.

[0078] Figure 5 A block diagram for a decoder according to an embodiment is shown.

[0079] like Figure 5 As shown, the decoder 500 may include a demultiplexer 501, a switch 503, a static mesh decoder 505, a mesh buffer 507, a motion decoder 509, a basic mesh reconstructor 511, a switch 513, an inverse quantizer 515, a video decoder 521, an image unpacker 523, an inverse quantizer 525, an inverse wavelet transform 527, a deformable mesh reconstructor 529, a video decoder 531, and a color space converter 533.

[0080] Demultiplexer 501 can receive compressed bit stream b(i) from encoder 400 to extract compressed base grid bit stream, compressed displacement bit stream and compressed attribute bit stream from compressed bit stream b(i).

[0081] Switch 503 can determine whether the compressed base mesh bitstream contains inter-coded mesh frames or intra-coded mesh frames. If the compressed base mesh bitstream contains inter-coded mesh frames, switch 503 can transmit the inter-coded mesh frames to motion decoder 509. If the compressed base mesh bitstream contains intra-coded mesh frames, switch 503 can transmit the intra-coded mesh frames to static mesh decoder 505.

[0082] The static grid decoder 505 can decode intra-coded grid frames to generate reconstructed quantized base grid frames.

[0083] The grid buffer 507 can store the reconstructed quantized base grid frame and the inter-coded grid frame for future use in decoding subsequent inter-coded grid frames. The reconstructed quantized base grid frame can be used as a reference grid frame.

[0084] The motion decoder 509 can obtain the motion vector of the grid frame for the current inter-frame coding based on the data stored in the grid buffer 507 and the syntax elements of the grid frame for the current inter-frame coding in the bitstream. In some embodiments, the syntax elements of the grid frame for the current inter-frame coding in the bitstream can be the motion vector difference.

[0085] The base mesh reconstructor 511 can generate a reconstructed quantized base mesh frame based on the motion vector of the mesh frame used for the current inter-frame coding by using syntax elements of the mesh frame used for the current inter-frame coding in the bitstream.

[0086] If the compressed base grid bitstream has intra-frame encoded grid frames, switch 513 can send the reconstructed quantized base grid frames from static grid decoder 505 to dequantizer 515. If the compressed base grid bitstream has inter-frame encoded grid frames, switch 513 can send the reconstructed quantized base grid frames from static grid decoder 511 to dequantizer 515.

[0087] The dequantizer 515 can perform dequantization using the reconstructed quantized base grid frame m'(i) to generate the reconstructed base grid frame m''(i).

[0088] The video decoder 521 can decode the shifted bitstream to generate packed, quantized shifted wavelet coefficients dispQuantCoeffFrame.

[0089] Image unpacker 523 can unpack the packed quantized displacement wavelet coefficients dispQuantCoeffFrame to generate an array of quantized displacement wavelet coefficients dispQuantCoeffArray.

[0090] The dequantizer 525 can perform dequantization using an array of quantized shift wavelet coefficients, dispQuantCoeffArray, to generate shift wavelet coefficients.

[0091] The inverse wavelet transform 527 can perform an inverse wavelet transform using displacement wavelet coefficients to generate displacement.

[0092] The deformable mesh reconstructor 529 can reconstruct the deformable mesh M''(i) based on the displacement and the reconstructed base mesh frame m''(i).

[0093] Before color space conversion, video decoder 531 can decode the attribute bitstream to produce attribute map A'(i).

[0094] Color space converter 533 can perform color space conversion on attribute map A'(i) from video decoder 531 to reconstruct attribute map A''(i).

[0095] The following references are incorporated herein by reference as if fully set forth herein: i) ISO / IEC 23090-5 video-based point cloud compression; ii) ISO / IEC 23090-29 video-based dynamic mesh coding (V-DMC); and iii) ISO / IEC 23090-10 video-based delivery of compressed point cloud data.

[0096] Users can simplify quick previews or special effects playback by utilizing two-dimensional (2D) snapshots of volumetric content. These 2D snapshots are generated at a specific point in time using a camera positioned and oriented in a specific location. Users can decode the 2D snapshots instead of the volumetric content. Volumetric content can be decoded by decoding the compressed bitstream and then compositing or rendering it, but rendering the volumetric content is costly. Users can alternatively present decoded 2D snapshots.

[0097] Figure 6 A basic block diagram of a 2D Snapshot Supplemental Enhancement Information (SEI) message processor 600 according to an embodiment of the present disclosure is shown.

[0098] Reference Figure 6 According to an embodiment, the 2D snapshot SEI message processor 600 includes a 3D to 2D image processor 601, a packing, sorting and filling processor 603, one or more high-efficiency video coding (HEVC) encoders 605 and a multiplexer 607.

[0099] A 3D-to-2D image processor 601 can render a camera-generated photograph based on the 3D position and orientation vector of the viewport camera in a 3D frame to generate a 2D snapshot image and some metadata. A packing, sorting, and padding processor 603 can pack, sort, and pad the 2D snapshot image to generate geometry frames, color frames, occupancy map frames, and some metadata. One or more HEVC encoders 605 can encode the geometry frames, color frames, and occupancy map frames to generate geometry sub-bitstreams, color sub-bitstreams, and occupancy map sub-bitstreams. A multiplexer 607 can combine the geometry sub-bitstreams, color sub-bitstreams, occupancy map sub-bitstreams, and metadata to generate a compressed bitstream.

[0100] Volumetric frames can be viewed from multiple directions, so a single 2D snapshot image may not provide enough information about that volumetric frame. Multiple 2D snapshot images can be provided by adding multiple snapshot image tracks. However, this approach is quite inefficient in certain use cases, such as when the camera is fixed for the entire volumetric frame in the track. Modern video codecs (such as HEVC, Universal Video Coding (VVC), or Basic Video Coding (EVC)) provide mechanisms for including multiple subsets of images (such as tiles or sub-pictures). ISO / IEC 14496-15 allows each subset to be indicated as a separate subsample, and this feature can be used to include multiple snapshot images when the camera information is the same for all samples in the track.

[0101] Figure 7 An example of a V3C bitstream according to an embodiment of this disclosure is shown.

[0102] Reference Figure 7 The V3C bitstream 700 includes an encoded V3C preview image 701 and encoded V3C content 703. The V3C preview image 701 and V3C content 703 are encoded using widely available video and image coding standards. The V3C bitstream is formed by multiplexing the encoded V3C preview image 701 and the encoded V3C content 703. The V3C preview image 701 includes one or more samples of the V3C content, where the samples include subsamples of 2D snapshot images. The V3C content 703 includes a V3C atlas bitstream, a V3C attribute bitstream, a V3C geometry bitstream, and a V3C occupancy graph bitstream.

[0103] The V3C preview image 701 can be decoded to generate a 2D snapshot image. Each sample of the volumetric frame comprises multiple subsamples of the 2D snapshot image. Each subsample of the 2D snapshot image is rendered simultaneously using a camera at a different position and orientation.

[0104] In some embodiments, a multi-directional snapshot image track may include one or more samples of coded volumetric frames. In some embodiments, each of the one or more samples may be associated with a corresponding coded volumetric frame in the one or more coded volumetric frames. Each sample includes multiple subsamples. Each subsample includes a 2D projected image of the coded volumetric frame associated with the sample to which the subsample belongs.

[0105] In some embodiments, the set of positions and projection orientations of one subsample in a sample may differ from the set of positions and orientations of another subsample in the same sample. For example, the combination of positions and orientations of a camera used to render a 2D snapshot image within one subsample in a sample may differ from the combination of positions and orientations of a camera used to render a 2D snapshot image within another subsample in the same sample.

[0106] In some embodiments, the multi-directional snapshot image track includes a SubSampleInformationBox that provides subsample information. For example, the SubSampleInformationBox can provide information for one or more subsamples. Such information may include the projection orientation and position of the camera used to render the 2D snapshot image in the subsample.

[0107] In some embodiments, the set of camera positions and orientations used for projection may remain the same across one or more samples in a single track. For example, the set of positions and camera combinations used to render a 2D snapshot image of a subsample in any sample within a single track may be the same as the set of camera positions and orientation combinations used for a subsample in another sample within that single track.

[0108] In some embodiments, the synthesis time of the 2D snapshot image of the encoded volume frame is the same as the synthesis time of the encoded volume frame associated with the sample containing the 2D snapshot image.

[0109] In some embodiments, a 2D snapshot image track can be associated with a track including V3C data via a track reference. The V3C data can include V3C video components, which can be encoded as independent sub-pictures within a frame and then divided into tracks for each V3C video component. The track for each V3C video component can include the following sub-picture tracks: occupancy video component track, geometric video component track, and attribute video component track. Track references can be implemented using the track reference tool of ISO / IEC 14496-12. One or more TrackReferenceTypeBoxes can be added to the TrackReferenceBox within the TrackBox of a V3C atlas track or V3C atlas tile track. For convenience, a V3C atlas track or V3C atlas tile track can be referred to as a volumetric atlas track. In some embodiments, a volumetric atlas track can refer to an occupancy video component track, a geometric video component track, and an attribute video component track. For convenience, an occupancy video component track, a geometric video component track, and an attribute video component track can be referred to as a volumetric data track. A TrackReferenceTypeBox can be added to a TrackReferenceBox used for 2D snapshot image tracks. A TrackReferenceTypeBox can include an array of track_IDs. The track_ID can specify the track that includes the 2D snapshot image referenced by a V3C atlas track or a V3C atlas tile track. In some embodiments, the 4CC value of the reference_type of the TrackReferenceTypeBox may need to be "mdsi".

[0110] In some embodiments, the restrictions on 2D snapshot image tracks can be as follows: 2D snapshot image tracks referenced by V3C atlas tracks or V3C atlas tile tracks with reference_type "mdsi" can be represented as restricted videos in the file. 2D snapshot image tracks can use a generic sample entry "resv" with additional requirements. These additional requirements may include the presence of a variable SchemeTypeBox in the RestrictedSchemeInfoBox and scheme_type set to "mdst," all samples in the track potentially being synchronous samples, and the presence of a variable SubSampleInformationBox. The subsample_count value of SubSampleInformationBox is the same for all entries and is greater than 1.

[0111] In some embodiments, a SchemeInformationBox can be used, where the variable boxtype is set to "mdst", the variable mandatory is set to "Yes", and the variable Quantity is set to "One" when SchemeType is set to "mdst". The multi-dimensional snapshot camera information box can be used to indicate: camera information for rendering a volumetric frame snapshot of each subsample of a plurality of 2D snapshot images indicated by the SubSampleInformationBox. The i-th viewport information indicates the i-th subsample within the sample in bitstream order.

[0112] In some embodiments, the multidimensional fast camera information box can be defined according to the syntax shown in Table 1.

[0113] [Table 1]

[0114] In Table 1, the i-th viewport information indicates the i-th sub-sample within the sample in bitstream order. The i-th viewport provides information about the camera used for the i-th sub-sample in bitstream order.

[0115] The variable `num_viewport` indicates the number of viewports signaled in the sample. The value of this field is set to the value of `subsample_count` of the `SubSampleInformationBox`. For example, the number of viewports / cameras in a sample is the same as the number of subsamples in the sample.

[0116] The variable `camera_intrinsic_flag[i]` is a flag indicating whether camera intrinsic parameters exist in the i-th viewport. A value of 1 indicates that camera intrinsic parameters exist in the i-th viewport. A value of 0 indicates that camera intrinsic parameters do not exist in the i-th viewport. The number of flags in the variable `camera_intrinsic_flag[i]` of a sample is the same as the number of subsamples in the sample.

[0117] The variable `camera_extrinsic_flag[i]` is a flag indicating whether camera extrinsic parameters exist in the i-th viewport. A value of 1 indicates that camera extrinsic parameters exist in the i-th viewport. A value of 0 indicates that camera extrinsic parameters do not exist in the i-th viewport. The number of flags in the variable `camera_extrinsic_flag[i]` of a sample is the same as the number of subsamples in the sample.

[0118] The variable ViewportInfo provides information about the camera used for the i-th subsample in bitstream order. The variable ViewportInfo can be defined according to the syntax shown in Table 2.

[0119] [Table 2]

[0120] In Table 2, the variable `center_view_flag` is a flag indicating whether the signaled viewport position corresponds to the center of the viewport or one of the two stereo positions of the viewport. A value of 1 indicates that the signaled viewport position corresponds to the center of the viewport. A value of 0 indicates that the signaled viewport position corresponds to one of the two stereo positions of the viewport.

[0121] The variable `left_view_flag` indicates whether the signaled viewport information corresponds to the left or right stereo position of the viewport. A value of 1 indicates that the signaled viewport information corresponds to the left stereo position of the viewport. A value of 0 indicates that the signaled viewport information corresponds to the right stereo position of the viewport.

[0122] The variable extCamInfo is an instance of ExtCameraInfo that defines the camera's external parameters for the viewport.

[0123] The variable intCamInfo is an instance of IntCameraInfo that defines the camera's internal parameters for the viewport.

[0124] Figure 8 This is a flowchart illustrating the operation of an encoder according to another embodiment.

[0125] Reference Figure 8 Processing 800 begins with operation 801. In operation 801, the V-DMC encoder 400 obtains one or more samples from a plurality of samples corresponding to an encoded volumetric frame. Each of the one or more samples comprises multiple subsamples of a 2D snapshot image. Each subsample is associated with a corresponding position and orientation from a plurality of positions and orientations of a camera used to project the 2D snapshot image. The synthesis time of each subsample is the same as the synthesis time of the corresponding encoded volumetric frame.

[0126] In operation 803, the V-DMC encoder 400 encodes one or more samples corresponding to the encoded volume frame to generate one or more encoded samples.

[0127] In operation 805, the V-DMC encoder 400 determines camera information. This camera information includes multiple positions and orientations for each camera used to project a 2D snapshot image of a corresponding subsample from a plurality of subsamples. The camera information includes the camera position at coordinates x, y, and z, and camera orientation components x, y, and z. The camera orientation components determine the orientation vector as the direction the camera is facing.

[0128] In operation 807, the V-DMC encoder 400 generates an ISO Basic Media File Format (ISOBMFF) file based on one or more encoded samples and camera information. The ISOBMFF file includes multi-directional snapshot image tracks and viewport information timing metadata tracks. In some embodiments, the ISOBMFF file may also include atlas tracks containing an array of track identifiers specifying the track containing the 2D snapshot image referenced by the atlas track. In some embodiments, the atlas track may be a V3C atlas track or a V3C atlas tile track. In the case of a multi-track container, a V3C atlas track may represent a volumetric visual track containing a V3C atlas bitstream. In the case of a multi-track container, a V3C atlas tile track may represent a volumetric visual track containing a portion of the V3C atlas bitstream corresponding to one or more tiles.

[0129] In operation 809, the V-DMC encoder 400 sends an ISOBMFF file.

[0130] Figure 9 This is a flowchart illustrating the operation of a decoder according to another embodiment.

[0131] Reference Figure 9 Processing 900 begins with operation 901. In operation 901, the V-DMC decoder 500 receives an ISOBMFF file that includes a multi-directional snapshot image track and a viewport information timing metadata track.

[0132] In some embodiments, the multi-directional snapshot image track includes one or more samples corresponding to encoded volumetric frames. Each sample includes multiple subsamples of the 2D snapshot image. In some embodiments, the viewport information timing metadata track may contain camera information. The camera information includes position and orientation information associated with each subsample of the 2D snapshot image.

[0133] In some embodiments, an ISOBMFF file may further include atlas tracks containing an array of track identifiers that specify the track containing the 2D snapshot image referenced by the atlas track. In some embodiments, the atlas track may be a V3C atlas track or a V3C atlas tile track. In the case of a multi-track container, a V3C atlas track may represent a volumetric visual track containing a V3C atlas bitstream. In the case of a multi-track container, a V3C atlas tile track may represent a volumetric visual track containing a portion of a V3C atlas bitstream corresponding to one or more tiles.

[0134] In operation 903, the V-DMC decoder 500 decodes the ISOBMFF file to generate a multi-directional snapshot image track and a viewport information timing metadata track. The multi-directional snapshot image track includes one or more samples corresponding to encoded volumetric frames, each sample including multiple sub-samples of the 2D snapshot image. The viewport information timing metadata track includes camera information.

[0135] In operation 905, the V-DMC decoder 500 renders at least one of the 2D snapshot images based on at least one subsample. In some embodiments, at least one of the 2D snapshot images may be rendered based on associated camera information.

[0136] In one embodiment, an improved fast preview or special effects playback operation for volumetric content is provided by implementing a multi-directional snapshot image track.

[0137] In this embodiment, a series of two-dimensional (2D) snapshot images of the volumetric content are introduced. This series of 2D snapshot images of the volumetric content eliminates the need for decoding and rendering the volumetric content, thereby reducing computational costs. The 2D snapshot images for the volumetric content can be carried in a multi-directional snapshot image track.

[0138] According to an embodiment, an apparatus is provided, comprising a communication interface and a processor operatively coupled to the communication interface. The communication interface is configured to receive a track. The track includes one or more samples. A corresponding sample among the one or more samples includes multiple encoded two-dimensional projection images of an encoded volumetric frame as multiple sub-samples. The processor is configured to decode at least one of the multiple encoded two-dimensional projection images to generate at least one two-dimensional projection image. The processor is further configured to render the at least one two-dimensional projection image.

[0139] In this embodiment, the communication interface is also configured to receive a multi-dimensional fast camera information box. The multi-dimensional fast camera information box includes a plurality of viewport information elements. Each of the plurality of viewport information elements is associated with a corresponding sub-sample among the plurality of sub-samples. The corresponding viewport information element among the plurality of viewport information elements provides camera information for the associated sub-sample.

[0140] In an embodiment, the multidimensional fast camera information box includes quantity information indicating the number of subsamples in the corresponding one of the one or more samples.

[0141] In an embodiment, the multi-dimensional fast camera information box includes a plurality of camera exterior markers. Each of the plurality of camera exterior markers is associated with a corresponding viewport information element among the plurality of viewport information elements and indicates whether camera exterior information exists in the associated viewport information element. If a corresponding camera exterior marker among the plurality of camera exterior markers indicates that camera exterior information exists in the corresponding viewport information element among the plurality of viewport information elements associated with that corresponding camera exterior marker, then the multi-dimensional fast camera information box also includes the camera exterior information associated with that corresponding camera exterior marker. The number of the plurality of camera exterior markers is the same as the number of the plurality of subsamples.

[0142] In an embodiment, the multi-dimensional fast camera information box includes a plurality of camera interior markers. Each of the plurality of camera interior markers is associated with a corresponding viewport information element among the plurality of viewport information elements, and indicates whether camera interior information exists in the associated viewport information element. If a corresponding camera interior marker among the plurality of camera interior markers indicates that camera interior information exists in the corresponding viewport information element among the plurality of viewport information elements associated with that corresponding camera interior marker, then the multi-dimensional fast camera information box also includes the camera interior information associated with that corresponding camera interior marker. The number of the plurality of camera interior markers is the same as the number of the plurality of subsamples.

[0143] In one embodiment, the at least one two-dimensional projected image is presented based on the multi-dimensional fast camera information box.

[0144] In an embodiment, the corresponding viewport information element among the plurality of viewport information elements includes camera position information and orientation information for rendering two-dimensional projected images in associated subsamples among the plurality of subsamples.

[0145] In this embodiment, the position and orientation information of the camera used for one of the plurality of subsamples are different from the position and orientation information of the camera used for another of the plurality of subsamples.

[0146] In this embodiment, the set of position and orientation information of the cameras for the plurality of subsamples remains the same in a single track.

[0147] According to an embodiment, an apparatus is provided, comprising a communication interface and a processor operatively coupled to the communication interface. The processor is configured to encode a plurality of two-dimensional projection images of one or more volumetric frames to generate a plurality of encoded two-dimensional projection images. The processor is further configured to generate a track comprising one or more samples. Each of the one or more samples is associated with a corresponding volumetric frame among the one or more volumetric frames and includes at least two encoded two-dimensional projection images associated with the volumetric frame as a plurality of subsamples. The processor is further configured to transmit the track.

[0148] In one embodiment, the processor is further configured to generate a multi-dimensional fast camera information box. The multi-dimensional fast camera information box includes a plurality of viewport information elements. Each of the plurality of viewport information elements is associated with a corresponding subsample among the plurality of subsamples. The corresponding viewport information element provides camera information for the associated subsample. The processor is also configured to transmit the multi-dimensional fast camera information box.

[0149] In an embodiment, the multidimensional fast camera information box includes quantity information indicating the number of subsamples in the corresponding one of the one or more samples.

[0150] In an embodiment, the multi-dimensional fast camera information box includes a plurality of camera exterior markers. Each of the plurality of camera exterior markers is associated with a corresponding viewport information element among the plurality of viewport information elements and indicates whether camera exterior information exists in the associated viewport information element. If a corresponding camera exterior marker among the plurality of camera exterior markers indicates that camera exterior information exists in the corresponding viewport information element among the plurality of viewport information elements associated with that corresponding camera exterior marker, then the multi-dimensional fast camera information box also includes the camera exterior information associated with that corresponding camera exterior marker. The number of the plurality of camera exterior markers is the same as the number of subsamples.

[0151] In an embodiment, the multi-dimensional fast camera information box includes a plurality of camera interior markers. Each of the plurality of camera interior markers is associated with a corresponding viewport information element among the plurality of viewport information elements and indicates whether camera interior information exists in the associated viewport information element. If a corresponding camera interior marker among the plurality of camera interior markers indicates that camera interior information exists in the corresponding viewport information element among the plurality of viewport information elements associated with that corresponding camera interior marker, then the multi-dimensional fast camera information box also includes the camera interior information associated with that corresponding camera interior marker. The number of the plurality of camera interior markers is the same as the number of subsamples.

[0152] In an embodiment, the corresponding viewport information element among the plurality of viewport information elements includes camera position information and orientation information for rendering two-dimensional projected images in associated subsamples among the plurality of subsamples.

[0153] In this embodiment, the position and orientation information of the camera used for one of the plurality of subsamples are different from the position and orientation information of the camera used for another of the plurality of subsamples.

[0154] In this embodiment, the set of position and orientation information of the cameras for the plurality of subsamples remains the same in a single track.

[0155] According to an embodiment, a method performed by a device is provided. The method includes receiving a track. The track includes one or more samples. A corresponding sample among the one or more samples includes multiple coded two-dimensional projection images of coded volumetric frames as multiple sub-samples. The method further includes decoding at least one of the multiple coded two-dimensional projection images to generate at least one two-dimensional projection image. The method further includes presenting the at least one two-dimensional projection image.

[0156] In one embodiment, the method further includes receiving a multi-dimensional fast camera information box. The multi-dimensional fast camera information box includes a plurality of viewport information elements. Each of the plurality of viewport information elements is associated with a corresponding subsample among the plurality of subsamples. The corresponding viewport information element among the plurality of viewport information elements provides camera information for the associated subsample.

[0157] In an embodiment, the multidimensional fast camera information box includes quantity information indicating the number of subsamples in the corresponding one of the one or more samples.

[0158] Various descriptive blocks, units, modules, components, methods, operations, instructions, items, and algorithms can be implemented or executed using processing circuitry.

[0159] Unless otherwise specified, references to elements in the singular form are not intended to indicate one and only one, but rather one or more. For example, a “one” module may refer to one or more modules. Without further constraints, elements beginning with “a,” “an,” “the,” or “the” do not exclude the existence of other identical elements.

[0160] Titles and subtitles (if any) are used for convenience only and do not limit the technical scope of the subject matter. The term "exemplary" is used to indicate that it is used as an example or illustration. With regard to the use of the terms "comprising," "having," "carrying," "including," etc., such terms are interpreted in a similar manner to the term "comprising" when used as a transition word in the claims, and are intended to be inclusive. Relational terms such as "first" and "second" can be used to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between these entities or actions.

[0161] Phrases such as aspect, that aspect, on the other hand, some aspects, one or more aspects, implementation, that implementation, another implementation, some implementations, one or more implementations, embodiment, that embodiment, another embodiment, some embodiments, one or more embodiments, configuration, that configuration, another configuration, some configurations, one or more configurations, subject matter, disclosure, this disclosure, other variations thereof, etc., are for convenience and do not imply that disclosures associated with such phrases are essential to the subject matter, or that such disclosures apply to all configurations of the subject matter. Disclosures associated with one or more such phrases may apply to all configurations or one or more configurations. Disclosures associated with one or more such phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects, and vice versa, and this similarly applies to other foregoing phrases.

[0162] The phrase "at least one" following a list of items modifies the entire list, not each member of the list, when separated by the terms "and" or "or". The phrase "at least one of..." does not require selection of at least one item; rather, it allows for the inclusion of at least one of any one item, and / or at least one of any combination of items, and / or at least one of each item. For example, each of the phrases "at least one of A, B, and C" or "at least one of A, B, or C" refers to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0163] It should be understood that the specific order or hierarchy of the disclosed steps, operations, or processes is an illustrative example. Unless otherwise expressly stated, it should be understood that the specific order or hierarchy of steps, operations, or processes may be performed in a different order. Some steps, operations, or processes may be performed simultaneously, or may be performed as part of one or more other steps, operations, or processes. The appended method claims (if any) present elements of various steps, operations, or processes in an exemplary order, but this does not imply limitation to the specific order or hierarchy presented. These may be performed serially, linearly, in parallel, or in a different order. It should be understood that the described instructions, operations, and systems may generally be integrated together in a single software / hardware product or packaged into multiple software / hardware products.

[0164] This disclosure is provided to enable any person skilled in the art to practice the various aspects described herein. In some cases, well-known structures and components are shown in block diagram form to avoid obscuring the concept of the subject matter. This disclosure provides various examples of the subject matter, and the subject matter is not limited to these examples. Various modifications to these aspects will be apparent to those skilled in the art, and the principles described herein can be applied to other aspects.

[0165] All structural and functional equivalents of elements throughout the various aspects described in this disclosure are expressly incorporated herein by reference and are intended to be covered by the claims, and are known or would be known to one of ordinary skill in the art. Furthermore, nothing disclosed herein is intended to be made public, whether or not such disclosure is explicitly stated in the claims.

[0166] The title, background information, description of the drawings, abstract, and figures are incorporated herein by reference and are provided as illustrative examples rather than as limiting descriptions. It should be understood that, at the time of filing, they are not intended to limit the scope or meaning of the claims. Furthermore, in the detailed description, illustrative examples may be provided, and various features may be combined in various embodiments for the purpose of simplifying this disclosure. This approach to disclosure should not be construed as reflecting an intention to require more features than expressly recited in each claim. Rather, as reflected in the appended claims, the inventive subject matter lies in all features of fewer than those in a single disclosed configuration or operation. The appended claims are incorporated herein by reference, wherein each claim is independently claimed as a separate subject matter.

[0167] The embodiments are provided merely as examples for understanding the invention. They are not intended and should not be construed as limiting the scope of the invention in any way. Although certain embodiments and examples have been provided, those skilled in the art will understand from the disclosure herein that changes can be made to the illustrated embodiments and examples without departing from the scope of the invention.

[0168] The claims are not intended to be limited to the aspects described herein, but rather to conform to the full scope consistent with the language claims and to include all legal equivalents. Nevertheless, no claim is intended to include subject matter that does not meet the requirements of applicable patent law, nor should they be interpreted in this manner.

Claims

1. An apparatus (500), comprising: A communication interface is configured to acquire a track comprising one or more samples, wherein a corresponding sample among the one or more samples comprises multiple encoded two-dimensional projection images of an encoded volumetric frame as multiple sub-samples; and A processor, operatively coupled to the communication interface, is configured to: Decode at least one of the plurality of encoded two-dimensional projection images to generate at least one two-dimensional projection image, and Present the at least one two-dimensional projected image.

2. The device (500) according to claim 1, wherein, The communication interface is also configured to acquire a multi-dimensional camera information box, wherein the multi-dimensional camera information box includes multiple viewport information elements. Each of the plurality of viewport information elements is associated with a corresponding subsample among the plurality of subsamples, and One of the plurality of viewport information elements provides camera information for the associated subsample.

3. The device (500) according to any one of claims 1 to 2, wherein, The multidimensional fast camera information box includes quantity information indicating the number of subsamples in the corresponding one of the one or more samples.

4. The device (500) according to any one of claims 1 to 3, wherein, The multi-dimensional fast camera information box includes multiple camera exterior markers, wherein each of the multiple camera exterior markers is associated with a corresponding viewport information element among the multiple viewport information elements, and indicates whether camera exterior information exists in the associated viewport information element. If a corresponding camera exterior marker among the plurality of camera exterior markers indicates that camera exterior information exists in the corresponding viewport information element associated with that corresponding camera exterior marker among the plurality of viewport information elements, then the multi-dimensional fast camera information box also includes the camera exterior information associated with that corresponding camera exterior marker, and The number of the multiple camera external markers is the same as the number of the multiple subsamples.

5. The device (500) according to any one of claims 1 to 4, wherein, The multi-dimensional fast camera information box includes multiple camera internal markers, wherein each of the multiple camera internal markers is associated with a corresponding viewport information element among the multiple viewport information elements, and indicates whether camera internal information exists in the associated viewport information element. If a corresponding camera interior marker among the plurality of camera interior markers indicates that camera interior information exists in the corresponding viewport information element associated with that corresponding camera interior marker among the plurality of viewport information elements, then the multi-dimensional fast camera information box also includes the camera interior information associated with that corresponding camera interior marker, and The number of internal markers of the multiple cameras is the same as the number of the multiple subsamples.

6. The device (500) according to any one of claims 1 to 5, wherein, The at least one two-dimensional projected image is presented based on the multi-dimensional fast camera information box.

7. The device (500) according to any one of claims 1 to 6, wherein, The corresponding viewport information element among the plurality of viewport information elements includes camera position and orientation information for rendering two-dimensional projected images in associated subsamples among the plurality of subsamples.

8. The device (500) according to any one of claims 1 to 7, wherein, The position and orientation information of the camera used for one of the plurality of subsamples are different from the position and orientation information of the camera used for another of the plurality of subsamples.

9. The device (500) according to any one of claims 1 to 8, wherein, The set of position and orientation information for the cameras used for the multiple subsamples remains the same in a single orbit.

10. An apparatus (400), comprising: Communication interface; as well as A processor, operatively coupled to the communication interface; The processor is configured to: Encoding multiple 2D projected images of one or more volumetric frames to generate multiple encoded 2D projected images; and Generate an orbit comprising one or more samples, wherein each of the one or more samples is associated with a corresponding volume frame among the one or more volume frames, and includes at least two coded two-dimensional projection images associated with the volume frame as multiple sub-samples.

11. The device (400) according to claim 10, wherein, The processor is also configured to: A multi-dimensional fast camera information box is generated, wherein the multi-dimensional fast camera information box includes multiple viewport information elements, each of the multiple viewport information elements is associated with a corresponding subsample among the multiple subsamples, and the corresponding viewport information element among the multiple viewport information elements provides camera information for the associated subsample.

12. The device (400) according to any one of claims 10 to 11, wherein, The multidimensional fast camera information box includes quantity information indicating the number of subsamples in the corresponding one of the one or more samples.

13. The device (400) according to any one of claims 10 to 12, wherein, The multi-dimensional fast camera information box includes multiple camera exterior markers, wherein each of the multiple camera exterior markers is associated with a corresponding viewport information element among the multiple viewport information elements, and indicates whether camera exterior information exists in the associated viewport information element. If a corresponding camera exterior marker among the plurality of camera exterior markers indicates that camera exterior information exists in the corresponding viewport information element associated with that corresponding camera exterior marker among the plurality of viewport information elements, then the multi-dimensional fast camera information box also includes the camera exterior information associated with that corresponding camera exterior marker, and The number of the multiple camera external markers is the same as the number of the multiple subsamples.

14. A method for transmitting a track generated by the device (400) according to any one of claims 10 to 13.

15. A method performed by a device (500), comprising: Obtain (901) an orbit comprising one or more samples, wherein a corresponding sample among the one or more samples comprises multiple encoded two-dimensional projection images of encoded volumetric frames as multiple subsamples; Decoding (903) at least one of the plurality of encoded two-dimensional projection images to generate at least one two-dimensional projection image; and Present at least one two-dimensional projection image as described in (905).

Images