Multi-atlas encapsulation of immersive media

Abstract

[0003] Methods, apparatus, and systems are disclosed that provide flexible encapsulation of volumetric video data within a media file. In one exemplary aspect, a method for video processing includes receiving three-dimensional (3D) volumetric video data and encoding the three-dimensional volumetric video data into a media file. The 3D volumetric video data corresponds to one or more atlases, each comprising atlas data and one or more two-dimensional (2D) components. The atlas data and the one or more 2D components are stored in one or more media tracks within the media file.

Description

[Technical field]

[0001] This patent document is generally directed to video processing. [Background technology]

[0002] Users prefer to experience content in a way that makes it seem as if it is real. As technology advances, immersive media brings content that is more realistic than ever before. Immersive media includes 360-degree images and videos, virtual reality (VR), augmented reality (AR), mixed reality (MR), and non-traditional formats driven by emerging technology platforms such as wearables and the Internet of Things (IoT). Summary of the Invention [Means for solving the problem]

[0003] This patent document describes, among other things, techniques that can be implemented to provide flexible encapsulation of volumetric video data within media files.

[0004] In one example aspect, a method for video processing includes receiving three-dimensional (3D) volumetric video data and encoding the three-dimensional volumetric video data into a media file. The 3D volumetric video data corresponds to one or more atlases, each comprising atlas data and one or more two-dimensional (2D) components. The atlas data and the one or more 2D components are stored in one or more media tracks in the media file.

[0005] In another exemplary aspect, a method for video processing includes decoding a media file representing three-dimensional (3D) volumetric video data and reconstructing the 3D volumetric video data based on at least one of the one or more media tracks. The 3D volumetric video data corresponds to one or more atlases, each comprising atlas data and one or more two-dimensional (2D) components. The atlas data and the one or more 2D components are stored in one or more media tracks in the media file.

[0006] In another exemplary aspect, a communications apparatus is disclosed that includes a processor configured to implement the method described above.

[0007] In yet another exemplary aspect, a computer program storage medium is disclosed that includes code stored thereon that, when executed by a processor, causes the processor to implement the methods described.

[0008] In yet another exemplary aspect, a computer program storage medium is disclosed that stores media files generated according to the methods described herein.

[0009] These and other aspects are described herein. The present invention provides, for example, the following: (Item 1) 1. A method for video processing, comprising: Receiving three-dimensional (3D) volumetric video data; encoding said 3D volumetric video data into a media file; Including, The method, wherein the 3D volumetric video data corresponds to one or more atlases, each atlas comprising atlas data and one or more two-dimensional (2D) components, and wherein the atlas data and the one or more 2D components are stored in one or more media tracks within the media file. (Item 2) 1. A method for video processing, comprising: Decoding a media file representing three-dimensional (3D) volumetric video data, the 3D volumetric video data corresponding to one or more atlases, each atlas comprising atlas data and one or more two-dimensional (2D) components, the atlas data and the one or more 2D components being stored in one or more media tracks within the media file; reconstructing the 3D volumetric video data based on at least one of the one or more media tracks; A method comprising: (Item 3) 3. The method of claim 1 or 2, wherein the one or more 2D components for each atlas comprise at least a geometry component, an occupancy component, or an attribute component. (Item 4) 4. The method of any of items 1-3, wherein the atlas data comprises a projective relationship that projects the 3D volumetric video data from a 3D space onto a 2D plane. (Item 5) 5. The method of any of items 1-4, wherein at least one media track in the media file contains information representing some or all of the 3D volumetric video data. (Item 6) 6. The method of any of items 1-5, wherein the one or more media tracks include a volumetric visual track with specific sample entry types, the volumetric visual track comprising parameter information common to all of the one or more atlases. (Item 7) 7. The method of claim 6, wherein the volumetric visual track includes zero samples. (Item 8) 7. The method of claim 6, wherein the volumetric visual track includes one or more samples, each sample carrying atlas data relating to one or more atlases. (Item 9) 9. The method of any of items 1-8, wherein the one or more media tracks include an atlas track with specific sample entry types, the atlas track comprising atlas data for one or more atlases. (Item 10) 10. The method of claim 1, wherein the one or more media tracks include a first component track with a first sample entry type, the first component track comprising data for the one or more components corresponding to the one or more atlases. (Item 11) 11. A method according to any of items 1-10, wherein the one or more media tracks include a second component track with a second sample entry type, the second component track comprising the atlas data and data for the one or more components corresponding to the one or more atlases. (Item 12) 12. The method of any of items 1-11, wherein the one or more media tracks include a third component track with a third sample entry type, the third component track carrying a particular type of component data from the one or more atlases. (Item 13) 13. The method of any of items 1-12, wherein the one or more media tracks are organized into a plurality of groups. (Item 14) 14. The method of claim 13, wherein each group corresponds to one of the one or more atlases. (Item 15) Item 14. The method of item 13, wherein each group corresponds to a plurality of atlases that are arranged together due to a grouping criterion associated with the 3D volumetric video data. (Item 16) A video processing device, the video processing device comprising a processor configured to implement the method described in any one or more of items 1-15. (Item 17) A computer program product having code stored thereon which, when executed by a processor, causes the processor to implement a method described in any one or more of items 1-15. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an exemplary structure of a media file.

[0011] [Diagram 2]FIG. 2 illustrates another exemplary structure of a media file.

[0012] [Diagram 3] FIG. 3 illustrates an exemplary encoding and decoding process for volumetric video data in accordance with the present technique.

[0013] [Figure 4] FIG. 4 illustrates an example device that may be used to implement an encoder-side or decoder-side module in accordance with the present techniques.

[0014] [Diagram 5] FIG. 5 is a flowchart representation of a method for video encoding in accordance with the present technique.

[0015] [Figure 6] FIG. 6 is a flowchart representation of another method for video encoding in accordance with the present technique.

[0016] [Figure 7] FIG. 7 illustrates an exemplary encapsulation structure of a media file in accordance with the present technology.

[0017] [Figure 8] FIG. 8 illustrates an exemplary encapsulation structure of a media file in accordance with the present technology.

[0018] [Figure 9] FIG. 9 illustrates an exemplary encapsulation structure of a media file in accordance with the present technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Detailed Description Section headings are used herein to improve readability only and do not limit the scope of the disclosed embodiments and techniques within each section to only that section. The disclosed techniques are applicable to a variety of imaging and / or video standards, including, but not limited to, International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) based media file format standards.

[0020] Immersive media refers to the use of multimedia technologies such as audio and video to give users an immersive and completely different media experience. When watching immersive media content, users can participate in the media in real time (e.g., by wearing a virtual reality headset). Immersive media has been successfully extended to many fields such as movies, games, news, and / or medical treatment. Immersive media content is different from traditional media content, i.e., the visual content is typically 360-degree panoramic visual data, which covers a large image area and a wide field of view and provides users with more flexibility in adjusting the content based on their own preferences.

[0021] Currently, immersive media systems use volumetric video to represent three-dimensional (3D) scenes and objects. The 3D video data is represented by voxels, so any object in the scene can be viewed through six degrees of freedom of movement. Volumetric video can be constructed using data collected simultaneously from multiple cameras, or using a three-dimensional (3D) representation of the data (e.g., point cloud data). The 3D video data is then projected onto a two-dimensional (2D) space as 2D patches (also referred to as 2D components). For example, the volumetric video data can be represented as a rectangular image after mapping (e.g., an orthogonal projection image). Alternatively, the volumetric video data can be represented as a rectangular image after mapping and patch packing. In some embodiments, the 3D volumetric video data can be stored in a media file. The characteristics of the 2D data are encapsulated in the media file as 2D components, such as geometry, attributes, and / or occupancy map data. In addition to the 2D components, the projective relationship between the 3D and 2D spaces, referred to as atlas data, is also encapsulated within the media file.

[0022] The following discussion focuses on the ISO Basic media file format, however, the relevant considerations and techniques are applicable to other media file formats, including the High Efficiency Video Coding (HEVC) format, the Versatile Video Coding (VVC) format, etc.

[0023] The ISO base media file format, such as the restricted scheme information box, the track reference box, and the track group box, is formulated based on the MPEG (Moving Picture Experts Group) Part 12 ISO base media file format, called MPEG-4. The volumetric video projection, packaging steps, and the base format can be operated with reference to MPEG-I Part 10, formulated by MPEG. In the ISO base file format, all data is packed in a box. That is, the ISO base file format represented by the MP4 file includes several boxes, each having a certain type and length. Some boxes are called container boxes because they may contain one or more other boxes. The MP4 media file includes an "ftyp" type box, which serves as a file format mark and contains certain information about the file. The "ftyp" type box is followed by only one movie box ("MOOV" type box). The movie box is a container box that includes additional boxes with metadata information of the media. A media file can contain zero or more Media Data boxes ("MADT" type boxes). For example, when all media data reference other files, a media file contains zero Media Data boxes. A Media Data box is also a container box that contains additional boxes with media data. The structure of the media data is determined by the metadata description. To further support the metadata description of the media, a metadata box ("Meta" type box) can optionally be used to describe some general or additional non-timed metadata.

[0024] As discussed above, the 2D representation of the volumetric video projected onto the 2D plane is represented by 2D component data, including geometric component data, attribute component data, and / or occupancy component data. The atlas data represents the projective relationship between the 3D space and the 2D plane. There can be at least two types of tracks in a multi-track encapsulated video data container: a volumetric visual track (also referred to as a V3C track) that carries atlas information, such as atlas parameter information or atlas bitstream, and a video track (also referred to as a V3C component track) that carries 2D video encoding data related to any of the occupancy, geometry, or attribute component video bitstreams. FIG. 1 illustrates an exemplary structure of a media file. According to the ISO / IEC coding standard, for example, a container file can include a single track encapsulation of volumetric data (e.g., track-1). That is, the corresponding bitstream is represented by a single track declaration in the media file. In some embodiments, multiple volumetric visual tracks can be present in the file as well. FIG. 2 illustrates another exemplary structure of a media file. In the embodiment shown in FIG. 2, data is mapped to individual tracks within a container file based on component type.

[0025] A single track encapsulation of V3C data as shown in Figure 1 can be used for simple encapsulation of V3C encoded bitstreams. However, single track encapsulation offers very little flexibility for complex 3D scenes. Because everything is encapsulated in a single track, all data needs to be transmitted for decoding purposes, leading to a potential waste of bandwidth and processing power.

[0026] Multi-track encapsulation provides more flexibility for complex scenes and allows different types of component data to be separated into different tracks. As shown in FIG. 2, each video track carries a single type of component video bitstream. However, decoding video data often requires multiple components of video data (e.g., geometry and texture). Furthermore, decoding several video bitstreams in parallel can lead to synchronization and memory utilization issues. For example, synchronization of multiple video bitstreams is difficult when the decoder skips frames. Implementation complexity increases significantly for multi-stream synchronization and can lead to unnecessary up-front delays in decoding time. Furthermore, 3D to 2D projection can result in multiple atlases. For example, when a volumetric video is constructed using data from multiple cameras, one atlas can be built for data from one or more selected cameras. In some embodiments, images captured from multiple cameras can be partially overlapping or repeated. Extraction can be performed on the overlapped portions before the captured images are encoded into an atlas. The multi-track encapsulation structure shown in FIG. 2 cannot provide flexibility in data transmission and / or decoding when only a subset of the atlas is needed.

[0027] This patent document discloses techniques that may be implemented in various embodiments to provide a more flexible encapsulation scheme to reduce decoding complexity and increase bandwidth consumption. In some embodiments, the disclosed techniques may be implemented to enable encapsulation of multiple atlases in a media box, thereby allowing video processing to be performed on an atlas basis. Component tracks may be organized based on corresponding atlases and / or component types, thereby reducing or minimizing the need for decode-side synchronization.

[0028] FIG. 3 illustrates an exemplary encoding and decoding process for volumetric video data according to the present technology. As shown in FIG. 3, the immersive media system 300 includes an immersive media production platform 310 (encoder side) and a player terminal 320 (decoder side). The immersive media production platform 310 includes at least a content production module 311, a storage module 312, and a transmission module 313. The content production module 311 is used for encoding and encapsulation processing of captured volumetric video. The storage module 312 is used to store encapsulated media files. The transmission module 313 is used to receive request messages from client terminals and / or transmit data (such as media files) stored in the storage module 312. The reception and / or transmission of video data can be implemented through a wireless network provided by a communication provider, a local wireless area network, or a wired method.

[0029] The player terminal 320 includes at least a transmission module 321, a media processing module 322, and a media presentation module 323. The transmission module 321 is used to receive data (e.g., media files) and / or transmit messages (e.g., media file requests) to the transmission module 313. The media processing module 322 is used to decode and decapsulate the received media files and reconstruct the volumetric video according to the user's current viewing state, such as the viewing position, the viewing direction, etc. The media processing module 322 also performs rendering processing. The media presentation module 323 is used to present the reconstructed volumetric video in the user's viewport.

[0030] FIG. 4 shows an exemplary device 400 that may be used to implement an encoder-side or decoder-side module according to the present patent technology. The device 400 includes a processor 402 that may be configured to perform encoder-side or decoder-side techniques or both. The device 400 may also include memory (404) for storing processor-executable instructions and for storing video bitstreams and / or display data. The device 400 may include video processing circuitry (not shown), such as conversion circuitry, arithmetic coding / decoding circuitry, etc. The video processing circuitry may be included partially within the processor and / or partially within other dedicated circuitry, such as a graphics processor, field programmable gate array (FPGA), etc.

[0031] 5 is a flowchart representation of a method 500 for video encoding in accordance with the present technology. The method 500 includes receiving three-dimensional (3D) volumetric video data at operation 510. The method 500 also includes encoding the three-dimensional volumetric video data into a media file at operation 520. The 3D volumetric video data corresponds to one or more atlases, each comprising atlas data and one or more two-dimensional (2D) components. The atlas data and the one or more 2D components are stored in one or more media tracks in the media file.

[0032] 6 is a flowchart representation of a method 600 for video decoding according to the present technology. The method 600 includes, at operation 610, decoding a media file representing three-dimensional (3D) volumetric video data. The 3D volumetric video data corresponds to one or more atlases, each comprising atlas data and one or more two-dimensional (2D) components. The atlas data and the one or more 2D components are stored in one or more media tracks in the media file. The method 600 also includes, at operation 620, reconstructing the 3D volumetric video data based on at least one media track of the one or more media tracks.

[0033] In both encoding and decoding cases, one or more 2D components of each atlas comprise at least a geometry component, an occupancy component, or an attribute component. In some embodiments, the atlas data comprises a projection relationship that projects the 3D volumetric video data from the 3D space onto a 2D plane. In some embodiments, at least one media track in a media file includes information that represents some or all of the 3D volumetric video data. For example, a subset of tracks of all tracks stored in a media file are transmitted for decoding to reduce transmission costs.

[0034] In some embodiments, the one or more media tracks include a volumetric visual track with a specific sample entry type (e.g., V3C). The volumetric visual track comprises parameter information common to all of the one or more atlases. In some embodiments, the volumetric visual track includes zero samples. In some embodiments, the volumetric visual track includes one or more samples, each sample carrying atlas data for one or more atlases.

[0035] In some embodiments, one or more media tracks include one or more atlas tracks with a specific sample entry type (e.g., "v3ca" as described in embodiment 3 below). The atlas tracks comprise atlas data for one or more atlases.

[0036] In some embodiments, the one or more media tracks include one or more first component tracks with a first sample entry type (e.g., “v3cc” as described in embodiment 1 below). The first component track comprises one or more component data corresponding to one or more atlases. In some embodiments, the one or more media tracks include one or more second component tracks with a second sample entry type (e.g., “v3ac” as described in embodiment 2 below). The second component track comprises atlas data and one or more component data corresponding to one or more atlases. In some embodiments, the one or more media tracks include one or more third component tracks with a third sample entry type (e.g., “v3cm” as described in embodiment 3 below). The third component track carries a particular type of component data from one or more atlases.

[0037] In some embodiments, the one or more media tracks are organized into multiple groups. In some embodiments, each group corresponds to one of the one or more atlases. In some embodiments, each group corresponds to multiple atlases that are arranged together due to a grouping criterion associated with the 3D volumetric video data, such as the location of the view within the 3D scene.

[0038] In some embodiments, a media file encoded according to one of the encoding methods described above may be stored on a computer-readable medium.

[0039] Some examples of the disclosed techniques are further described in the following exemplary embodiments.

[0040] EMBODIMENT 1

[0041] In some embodiments, the atlas data for all atlases is stored in the same track (e.g., a V3C track). Figure 7 illustrates an example encapsulation structure 700 of a media file in accordance with the present technology. As shown in Figure 7, a volumetric video media file (e.g., an ISO file) contains a V3C volumetric visual track, identified by a sample entry with type "v3am" in a MOOV box. The V3C volumetric visual track contains parameter information for the volumetric video and / or atlas information that is common to all atlases.

[0042] A VC3 volumetric visual track can contain one or more corresponding samples. When at least one sample is present, the corresponding MDAT box stores volumetric video parameter information, atlas parameter information, and / or all atlas data.

[0043] A V3C component track can be represented as a video with limitations in a media file and identified by "vvvc" in the scheme_type field of a SchemeTypeBox corresponding to a video sample entry. In some embodiments, different types of component data corresponding to an atlas are stored in one V3C component track. As shown in FIG. 7, Atlas_1 (701) has multiple components of different types, including component 1, component 2, etc. All components corresponding to Atlas_1 are stored in the same V3C component track. That is, characteristics such as geometry, attributes, occupancy data, etc. for Atlas_1 are organized together to reduce synchronization complexity for processing Atlas_1.

[0044] In some embodiments, an atlas can support multiple tracks, organized based on component type. As shown in FIG. 7, Atlas_2 (702) supports two tracks. Each track supports a particular component type. For example, Track_2 supports component type 1, while Track_3 supports component type 2.

[0045] An example syntax for a V3C volumetric visual track sample entry is shown in the table below. [Table 1]

[0046] The variables in the above table have the following definitions:

[0047] The multi_atlas_flag indicates whether the media track contains multiple atlases.

[0048] num_atlas indicates the number of atlases contained in the media track.

[0049] atlas_id indicates the identifier of the atlas.

[0050] The unit_header is a v3c_unit_header() as defined in the ISO / IEC format.

[0051] mix_component_flag indicates whether the component data are stored in the same media track. A value of 0 indicates that the components are stored in separate media tracks. A value of 1 indicates that the components are stored in the same media track.

[0052] The component_group_id indicates the identifier (ID) of a media track group that stores all component data belonging to the same atlas.

[0053] atlas_reference_track_id indicates the reference ID of the media track in which all component data corresponding to the atlas is stored.

[0054] V3C media tracks may reference / link to corresponding component media tracks using track references or other methods defined in the ISO / IEC format.

[0055] A V3C component track contains different types of component data, identified by a sample entry of type "v3cc". A V3C component track can contain component data from one atlas or multiple atlases. An example syntax for a V3C component track sample entry is shown in the table below. [Table 2]

[0056] The variables in the above table have the following definitions:

[0057] mix_component_flag indicates whether the component data are stored in the same media track. A value of 0 indicates that the components are stored in separate media tracks. A value of 1 indicates that the components are stored in the same media track.

[0058] num_atlas indicates the number of atlases contained in the media track.

[0059] atlas_id indicates the identifier of the atlas.

[0060] atlas_tile_flag indicates whether the component track contains atlas tiles. A value of 0 indicates that the component track does not contain atlas tiles. A value of 1 indicates that the component track contains one or more atlas tiles.

[0061] num_tile indicates the number of atlas tiles.

[0062] tile_id indicates the identifier of the atlas tile.

[0063] num_component indicates the number of components in the media track.

[0064] The component_type indicates the type of the component data.

[0065] The unit_header is a v3c_unit_header() as defined in the ISO / IEC format.

[0066] EMBODIMENT 2

[0067] In some embodiments, the atlas data and the component data of each atlas are stored in the same track. Figure 8 illustrates an exemplary encapsulation structure 800 of a media file in accordance with the present technology. A volumetric video media file includes a V3C track that stores atlas information that is common to all atlases. In this example, the V3C track contains zero samples. Thus, the MADT box does not contain any sample data for the V3C track.

[0068] Atlas data that is not common to all atlases can be stored with the corresponding component data for a particular atlas. As shown in FIG. 8, for Atlas_1 (801), the Atlas_1 data and all component data of different types (component 1, component 2) are stored in the same track. Similarly, for Atlas_2 (801), the Atlas_2 data and all component data of different types (component 1, component 2) are stored in the same track. That is, all related data for a particular atlas is packed together in a track, thereby minimizing synchronization delay and increasing transmission flexibility.

[0069] Here, V3C atlas and component tracks can be identified by a sample entry of type “v3ac.” Example syntax for V3C atlas and component track sample entries is shown in the table below. [Table 3]

[0070] The variables in the above table have the following definitions:

[0071] atlas_id indicates the identifier of the atlas.

[0072] The multi_atlas_flag indicates that the media track contains multiple atlases.

[0073] num_atlas indicates the number of atlases contained in the media track.

[0074] The unit_header is a v3c_unit_header() as defined in the ISO / IEC format.

[0075] EMBODIMENT 3

[0076] In some embodiments, component data of the same type from one or more atlases are stored in the same media track. The component data of all atlases can be organized into groups based on grouping criteria, for example, based on component type. Figure 9 illustrates an exemplary encapsulation structure 900 of a media file according to the present technology. As in embodiment 2, the media file includes a V3C track that stores atlas information that is common to all atlases, and the V3C track includes zero samples. Thus, the MADT box does not include any sample data for the V3C track.

[0077] In this embodiment, the media file includes a separate atlas track 901 (e.g., identified by an entry type "v3ca") to store atlas parameter information and / or data of all atlases. Multiple atlases are organized into one group based on grouping criteria. For example, multiple atlases may correspond to multiple camera layouts in a 3D area corresponding to 3D volumetric video data. For example, as shown in FIG. 9, component type I component data of multiple atlases (Atlas_1, Atlas_2, and Atlas_3 belonging to one group) are stored in the V3C component track 902. Similarly, component type II component data of multiple atlases (Atlas_1, Atlas_2, and Atlas_3) are stored in the V3C component track 903. That is, each V3C component track can include component data from one or more atlases. Such an organization allows more efficient processing when the decoding side needs to examine only one specific component type.

[0078] In addition to component types, the component tracks and the V3C atlas tracks can be organized into groups according to other grouping criteria. For example, a component track can store all types of component data of one group of atlases, and then the component tracks and the V3C atlas tracks can be grouped into one group. Various grouping criteria can be used to enable efficient partial processing of the 3D volumetric video data according to the characteristics of the 3D scene.

[0079] An example syntax for a V3C atlas track sample entry is shown in the table below. [Table 4]

[0080] The variables in the above table have the following definitions:

[0081] The multi_atlas_flag indicates that the media track contains multiple atlases.

[0082] num_atlas indicates the number of atlases contained in the media track.

[0083] atlas_id indicates the identifier of the atlas.

[0084] The atlas_group_flag indicates that all of the atlases in the V3C atlas track are grouped into one group.

[0085] atlas_group_id indicates the ID of the group to which all of the atlases in the V3C atlas track belong.

[0086] The unit_header is a v3c_unit_header() as defined in the ISO / IEC format.

[0087] A V3C component media track can be identified by a sample entry of type "v3cm". An example syntax for a V3C component track sample entry is shown in the table below. [Table 5]

[0088] The variables in the above table have the following definitions:

[0089] The component_type indicates the type of the component.

[0090] num_atlas indicates the number of atlases contained in the media track.

[0091] atlas_id indicates the identifier of the atlas.

[0092] The unit_header is a v3c_unit_header() as defined in the ISO / IEC format.

[0093] It should be appreciated that this document discloses techniques that may be embodied in various embodiments to provide flexible encapsulation of 3D volumetric video data. The disclosed techniques allow multiple atlases to be encapsulated in a single media file. Atlas data and component data of each atlas can be organized in different ways to reduce transmission costs and minimize synchronization complexity. The disclosed media content processing system and other embodiments, modules, and functional operations described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or to control the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition producing a machine-readable propagating signal, or one or more combinations thereof. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. In addition to hardware, the apparatus may include code that creates an execution environment for the computer program, such as processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiver apparatus.

[0094] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored within part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), within a single file dedicated to the program, or within multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0095] The processes and logic flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by, and apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0096] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from, or transfer data to, one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0097] Although this patent document contains many details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Also, although features may be described above as acting in a combination and may be initially claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0098] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed, to achieve desirable results. Additionally, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0099] Only some implementations and examples have been described; other implementations, improvements, and modifications can be made based on what is described and illustrated in this patent document.

Claims

1. 1. A method for video processing, the method comprising: Receiving three-dimensional (3D) volumetric video data; encoding the three-dimensional volumetric video data into a media file according to a file format; Including, 11. The method of claim 10, wherein the 3D volumetric video data corresponds to a plurality of atlases, each atlas comprising atlas data and one or more two-dimensional (2D) components, the file format specifies that the atlas data and the one or more 2D components for each atlas are stored in one or more media tracks within the media file, the file format further specifies that at least one 2D component of the one or more 2D components corresponding to an atlas is stored in a component track of the one or more media tracks, and a portion of the atlas data to which the 2D component corresponds and that is not common to all of the plurality of atlases is stored in the component track.

2. 1. A method for video processing, the method comprising: decoding a media file representing three-dimensional (3D) volumetric video data according to a file format, the 3D volumetric video data corresponding to a plurality of atlases, each atlas comprising atlas data and one or more two-dimensional (2D) components, the file format specifying that the atlas data and the one or more 2D components for each atlas are stored in one or more media tracks in the media file, the file format further specifying that at least one 2D component of the one or more 2D components corresponding to an atlas is stored in a component track of the one or more media tracks, and a portion of the atlas data to which the 2D component corresponds and which is not common to all of the plurality of atlases is stored in the component track; reconstructing the 3D volumetric video data based on the one or more media tracks; and A method comprising:

3. The method of claim 1 or claim 2, wherein the one or more 2D components for each atlas comprise at least a geometry component, an occupancy component, or an attribute component.

4. The method of any of claims 1-3, wherein the atlas data comprises a projective relationship that projects the 3D volumetric video data from a 3D space onto a 2D plane.

5. The method of any of claims 1 to 4, wherein at least one media track in the media file contains information representing some or all of the 3D volumetric video data.

6. The method of any one of claims 1 to 5, wherein the one or more media tracks include a volumetric visual track having a particular sample entry type, the volumetric visual track comprising parameter information common to all of the plurality of atlases.

7. The method of claim 6 , wherein the volumetric visual track includes zero samples.

8. The method of claim 6 , wherein the volumetric visual track includes one or more samples, each sample carrying atlas data for one or more atlases.

9. The method of any of claims 1 to 8, wherein the one or more media tracks include an atlas track having a particular sample entry type, the atlas track comprising atlas data for one or more atlases.

10. 10. The method of claim 1, wherein the one or more media tracks include a first component track having a first sample entry type, the first component track comprising data for the one or more 2D components corresponding to the plurality of atlases.

11. 11. The method of claim 1, wherein the one or more media tracks include a second component track having a second sample entry type, the second component track comprising the atlas data and data for the one or more 2D components corresponding to the plurality of atlases.

12. The method of any of claims 1 to 11, wherein the one or more media tracks include a third component track having a third sample entry type, the third component track carrying a particular type of component data from the multiple atlases.

13. the one or more media tracks are organized into a plurality of groups; The method according to any of the preceding claims, wherein each group corresponds to one of said plurality of atlases depending on a grouping criterion associated with said 3D volumetric video data.

14. Video processing apparatus, said video processing apparatus comprising a processor, said processor being configured to implement the method according to any one of claims 1 to 13.

15. A computer program product having code stored therein, the code causing the processor to implement a method according to any one of claims 1 to 13 when executed by a processor.

Images