Method and apparatus for encapsulating 3D region-related annotations in media files - Patents.com

Abstract

The present invention relates to a method for encapsulating volumetric media data within a file, comprising: generating a volumetric media entity (200) describing the volumetric media data; generating a 3D region annotation entity (210) associated with a region of the volumetric media data, the 3D associated annotation entity being associated with the volumetric media entity; generating geometry data (220) associated with the 3D region annotation entity describing a geometry of the region of the volumetric media data; generating at least one annotation data structure (230, 231, 232, 931, 932, 933) associated with the 3D region annotation entity; and embedding the volumetric media entity, the 3D region annotation entity, the geometry data and the at least one annotation data structure within the file.

Description

[Technical field]

[0001] The present disclosure relates to methods and devices for encapsulating information relating to three-dimensional regions within media files. [Background technology]

[0002] LiDAR devices and volumetric media analysis services can generate localized metadata for volumetric media. Localized metadata is metadata that is related to a region, a portion, of the media content, rather than the entire media content. This media content is typically a point cloud, volumetric media, but may be any type of three-dimensional media content. For example, LiDAR can generate different regions for a point cloud according to its capture time. As another example, a deep learning system can identify objects within the volumetric media. These localized metadata can be viewed as three-dimensional region annotations.

[0003] Media content captured by a camera or LiDAR, or processed by a media analysis service, is stored on a storage device, such as a memory card. The media content is typically encoded to reduce the size of the data on the storage device. MPEG is currently standardizing the representation of volumetric media, i.e., 3D "images" or 3D "video".

[0004] G-PCC, geometry-based point cloud compression, allows to compress point clouds. A media is a set of 3D points with associated properties, called attributes. The compression of the media is based on the geometry of the point cloud, as defined by the coordinates of the points. Other properties of the point cloud are represented as point attributes, e.g., color, reflectance, opacity. Typically, G-PCC is used to encode point clouds generated by LiDAR.

[0005] V3C, Visual Volumetric Video-based Coding, allows for compressing volumetric media, while V-PCC, Video-based Point Cloud Compression, applies these techniques to point clouds. The 3D representation of the media is first converted into multiple 2D representations that are encoded using classical image or video codecs (e.g. AVC, HEVC, VVC, etc.). Typically, V3C or V-PCC are used to encode 3D objects used in virtual / augmented / mixed reality content.

[0006] Volumetric media encoded using G-PCC, V3C, or V-PCC can be stored in files using the ISO Base Media File Format (ISOBMFF - ISO / IEC 14496-12), or derived specifications. Extensions to ISOBMFF have been proposed for storing volumetric media, such as ISO / IEC 23090-10 for carrying visual volumetric video-based coding data, or ISO / IEC 23090-18 for carrying geometry-based point cloud compressed data. In the following, "ISOBMFF" refers to ISOBMFF and its extensions for carrying volumetric media.

[0007] While providing the ability to store documents containing metadata such as EXIF ​​or XMP documents, ISOBMFF and extensions to ISOBMFF for carrying volumetric media do not provide mechanisms adapted for linking annotations to three-dimensional regions of volumetric media. Summary of the Invention

[0008] The present invention has been designed to address one or more of the problems set forth above.

[0009] According to one aspect of the present invention, a method for encapsulating volumetric media data in a file is proposed, the method comprising the steps of: - generating a volumetric media entity describing the volumetric media data; - generating a 3D region annotation entity related to a region of the volumetric media data, the 3D region annotation entity being associated with the volumetric media entity; - generating geometry data associated with a 3D region annotation entity to describe a geometry of a region of the volumetric media data; - generating at least one annotation data structure associated with a 3D region annotation entity; - Embedding volumetric media entities, 3D region annotation entities, geometry data, and at least one annotation data structure in the file.

[0010] According to one embodiment, - the volumetric media entity is the first item, - the 3D region annotation entity is the second item, The geometry data is associated with a second item.

[0011] According to one embodiment, the at least one annotation data structure is a property of the second item.

[0012] According to one embodiment, at least one annotation data structure is an item associated with a second item.

[0013] According to one embodiment, the geometry data is included in the second item.

[0014] According to one embodiment, the geometry data is included in the first item.

[0015] According to one embodiment, the geometry data is determined from identifiers associated with the points of the volumetric media data.

[0016] According to one embodiment, the geometry data is the third item.

[0017] According to one embodiment, the at least one annotation data structure is a group of at least one item.

[0018] According to one embodiment, at least one item represents a plane within the volumetric media data.

[0019] According to one embodiment, a volumetric media entity is a volumetric media track, and the volumetric media data is contained in samples of the volumetric media track; - a 3D region annotation entity is a 3D region annotation track, -The geometry data is included in the sample 3D region annotation track.

[0020] According to one embodiment, a volumetric media entity is a volumetric media track, and the volumetric media data is contained in samples of the volumetric media track; - a 3D region annotation entity is a 3D region annotation track, The geometry data is contained in samples in a separate track that is associated with the 3D region annotation track.

[0021] According to one embodiment, the at least one annotation is an item property associated with a group of samples of the 3D region annotation track.

[0022] According to one embodiment, a 3D region annotation track is associated with a track that provides a representation of the region of interest described in the 3D region annotation track.

[0023] According to another aspect of the invention, a method for reading a file containing volumetric media data is proposed, the method comprising the steps of: - reading a volumetric media entity describing the volumetric media data to obtain the volumetric media data; - reading a 3D region annotation entity associated with a region of volumetric media data, the 3D region annotation entity being associated with the volumetric media entity; - reading geometry data associated with a 3D region annotation entity to describe a geometry of a region of the volumetric media data; - reading at least one annotation data structure associated with a 3D region annotation entity to obtain annotation data; - processing the acquired volumetric media data and the acquired at least one annotation data as a function of a geometry of a region of the volumetric media data.

[0024] According to another aspect of the invention there is provided a computer program product for a programmable device comprising sequences of instructions for implementing the method according to the invention when loaded and executed on a programmable device.

[0025] According to another aspect of the invention, a non-transitory computer readable storage medium is proposed, storing computer program instructions for implementing the method according to the invention.

[0026] According to another aspect of the invention, a computer program is proposed which, when executed, causes the method of the invention to be performed.

[0027] According to another aspect of the invention, a device for encapsulating volumetric media data in a file is proposed, the device comprising: - generating a volumetric media entity describing the volumetric media data; - generating a 3D region annotation entity related to a region of the volumetric media data, the 3D region annotation entity being associated with the volumetric media entity; - generating geometry data associated with a 3D region annotation entity to describe a geometry of a region of the volumetric media data; - generating at least one annotation data structure associated with a 3D region annotation entity; - a processor configured for embedding volumetric media entities, 3D region annotation entities, geometry data, and at least one annotation data structure into a file.

[0028] According to another aspect of the invention, a device for reading a file containing volumetric media data is proposed, the device comprising: - reading a volumetric media entity describing the volumetric media data to obtain the volumetric media data; - reading a 3D region annotation entity associated with a region of volumetric media data, the 3D region annotation entity being associated with the volumetric media entity; - reading geometry data associated with a 3D region annotation entity to describe a geometry of a region of the volumetric media data; - reading at least one annotation data structure associated with a 3D region annotation entity to obtain annotation data; - processing the acquired volumetric media data and the acquired at least one annotation data as a function of a geometry of a region of the volumetric media data.

[0029] At least some of the methods according to the invention may be computer-implemented. Accordingly, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally in this specification as a "circuit," "module," or "system." Furthermore, the invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

[0030] Since the present invention can be realized in software, the present invention can be embodied as computer readable code for providing to a programmable device on any suitable carrier medium. Tangible non-transitory carrier medium may include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices or solid state memory devices. Transient carrier medium may include signals such as electric, electronic, optical, acoustic, magnetic or electromagnetic signals, e.g. microwave or RE signals. [Brief description of the drawings]

[0031] Embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which: [Figure 1]Here is an example of an ISOBMFF file that contains one or more still images and possibly media data such as a video or sequence of images. [Figure 2a] 1 shows a high level diagram of the present invention for associating annotations with regions of volumetric media. [Figure 2b] 1 shows an example of 3D region annotation. [Diagram 3] 1 shows a first embodiment of the present invention. [Figure 4] 2 shows a second embodiment of the present invention. [Diagram 5] 3 shows a third embodiment of the present invention. [Figure 6] 1 illustrates the main steps of a process for adding a new 3D region annotation to a volumetric media item stored in an ISOBMFF file according to an embodiment of the present invention. [Figure 7] 1 illustrates the main steps of a process for reading an ISOBMFF file containing 3D region annotations according to an embodiment of the invention. [Figure 8] 1 illustrates a process for processing an ISOBMFF file that includes volumetric media and one or more 3D region annotation items associated with the volumetric media, according to an embodiment of the present invention. [Figure 9a] , and [Figure 9b] 4 shows a fourth embodiment of the present invention. [Figure 10] 13 illustrates a fifth embodiment for annotation regions in volumetric media tracks. [Figure 11] 1 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] In all descriptions, the term image is used as a general term encompassing both 2D images and 3D or volumetric images. Similarly, the term video is used as a general term encompassing both 2D video and 3D video or volumetric media. Volumetric media may encompass media encoded with G-PCC, V3C or V-PCC.

[0033] The ISOBMFF standard (see "Information technology - Encryption for audio-visual purposes - Part 12: ISO Base Media File Format", w20295, ISO / IEC 14496-12, 7th edition, May 2021) storing timed media data (e.g. video or image sequences) as a timed sequence of related samples; Storage of non-timed media data as a single item or a collection of items; This covers two forms of storage that serve different use cases:

[0034] The ISO Base Media File Format is object-oriented. It is composed of building blocks called boxes, which correspond to data structures characterized by a unique type identifier, typically a four-letter code (also called FourCC or 4CC). A full box is a data structure similar to a box that additionally contains version and flag value attributes. In the following, the term box may refer to a full box or both boxes. These boxes or full boxes are organized hierarchically or sequentially within the ISOBMFF file and define parameters that describe the encoded timed or non-timed media data, its structure and timing (if any). All data in the encapsulated media file (media data and metadata describing the media data) is contained in a box. There is no other data in this file. A file-level box is a box that is not contained in another box.

[0035] The presentation of timed media data is described in a file-level box called Movie Box (4-letter code "moov"). This Movie Box represents an initialization information container that includes a set of different boxes that describe the presentation and its timing. It is logically divided into tracks represented by track boxes (with the 4-letter code "trak"). Each track (uniquely identified by a track identifier (track_ID)) represents a timed sequence of media data that belongs to the presentation (e.g. a sequence of frames of a video, audio samples, or images). In contrast to frames of a video, a sequence of images specifies a timed sequence of images whose timing is only advisory, which may be the timing at the time of collection (e.g. of an image burst) or a suggested display timing (e.g. of a slide show).

[0036] Within each track, each timed unit of data is called a sample, which may be a frame of video, audio or timed metadata, or an image of a sequence of images. Samples are implicitly numbered in a decoding order sequence. Each track box contains a hierarchy of boxes that describe the samples of the track, for example the Sample Table box ("stbl") contains the time and data indexing of the media samples within the track. The actual sample data is stored at the same level as the movie box in boxes called Media Data Boxes (with the four letter code "mdat") or Identified Media Data Boxes (similar to Media Data Boxes but with the four letter code "imda" which contains an additional identifier).

[0037] Non-timed media data is described in a meta box (with the four-letter code "meta"). Units of non-timed media data under this box and their hierarchy relate to "information items" or "items" instead of to associated samples. Note that the expressions "box" and "container" may both be used interchangeably to refer to data structures that contain metadata describing the organization and / or properties of image data within a file.

[0038] Figure 1 shows an example of an ISOBMFF file that contains media data such as one or more still images and possibly one or more videos and / or one or more image sequences. The file contains a first "ftyp" box (FileTypeBox) 111 that contains an identifier of the type of file (usually a set of four character codes). The file contains a second box (MetaBox) 102 called "meta" that is used to contain general non-timed metadata including metadata structures describing one or more still images. This "meta" box 102 contains an "iinf" box (ItemInfoBox) 121 that describes several single images. Each single image is described by a metadata structure ItemInfoEntry, also denoted items 1211 and 1212. Each item has a unique 16-bit or 32-bit identifier item_ID. The media data corresponding to these items is stored in an "mdat" box 104, which is a container for media data. An "iloc" box (ItemLocationBox) 122 provides, for each item, the offset and length of the associated media data in "mdat" 104. An "iref" box (ItemReferenceBox) 123 may also be defined to describe the association of one item with other items via typed references.

[0039] Optionally, to describe the storage of an image sequence or a video, the ISOBMFF file 101 may contain a third box called "moov" (MovieBox) 103 that describes one or more image sequence or video tracks 131 and 132. Typically, track 131 may be an image sequence track designed to describe a set of images where time information is not necessarily meaningful, and track 132 may be a video ("vide") track designed to describe video content. Both tracks describe a series of image samples, where an image sample is a set of pixels captured at the same time, for example a frame of a video sequence. The main difference between the two tracks is that in the image sequence track, timing information is not necessarily meaningful, whereas for the "vide" track, timing information is intended to constrain the timing of the display of the samples. The data corresponding to these samples is stored in an "mdat" box 104, which is a container for media data.

[0040] The 'mdat' container 104 stores non-timed encoded images corresponding to the items represented by data portions 141 and 142 , and a timed encoded image corresponding to the samples represented by data portion 143 .

[0041] The ISOBMFF file 101 offers different alternatives for storing multiple images. For example, multiple images may be stored either as an item or as a sample track. The actual choice is typically made by the application or device generating the file according to the type of image and the expected use of the file.

[0042] The ISOBMFF specifies several alternatives for grouping samples or items depending on the containers that hold the samples or items to be grouped. These alternatives can be viewed as grouping data structures or mechanisms, i.e. boxes or data structures that provide metadata describing grouping criteria and / or group properties and / or group entities.

[0043] The first grouping mechanism, represented by EntityToGroupBox, applies to grouping of items or tracks. In this mechanism, the expression "entity" is used to refer to items or tracks or other EntityToGroupBoxes. This mechanism specifies a grouping of entities. EntityToGroupBox is defined according to the following syntax: aligned(8) class EntityToGroupBox(grouping_type, version, flags) extends FullBox(grouping_type, version, flags) { unsigned int(32) group_id; unsigned int(32) num_entities_in_group for (i = 0; i < num_entities_in_group; i++) { unsigned int (32) entity_id; / / The remaining data may be specified for a particular grouping_type }

[0044] The grouping_type is used to specify the type of group. The group_id provides an identifier for a group of entities. The entity_id represents the identifiers of the entities that make up the group, i.e. either a track_ID for a track, an item_ID for an item, or another group_id for an entity group. In Figure 1, a group of entities inheriting from EntityToGroup boxes 1241 and 1242 is contained in a container 124 identified by the four-letter code "gprl" of the GroupsListBox.

[0045] An entity grouping consists of associating a grouping type that identifies the reason for grouping a set of items, tracks, or other entity groups. In this document, grouping information is referred to as information in one of the EntityToGroups that conveys information for grouping a set of images.

[0046] ISOBMFF provides a mechanism for describing and associating properties with items. These properties are called item properties. The ItemPropertiesBox 125 allows the association of any item with an ordered set of item properties. The ItemPropertiesBox consists of two parts: an item property container box "ipco" 1251 that contains an implicitly indexed list of item properties 1253, and an item property association box "ipma" 1252 that contains one or more entries. Each entry in the item property association box associates an item with its item properties. The ISOBMFF standard extends this mechanism to allow the association of item properties with items and / or entity groups, for example by using the "unif" brand. It should be noted that in this description, we use item properties generically to specify both properties of items or properties of entity groups in a generic way. Item properties associated with an entity group apply to the entity group as a whole, not to each entity in the group individually.

[0047] The associated syntax is: aligned(8) class ItemProperty(property_type) extends Box(property_type) { } aligned(8) class ItemFullProperty(property_type, version, flags) extends FullBox(property_type, version, flags) { } aligned(8) class ItemPropertyContainerBox extends Box('ipco') { properties ItemProperty() []; / / Box is derived from / / ItemProperty or ItemFullProperty, fill the box } aligned(8) class ItemPropertyAssociation extends FullBox('ipma', version, flags) { unsigned int(32) emtry_count; for(i = 0; i < entry_count; i++) { if (version < 1) unsigned int(16) item_ID; else unsigned int(32) item_ID; unsigned int(8) association_count; for (i=0; i <association_count; i++) { bit(1) essential; if (flags & 1) unsigned int(15) property_index; else unsigned int(7) property_index; } } } aligned(8) class ItemPropertyBox extends Box('iprp') { ItemPropertyContainerBox property_container; ItemPropertyAssociation association[];}

[0048] The ItemProperty and ItemFullProperty boxes are designed for the description of item properties. ItemFullProperty allows to define several versions of the box syntax and may contain one or more parameters whose presence is conditioned by either the version or flag parameters.

[0049] The ItemPropertyContainerBox is designed to describe a set of item properties as an array of ItemProperty boxes or ItemFullProperty boxes.

[0050] The ItemPropertyAssociation box is designed to describe the association between items and / or entity groups and their item properties. It provides a description of a list of item identifiers and / or entity group identifiers, where each identifier (item_ID) is associated with a list of item property indexes that refer to the item properties in the ItemPropertyContainerBox.

[0051] None of these mechanisms make it possible to describe some properties associated with a part or region of a volumetric media. The present invention aims to solve this problem by providing a way to describe 3D region annotations and to associate these 3D region annotations with actual regions of the volumetric media.

[0052] Fig. 2a shows a high level diagram of the invention for associating annotations to regions of volumetric media. A region annotation 210 is associated with an entity 200. The entity 200 may correspond to volumetric media contained in a single item. The entity 200 may also correspond to an entry point of volumetric media contained in several items, i.e. an entry point item 201 and one or more component items 202 and 203. The entity is for example a G-PCC item, a V3C item or a V-PCC item describing some volumetric media. The entity 200 may also be another type of item, a track (as shown in Fig. 10) or an entity group. This association means that the region annotation 210 contains information related to a region that means a part of the volumetric media or entity group described by the entity 200.

[0053] For example, in a single item encapsulation, respectively a single track encapsulation, of G-PCC data, entity 200 is a G-PCC item, respectively a G-PCC track, describing a bitstream representing all components of a G-PCC frame, respectively a number of G-PCC frames. A region annotation 210 is associated with entity 200 representing a G-PCC item, respectively a G-PCC track, via an item reference, respectively a track reference, from region annotation 210 to entity 200.

[0054] Alternatively, in a multi-item encapsulation, respectively a multi-track encapsulation, of G-PCC data, the entity 200 may: - one G-PCC item or track carrying a G-PCC geometry component representing the entry point item or track 201, and - zero or more G-PCC items or tracks 202 and 203 carrying G-PCC attribute components, each represented by a G-PCC component track.

[0055] In this case, the region annotation 210 is preferably associated to the entry point 201 representing the G-PCC geometry component via an item reference, a respective track reference, from the region annotation 210 to the respective track of the G-PCC item representing the entry point 201. Alternatively, the region annotation 210 may be associated to the G-PCC item or track carrying the attribute component referenced by the region annotation 210 when the geometry of the region is determined from attribute values ​​as shown in FIG. 4).

[0056] Alternatively, in a multi-item encapsulation, respectively a multi-track encapsulation, of tiled G-PCC data, the entity 200 may: one G-PCC item or track representing a G-PCC tile-based item or a G-PCC tile-based track, carrying all of the parameter set and tile inventory data units and no geometry or attribute data units; and - at least one G-PCC tile item or G-PCC tile track that holds geometry and attribute components, each represented by a number of G-PCC items, each represented by a G-PCC track.

[0057] In a variant, a G-PCC tile may also be encapsulated into multiple G-PCC tile items or G-PCC tile tracks, i.e. a G-PCC tile item or G-PCC tile track carrying geometry components and zero or more G-PCC tile items or G-PCC tile tracks carrying attribute components.

[0058] In the above case, the G-PCC tile base item or the G-PCC tile base track represents the entry point 201. The region annotation 210 may be associated with the entry point 201 representing the G-PCC tile base item, respectively the G-PCC tile base track, via an item reference, respectively a track reference, from the region annotation 210 to the entry point 201. Alternatively, it may be useful to associate an annotation tile per tile in order to filter annotations according to the tile accessed by the parser or renderer. In such a case, the region annotation 210 describing the region to be annotated within the tile may be associated with the G-PCC tile item or the G-PCC tile track carrying the geometry components of the tile it annotates. Alternatively, the region annotation 210 may be associated with the G-PCC tile item or the G-PCC tile track carrying the attribute components referenced by the region annotation 210, for example when the geometry of the region is determined from attribute values ​​as shown by FIG. 4.

[0059] In another example, for a multi-item or multi-track encapsulation of V3C data, the entity 200 is represented by multiple items or tracks, with a V3C atlas item or V3C atlas track carrying encrypted atlas access units of the V3C data representing the entry point 201 and a V3C component item or V3C component track representing the components 202 and 203. In this case, the region annotation 210 is preferably associated with the entry point 201 representing the V3C atlas item, respectively the V3C atlas track, via an item reference, respectively a track reference, from the region annotation 210 to the entry point 201.

[0060] In the case of V3C tiled data, the region annotation 210 may be associated via an item reference, respectively a track reference, to a V3C atlas item, respectively a V3C atlast track, or a V3C atlast tile item, respectively a V3C atlast tile item, that corresponds to a tile for annotating from the region annotation 210 to the V3C atlas item, respectively a V3C atlast track, or to the V3C atlast tile item, respectively a V3C atlast tile item.

[0061] Alternatively, the region annotation 210 may be associated with the V3C item, respectively V3C track, carrying the geometry component, via an item reference, respectively track reference, from the region annotation 210 to the V3C item, respectively V3C track, carrying the geometry component. Alternatively, the region annotation 210 may be associated with the V3C item or V3C track carrying the attribute component referenced by the region annotation 210 when the geometry of the region is determined from the attribute value, as shown for example by FIG. 4). In the case of a media file having V3C items, respectively V3C tracks, with multiple atlases, the region annotation 210 may be associated with the V3C atlas item, respectively V3C atlas track, which corresponds from the region annotation 210 to the V3C atlas item, respectively V3C atlas track, via an item reference, respectively track reference, from the region annotation 210 to the V3C atlas item, respectively V3C atlas track. The item or track reference type may be "cdsc" to indicate that the region annotation provides a description of the referenced item or track.

[0062] When the entity is a G-PCC item, a V3C item, or a V-PCC item, the reconstructed media is the volumetric media resulting from decrypting an encrypted volumetric media item, or from applying a derived item operation to a set of input items (which themselves are encrypted volumetric media items or derived volumetric media items). The output volumetric media is the volumetric media resulting from applying potential transformation item properties of the volumetric media items to the reconstructed volumetric media. For example, the output volumetric media may be the volumetric media obtained after decrypting data associated with the items and applying transformation item properties to the decrypted volumetric media.

[0063] This 3D region annotation 210 may be defined by its geometry 220. The geometry may be defined as a position, a position and a shape, a set of points, or a mask. For example, the geometry of a region may be a cuboid defined by its center and its width, length, and height. The geometry may be defined by other means. The geometry may be divided into a position and, optionally, a shape. Several kinds of shapes may be defined, for example, a point (in such a case the geometry is only a position), a cuboid, or an ellipsoid. This position is the location in the annotated volumetric media of a reference point for the shape. For example, this reference point may be the center of a cuboid or the center of an ellipsoid. The geometry may also be defined as a mask that selects individual points in the volumetric media that correspond to the entity 200. Other shapes, such as 3D meshes, may also be used. The geometry may be described with one generic identifier and a parameter providing its type (point, cuboid, ellipsoid...) or as a specific identifier for each type. For volumetric media encapsulated in an ISOBMFF file, this identifier may be a four-letter code.

[0064] The geometry may be stored in the data for the 3D region annotation 210, typically in an "mdat" box. The geometry may be stored in one or more other entities associated with the 3D region annotation 210. The geometry may be stored in one or more item properties associated with the 3D region annotation 210.

[0065] The 3D region annotation 210 may be linked to one or more annotations 230, 231, and 232. These annotations are used to store different annotations corresponding to the 3D region annotation 210 of the entity 200. The annotations may be, for example, objects detected within the volumetric media, GPS locations corresponding to objects within the volumetric media, a description of a portion of the volumetric media, text included in a region of the volumetric media, user annotations of a portion of the volumetric media, or any type of information related to the region.

[0066] Fig. 2b shows an example of 3D region annotation. Volumetric data 250 is captured, for example, by a LiDAR, which adds information about two people detected at the time of capture. The people correspond to 3D regions 260 and 261 in the volumetric data 250. Both 3D regions are cuboid regions. This captured volumetric media may be stored in an ISOBMFF file using the present invention.

[0067] Later, for example during transfer of the volumetric media 250 to an online-based volumetric data storage, further processing is applied to the volumetric media to enhance the information describing the volumetric media. First, a recognition algorithm finds the name of the person represented by the region 261. Second, an object detection algorithm detects and identifies buildings within the region 270 of the volumetric media 250, which is an elliptical region. These processing steps result in annotated volumetric media that can be stored in an ISOBMFF file using the present invention. For example, the ISOBMFF file containing the volumetric media 250 is edited to store the description of the new 3D region annotations 270 and / or the name of the detected person in the 3D region 261.

[0068] Finally, the user edits the volumetric media 250, for example, to retain only the portions that include known people in region 261 and buildings in region 270. As a result, only portion 280 of the volumetric media 250 is retained. These editing steps result in edited volumetric media that can be stored in an ISOBMFF file using the present invention. Such an edited ISOBMFF file may then contain either the original volumetric media 250 and instructions for obtaining the cropped volumetric media 280, or may contain only the volumetric media description and data for region 280.

[0069] 3-8 illustrate some embodiments of the present invention for the ISOBMFF standard or any derived standard, where the 3D region annotation 210 is represented using items.

[0070] FIG. 3 illustrates a first embodiment of the present invention. The "meta" box 300 includes, for example, item 310, which corresponds to a volumetric media item. In some cases, item 310 is an entry point to the volumetric media. The "meta" box 300 also includes two other items 320 and 325, which correspond to 3D region annotations. The geometry of these two 3D region annotation items is described in their respective contents, shown as geometry 321 and geometry 326. These contents are identified in "iloc" boxes. The contents of the 3D region annotation items may preferably be stored in an "idat" box. The contents of the 3D region annotation items may also be stored in an "mdat" or "imda" box.

[0071] The "iref" box 330 contains two entries 331 and 332 that associate the 3D region annotation items 320 and 325, respectively, with the volumetric media item 310.

[0072] The "iprp" box 340 contains an "ipco" box 360 and an "ipma" box 350. The "ipco" box 360 contains item properties 361 and 362 that correspond to the annotations of the two 3D region annotation items 320 and 325. The "ipma" box 350 associates the 3D region annotation items 320 and 325 with their respective item properties 361 and 362 via two entries 351 and 352, respectively.

[0073] In this embodiment, a 3D region annotation item may be defined with an item_type of "3dan." The 3D region annotation item may be associated with its volumetric media item via an item reference box of type "cdsc."

[0074] The content syntax for a 3D Region Annotation item is: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(8) geometry_type; if (geometry_type == 0) { / / Point signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; } if (geometry_type == 1) { / / Polyline unsigned int(field_size) point_count; for (i=0; i < point_count; i++) { signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; } } if (geometry_type == 2) { / / plane unsigned int(field_size) point_count; for (i = 0; i < point_count; i++) { signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) normal_x; signed int(field_size) normal_y; signed int(field_size) normal_z; } } if (geometry_type == 3) { / / Cuboid signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) size_x; signed int(field_size) size_y; signed int(field_size) size_z; signed int(field_size) quaternion_w; signed int(field_size) quaternion_x; signed int(field_size) quaternion_y; signed int(field_size) quaternion_z; } if (geometry_type == 4) { / / Ellipsoid signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) size_x; signed int(field_size) size_y; signed int(field_size) size_z; signed int(field_size) quaternion_w; signed int(field_size) quaternion_x; signed int(field_size) quaternion_y; signed int(field_size) quaternion_z; } } } may be also possible.

[0075] The semantics of the 3D region annotation item are: -version is the version of the 3DRegionItem structure. -flags is a set of flags that define options for the structure. -field_size is an integer value that specifies the size in bits (e.g. 32 or 64 bits) of the field used to describe the geometry of the region. As an example, field_size is calculated here from the value of the least significant bit of the flags value. If this bit is set to 0, then field_size is 32 bits, if this bit is set to 1, then field_size is 64 bits.

[0076] -reference_size_x, reference_size_y, reference_size_z are fixed point numbers that specify the size in meters of the reference space in which the region is placed.

[0077] -region_count is the number of regions described in the structure.

[0078] -geometry_type is the type of the region's geometry.

[0079] -x, y, z are fixed point numbers specifying the location of the region in meters in the reference space. If the region is a point, x, y, z are the coordinates of this point. If the region is a set of lines, x, y, z are the coordinates of a point of this set of lines. If the region is a plane, x, y, z are the coordinates of a point in this plane. If the region is a cuboid or ellipsoid, x, y, z are the coordinates of the center of this cuboid or ellipsoid.

[0080] -point_count is the number of points that make up the polyline.

[0081] -normal_x, normal_y, normal_z are fixed point numbers that specify the normal vector of the plane if the region is a planar.

[0082] -size_x, size_ze, y_size_z are fixed point numbers that specify the dimensions of the rectangle or ellipse in meters if the region is a rectangle or ellipse.

[0083] - quaternion_w, quaternion_xquaternion_y, quaternion_z are fixed point numbers that specify the rotation of the rectangle or ellipsoid about its center, using a quaternion representation, if the region is a rectangle or ellipsoid.

[0084] As an example, the coordinates of the region are represented here using a fixed-point representation, which may use a 16.16 representation if field_size is 32 bits, or a 32.32 representation if field_size is 64 bits. Extensions of the 3D region annotation items may be achieved by defining new geometry types.

[0085] Other types of representations may be used for the coordinates of the region, e.g. integer representation, floating point representation.

[0086] Similarly, by way of example, the values ​​quaternion_w, quaternion_x, quaternion_y, quaternion_z representing the rotation of a rectangular prism or an ellipsoid are represented using a fixed-point representation, they may also be represented using other types of representations, such as integer or floating-point representations.

[0087] In this embodiment, all item properties associated through an "ipma" box entry to a 3D region annotation item are applied to each region of the annotated volumetric media defined by the geometry of the 3D region annotation item.

[0088] In some cases, entities may be associated with a region annotation item via an entry in the "iref" box 300 to represent an annotation of the region described by the region annotation item.

[0089] In this embodiment, the annotated volumetric media from FIG. 2b may be stored in an ISOBMFF file having the following structure: FileTypeBox 'ftyp': major-brand = 'gpci', compatible-brands = 'mif1' MetaBox 'meta': (container) HandlerBox 'hdlr': 'volv' PrimaryItemBox 'pitm': item_ID = 1; ItemInfoBox 'iinf': entry_count = 4 'infe': item_ID = 1, item_type = 'gpe1'; / / (Ref. 250) 'infe': item_ID = 2, item_type = '3dan', hidden = true; / / (Refer to 260) 'infe': item_ID = 3, item_type = '3dan', hidden = true; / / (Refer to 261) 'infe': item_ID = 4, item_type = '3dan', hidden = true; / / (Refer to 270) ItemLocationBox 'iloc': item_count = 1 item_ID = 1, extent_count = 1, extent_offset = X, extent_length = Y; item_ID = 2, extent_count = 1, extent_offset = O1, extent_length = L1; item_ID = 3, extent_count = 1, extent_offset = O2, extent_length = L2; item_ID = 4, extent_count = 1, extent_offset = O3, extent_length = L3; ItemReferenceBox 'iref': referenceType = 'cdsc', from_item_ID = 2, reference_count = 1, to_item_ID = 1; referenceType = 'cdsc', from_item_ID = 3, reference_count = 1, to_item_ID = 1; referenceType = 'cdsc', from_item_ID = 4, reference_count = 1, to_item_ID = 1; ItemPropertiesBox 'iprp': ItemPropertyContainerBox 'ipco': 1) 'gpcC' / / Annotated volumetric media configuration 2) 'gpsr' / / Size of the annotated volumetric media 3) 'udes', lang = en, name = 'person', description = ", tags = person 4) 'udes', lang = fr, name = 'Jean', description = ", tags = person 5) 'udes', lang = fr, name = 'Notre Dame', description = 'Notre Dame de Paris', tags = building ItemPropertyAssociation 'ipma': entry_count = 4 1) item_ID = 1, association_count = 2 essential = 1, property_index = 1; essential = 0, property_index = 2; 2) item_ID = 2, association_count = 1 essential = 0, property_index = 3; 3) item_ID = 3, association_count = 2 essential = 0, property_index = 3; essential = 0, property_index = 4; 4) item_ID = 4, association_count = 1 essential = 0, property_index = 5; MediaDataBox 'mdat' or 'idat': Volumetric media (at file offset X, with length Y) Region Annotation (at file offset O1, with length L1) geometry: type = 3, x = x0, y = y0, z = z0, size_x = sx0, size_y = sy0, size_z = sz0, quaternion_w = qw0, quaternion_x = qx0, quaternion_y = qy0, quaternion_z = qz0; Region Annotation (at file offset O2, with length L2) geometry: type = 3, x = x1, y = y1, z = z1, size_x = sx1, size_y = sy1, size_z = sz1, quaternion_w = qw1, quaternion_x = qx1, quaternion_y = qy1, quaternion_z = qz1; Region Annotation (at file offset O3, with length L3) geometry: type = 4, x = x2, y = y2, z = z2, size_x = sx2, size_y = sy2, size_z = sz2, quaternion_w = qw2, quaternion_x = qx2, quaternion_y = qy2, quaternion_z = qz2;

[0090] In this ISOBMFF file, volumetric media item 250 is represented by an item with an item_ID value of 1. Region 260 is represented by an item with an item_ID value of 2, which is associated with the item property at index 3 through the second entry in the "ipma" box. This property corresponds to a person annotation. The geometry of the 3D region annotation item is described in the first region annotation part of the MediaDataBox.

[0091] Region 261 is represented by an item with an item_ID value of 3. It is associated with item properties with indexes 3 and 4 through the third entry in the "ipma" box. These properties correspond to a person annotation and an annotation describing a person named "Jean", respectively. The geometry of the 3D region annotation item is described in the second region annotation part of the MediaDataBox.

[0092] Region 270 is represented by the item with item_ID value 4. It is associated with the item property at index 5 through the fourth entry in the "ipma" box. This property corresponds to an annotation describing "Notre Dame de Paris" as a building. The geometry of the 3D region annotation item is described in the third region annotation part of the MediaDataBox.

[0093] Each 3D region annotation item is associated to annotated volumetric media via an item reference of type "cdsc".

[0094] All 3D area annotation items MAY have a hidden property set to true that indicates to the reader that this item is not intended to be displayed (e.g., the item has (flags & 1) equal to 1 in its ItemInfoEntry).

[0095] In some cases, another item (not shown) may be associated with the 3D region annotation item 320, for example to provide a detailed view of the region. In this case, an additional entry may be added in the "iref" box 330, for example with a reference type "eroi" from the 3D region annotation item 320 to this other item. This other item may be a volumetric media item, an image item, or an entity group. The item may also be a track, for example a volumetric media track, a video track, an audio track, etc.

[0096] 4 illustrates a second embodiment of the invention. In this embodiment, the geometry of the region is not described in the 3D region annotation item itself, but relies on information contained in the volumetric media. In this embodiment, the volumetric media corresponding to item 310 includes an attribute that is used to define the region. This attribute may be stored inside item 310 or inside another item referenced by item 310. This attribute may be a region identifier attribute (e.g., a material identifier attribute from a G-PCC or V-PCC bitstream), which is a component attribute that associates a region identifier with each point of the volumetric media. Points with a common material identifier thus share a characteristic that can be used to identify an object or type of object.

[0097] 3D region annotation items 320 and 325 each contain a value that corresponds to a possible value of a region identifier attribute, shown as attribute value 421 and attribute value 426, respectively. The region corresponding to 3D region annotation item 320 or 325 consists of all points within volumetric media 310 whose region identifier attribute value is the same as the value specified within region annotation item 320 or 325, respectively.

[0098] The content syntax for a 3D Region Annotation item is: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(8)region_identifier_value; } } may be also possible.

[0099] The semantics of the 3D region annotation item are: -region_identifier_value is the value of the region identifier attribute of the volumetric media points contained in the region. - The other fields have the same semantics as in the previous embodiment.

[0100] In a variant, the region_identifier_value may correspond to a tile ID from the G-PCC tile inventory, or a tile ID from the V-PCC or V3C atlas.

[0101] In this embodiment, the annotated volumetric media from FIG. 2b may be stored in an ISOBMFF file having the following structure: FileTypeBox 'ftyp': major-brand = 'gpci', compatible-brands = 'mif1' MetaBox'meta': (container) HandlerBox 'hdlr': 'volv' PrimaryItemBox 'pitm': item_ID = 1; ItemInfoBox 'iinf': entry_count = 4 'infe': item_ID = 1, item_type = 'gpe1'; / / (Refer to 250) 'infe': item_ID = 2, item_type = '3dan', hidden = true; / / (Refer to 260) 'infe': item_ID = 3, item_type = '3dan', hidden = true; / / (Refer to 261) 'infe': item_ID = 4, item_type = '3dan', hidden = true; / / (Refer to 270) ItemLocationBox 'iloc': item_count = 1 item_ID = 1, extent_count = 1, extent_offset = X, extent_length = Y; item_ID = 2, extent_count = 1, extent_offset = O1, extent_length = L1; item_ID = 3, extent_count = 1, extent_offset = O2, extent_length = L2; item_ID = 4, extent_count = 1, extent_offset = O3, extent_length = L3; ItemReferenceBox 'iref': referenceType = 'cdsc', from_item_ID = 2, reference_count = 1, to_item_ID = 1; referenceType = 'cdsc', from_item_ID = 3, reference_count = 1, to_item_ID = 1; referenceType = 'cdsc', from_item_ID = 4, reference_count = 1, to_item_ID = 1; ItemPropertiesBox 'iprp': ItemPropertyContainerBox 'ipco': 1) 'gpcC' / / Annotated volumetric media configuration 2) 'gpsr' / / Size of the annotated volumetric media 3) 'udes', lang = en, name = 'person', description = ", tags = person 4) 'udes', lang = fr, name = 'Jean', description = ", tags = person 5) 'udes', lang = fr, name = 'Notre Dame', description = 'Notre Dame de Paris', tags = building ItemPropertyAssociation 'ipma': entry_count = 4 1) item_ID = 1, association_count = 2 essential = 1, property_index = 1; essential = 0, property_index = 2; 2) item_ID = 2, association_count = 1 essential = 0, property_index = 3; 3) item_ID = 3, association_count = 2 essential = 0, property_index = 3; essential = 0, property_index = 4; 4) item_ID = 4, association_count = 1 essential = 0, property_index = 5; MediaDataBox 'mdat' or 'idat': Volumetric media (at file offset X, with length Y) / / Contains the region identifier attribute value Region Annotation (at file offset O1, with length L1) region_identifier_value = ri0; Region Annotation (at file offset O2, with length L2) region_identifier_value = ri1; Region Annotation (at file offset O3, with length L3) region_identifier_value = ri2;

[0102] A specific region identifier attribute value, e.g., 0, may be reserved for points that do not belong to any region. A default value for the region identifier attribute value may be set to a value that corresponds to points that do not belong to any region.

[0103] In some cases, the region identifier attribute may be a set of bits, with each bit corresponding to a different region. For example, a first region may be associated with a value of 1, a second region may be associated with a value of 2, a third region may be associated with a value of 4, etc. This allows a point to be specified as belonging to multiple regions. For example, a point may have a region identifier attribute value of 3 to indicate that it belongs to both a first and a second region.

[0104] In some cases, a 3D region annotation item may contain, for each region, a list of values ​​for the region_identifier_value field. This makes it possible to specify that several regions may intersect. For example, a first region may be associated with values ​​1 and 2, a second region with values ​​2 and 3. Points that belong only to the first region have a value of 1 for its region identifier attribute. Points that belong only to the second region have a value of 3 for its region identifier attribute. Points that belong to both regions have a value of 2 for its region identifier attribute.

[0105] In this variation, the syntax of the content of the 3D region annotation item is: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(8) value_count; for (i=0; i < value_count; i++) { unsigned int(8) region_identifier_value; } } } may be also possible.

[0106] In some cases, the 3D region annotation item may contain a range of values ​​per region. This makes it possible to take into account that the encoding of the region identifier attribute may be lossy. In some cases, the 3D region annotation item may contain one or more ranges of values ​​for each region. In some cases, different attributes may be used for each region. The possible values ​​for each attribute are Boolean.

[0107] In some cases, the default value of these attributes may be false to specify that points do not belong to the corresponding region by default. This default value may be specified using the mechanism for specifying default attribute values. This default value may be specified by the attribute's definition.

[0108] In this variation, the syntax of the content of the 3D region annotation item is: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { oid(v) attribute_oid; } } may be also possible.

[0109] The semantics of the 3D region annotation item are: - attribute_oid is an identifier of the attribute used to define the region. This attribute_oid field is represented using an ASN.1 object identifier. In some cases, other representations may be used to identify the attribute used to define the region.

[0110] In some cases, the 3D region annotation item may contain a list of object identifiers for each region, allowing multiple regions to be combined, or the object identifiers may be universally unique identifiers (UUIDs).

[0111] In some cases, the region identifier attribute may be a predefined attribute identified using the "attr_label" from the G-PCC. The region identifier attribute may also be identified using an object identifier in the G-PCC. It may also be identified using an attribute type from the V3C or V-PCC.

[0112] In some cases, region identification may reuse existing attributes, e.g. opacity, reflectance, transparency, data ID...

[0113] In some cases, the identification of the region may use a new attribute that may also be used to represent another type of information. This new attribute may correspond to the type of object to which the point belongs, or the object part to which the point belongs. For example, the attribute may indicate that the point belongs to a car or a person. This attribute may also correspond to the object instance to which the point belongs. For example, the attribute may indicate that the point belongs to the first car or the second car, or the first building.

[0114] FIG. 5 illustrates a third embodiment of the invention. In this embodiment, the geometry of a region is described in another item, typically a volumetric media item. In this embodiment, a 3D region annotation item 320 is linked to a volumetric media item 511 via a reference of type "3drg" 533. This reference goes from the 3D region annotation item to the volumetric media item 511. The region is defined by the volumetric media contained in the item referenced in the "3drg" link from the 3D region annotation item. In some cases, the direction of the reference may be reversed. Similarly, a volumetric media item 512 defines a region corresponding to a 3D region annotation item 325.

[0115] If the volumetric media item 511 corresponds to a point cloud encoded, for example, using G-PCC or V-PCC, then any point from the volumetric media item 310 that is located at the same position as a point from item 511 belongs to the region corresponding to the 3D region annotation item 320.

[0116] The encoding of the geometry of the volumetric media may involve degradation. As a result, decoding the same position from the volumetric media of item 310 and from the volumetric media of item 511 may result in slightly different values. To ensure that a point from the volumetric media of item 310 matches a corresponding point in the volumetric media of item 511, a comparison of their respective decoded positions may be realized with a tolerance. If the distance between the two positions is equal to or less than the tolerance, the point from the volumetric media of item 310 is considered to belong to the region. This tolerance may be predefined as an absolute value, for example 1 mm. This tolerance may be predefined to a fraction of the maximum dimension of the volumetric media of item 511, for example 1 / 1000. This tolerance may be set to the sum of the maximum position encoding errors of the two volumetric media. This tolerance may also be specified inside the 3D region annotation item 320.

[0117] This tolerance may be used to define a region as any portion of the volumetric media item 310 that is located within a sphere centered on the volumetric media point of the item 511, with a radius equal to the tolerance value.

[0118] If the volumetric media 511 corresponds to visual volumetric media that has been encoded using V3C, for example, then any portion of the volumetric media item 310 that is located inside the volumetric media from item 511 belongs to the region corresponding to the 3D region annotation item 320.

[0119] For point clouds, encoding the geometry of the volumetric media may be lossy, therefore a tolerance may be applied to the definition of the interior volume of the volumetric media 511. Using this tolerance, this region may be defined as any part inside the volume of the volumetric media 511 or within a distance from the surface of the volumetric media less than or equal to the tolerance value.

[0120] A 3D region annotation item may be linked with several volumetric media items. This link may be realized by a single "3drg" reference. This link may be realized by several "3drg" references.

[0121] The content syntax for a 3D Region Annotation item is: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(field_size) offset_x; unsigned int(field_size) offset_y; unsigned int(field_size) offset_z; unsigned int(field_size) tolerance; } } may be also possible.

[0122] The semantics of the 3D region annotation item are: offset_x, offset_y, and offset_z define the relative position of the volumetric media that defines the region inside the volumetric media containing the media. In some cases, these fields are not present and the coordinate system of the volumetric media that defines the region and the volumetric media containing the region is used directly. These fields may be optional. Tolerance is the tolerance value for comparing the location of a point from the volumetric media that defines the region with a point from the volumetric media to which the region is applied. Each tolerance value corresponds to the area represented by a volumetric media item at the same position in the list of items linked via a "3drg" reference from the 3D area annotation item.

[0123] The other fields have the same semantics as in the previous embodiment.

[0124] In a variation of this embodiment, the boundary of the region may be defined by the convex hull of points from the volumetric media item referenced by the 3D region annotation item.

[0125] In a variant, a tolerance may be used using a Manhattan-like distance, and the region may be defined as any portion of the volumetric media item 310 that has a size equal to the tolerance and is located within a cube centered on the volumetric media point of the item 511. In some cases, a rectangular prism may be used instead of a cube, using a tolerance of three.

[0126] In a variant, attributes from the media items 511 or 512 may be used as a confidence level that the location of the volumetric media item 310 belongs to a region. For example, a luminance or opacity value associated with a point of a media item 511 or 512 may be used to represent a confidence level that the point is inside a region defined by the media item.

[0127] In a variant, the region annotation item 320 or 325 describes a coarse geometry of the region, e.g. using the geometric description described in the first embodiment, and the geometry described in the associated volumetric media item 511 or 512 provides a finer description of the region's geometry. This finer geometry can be seen as a masking of the "coarse region" described in the region annotation item.

[0128] FIG. 6 shows the main steps of a process for adding a new 3D region annotation to a volumetric media item stored in an ISOBMFF file when the 3D region annotation is described by the item, according to an embodiment of the present invention. These steps can be applied to ISOBMFF files stored on disk, in memory, or stored with an adaptive representation in memory. The new 3D region annotation includes the geometry of the region and the annotation itself. In some cases, these steps may be modified to add several annotations to a region of a volumetric media item simultaneously. In some cases, these steps may also be modified to add annotations to several regions of a volumetric media item simultaneously.

[0129] This process can be used when creating a new ISOBMFF file or when modifying an existing ISOBMFF file.

[0130] The first step 600 determines whether a 3D region annotation item with the same geometry already exists in the ISOBMFF file.

[0131] If a 3D region annotation item with the same geometry already exists, the next step is step 610 , otherwise the next step is step 620 .

[0132] In step 610, the item_ID value that corresponds to an existing 3D region annotation item is selected. The next step is step 640.

[0133] In step 620, a new 3D region annotation item is created to represent the region. An "infe" box describing the 3D region annotation item may be created in the "iinf" box of the ISOBFMM file. An entry in the "iloc" box may be added to indicate the location of the 3D region annotation item's content. An item_ID value is associated with this new 3D region annotation item.

[0134] In the case of the first or fourth embodiment, the geometry of the region is stored inside the content of the 3D region annotation item.

[0135] For the second embodiment, the value of the region identifier attribute corresponding to the region is stored within the content of the 3D region annotation item. If the volumetric media with which the region is associated does not have a region identifier attribute, the value of the region identifier attribute may be encoded and associated with the volumetric media item with which the region is associated. If the volumetric media has a region identifier attribute and a new 3D region annotation item is associated with a new value of this region identifier attribute, the region identifier attribute value may be decoded and updated with values ​​corresponding to the new region of points within this new region, and these values ​​may be re-encoded and associated with the volumetric media item.

[0136] For the third embodiment, the tolerance associated with the region is stored inside the content of the 3D region annotation item. A new volumetric media item corresponding to the geometry of the region may be created, added to the ISOBMFF file, and associated with the new 3D region annotation item.

[0137] The new 3D region annotation item is then associated with the volumetric media item in step 630. A new reference of type "cdsc" is created in the "iref" box of the ISOBMFF file. This reference associates the 3D region annotation item with the volumetric media item.

[0138] The next step is step 640.

[0139] In step 640, it is determined whether an item property corresponding to the annotation already exists in the ISOBMFF file. If the item property already exists, the next step is step 650; otherwise, the next step is step 660.

[0140] If an annotation is stored in an item, for example if the annotation is some XMP data or if the annotation is a 2D or 3D image, then it is determined whether an item corresponding to the annotation already exists in the ISOBMFF file. If the item already exists, the next step is step 650, otherwise the next step is step 660.

[0141] In step 650, an existing item property or item is selected.

[0142] In step 660, a new item property is created to represent the annotation. The type of the item property depends on the content of the annotation. The information contained in the annotation is stored in the item property.

[0143] If the annotation is stored in an item, then in the same step 660, a new item is created to represent the annotation. The type of the item depends on the content of the annotation. The information contained in the annotation is stored inside the item content, e.g. in an "mdat".

[0144] After either step 650 or step 660 , the next step is step 670 .

[0145] In step 670, an item property or item is associated with the 3D region annotation item.

[0146] If the 3D area annotation item already has an associated entry in the "ipma" box, the item property's index is added to this entry.

[0147] If the 3D area annotation item does not have an associated entry in the "ipma" box, a new association is created for this item and the item property index is set to this new association.

[0148] If the annotation is stored inside an item, then if an item reference with the appropriate type already exists, the list of references is updated with a reference to this new item, otherwise a new item reference with the appropriate type is created between the 3D area annotation item and the new item.

[0149] In some cases, a 3D region annotation item may contain multiple regions, meaning the same annotation is associated with multiple different regions, and multiple steps may be modified. In step 600, it is determined whether a 3D region annotation item exists that contains a single region with the same geometry. In step 620, it is determined whether an existing 3D region annotation item is associated with a volumetric media item that has a set of properties that correspond to the annotation of the new 3D region annotation item. In this case, the geometry of the new region is added to the existing 3D region annotation item and steps 640-670 are not performed. If not, a new 3D region annotation item may be created and a reference added accordingly.

[0150] For the fourth embodiment, multiple item properties or items may be created in step 660 or selected in step 650 .

[0151] FIG. 7 shows the main steps of a process for reading an ISOBMFF file containing 3D region annotations when the region annotations are described by items, according to an embodiment of the invention.

[0152] In a first step 700, the volumetric media items are extracted from the files. In some cases, only a portion of the metadata describing the volumetric media items is extracted.

[0153] In step 710, a first item that is different from the volumetric media item is extracted from the file. If there are no other items in the file, the algorithm proceeds directly to step 770.

[0154] Next, in step 720 it is determined whether the other item is a 3D region annotation item. If it is, the next step is step 730; if not, the next step is step 750.

[0155] In step 730, it is determined whether a 3D region annotation item is associated with a volumetric media item by a reference of type "cdsc" in the "iref" box. If a 3D region annotation item is associated with a volumetric media item by a reference of type "cdsc" in the "iref" box, the next step is step 740; otherwise, the next step is step 750.

[0156] In step 740, item properties associated with the 3D area annotation item via the entry in the "ipma" box are extracted. Items associated with the area annotation item are also extracted. In some cases, only item properties or items associated with the 3D area annotation item are extracted, or neither of them are extracted.

[0157] The geometry of the 3D area annotation items is then extracted.

[0158] In the context of the first or fourth embodiment, geometry is extracted from the content of the 3D region annotation item.

[0159] In the context of the second embodiment, a value of a region identifier attribute corresponding to a region is extracted from the 3D region annotation item. Points from the volumetric media item whose region identifier attribute value matches this extracted value are then extracted. These extracted points define the geometry of the region.

[0160] In the context of the third embodiment, from the 3D region annotation items, the items referenced through the "3drg" reference are extracted. These items define the geometry of the region.

[0161] The region with the extracted item properties and the region with the extracted items are associated with the volumetric media item. Information contained in the item properties or items may be extracted and associated with the geometry of the region for the volumetric media item.

[0162] If the 3D region annotation item contains several regions, each of these regions with extracted item properties and extracted items are associated with a volumetric media item. Information contained in the item properties or items may be extracted and associated with the geometry of each of these regions to the volumetric media item.

[0163] The next step is step 750.

[0164] In step 750, it is determined whether there are other items in the ISOBMFF file that are different from the volumetric media items. If there are other items in the ISOBMFF file that are different from the volumetric media items, the next step is step 760;

[0165] In step 760, the other items are extracted from the ISOBMFF file.

[0166] In step 770, the process ends.

[0167] FIG. 8 illustrates a process for processing an ISOBMFF file that includes volumetric media and one or more 3D region annotation items associated with the volumetric media, according to an embodiment of the present invention. The process may be a volumetric media rendering process. The process may be a volumetric media editing process. The process may be a process for modifying the content of the volumetric media, such as applying a filter or drawing on the volumetric media. The process may be a metadata editing process. The process may be a process for removing private metadata. The process may be a process for filtering metadata. The process may be a process for translating metadata, or any other process that manipulates files.

[0168] In a first step 800, the process is applied to a volumetric media item.

[0169] In some cases, the volumetric media associated with the volumetric media item may be modified. In some cases, the result of this process may be stored in another volumetric media item as a derived volumetric media item. In this case, the 3D region annotation items associated with the original volumetric media item may also be associated with this derived volumetric media item. If the result of the process is stored as a derived volumetric media item, in subsequent steps the processed volumetric media item is a derived volumetric media item.

[0170] In step 810, a first region annotation associated with the processed volumetric media item is retrieved. If no region annotations are associated with the processed volumetric media item, the next step is step 870.

[0171] In step 820, it is determined whether a region annotation should be removed. Depending on the process, different references may be used. The process may remove all region annotations. The process may remove any region annotations that have a particular type. For example, a privacy preservation filter may remove any region annotations that are represented by user-defined item properties. The process may remove region annotations depending on their location.

[0172] If it is determined that the region annotation should be removed, the next step is step 825. If not, the next step is step 830.

[0173] In step 825, the region annotations are removed. The next step is step 850.

[0174] In step 830, it is determined whether the geometry of the region annotation should be modified. Any process that transforms the geometry of the volumetric media should modify the geometry of the region annotation in an appropriate manner.

[0175] If it is determined that the geometry of the region annotation should be modified, the next step is step 835. If not, the next step is step 840.

[0176] In step 835, the geometry of the region annotation is modified according to the process. In some cases, the modified geometry is an exact result of applying a geometric transformation to the geometry of the region annotation. In some cases, the modified geometry is an approximate result of applying a geometric transformation to the geometry of the region annotation.

[0177] The next step is step 840.

[0178] In step 840, it is determined whether the annotation of the region annotation should be modified. Depending on the process, different references may be used. A process that translates a text annotation may modify the text that represents the annotation. A process that filters volumetric media may modify the annotation, for example by applying a blur, to remove a precise portion from the annotation.

[0179] If it is determined that the annotation of the region annotation should be modified, the next step is step 845. Otherwise, the next step is step 850.

[0180] In step 845, the annotations of the region annotation are modified according to the process.

[0181] For example, in Figure 2b, region 261 has a region annotation that corresponds to a description of a person. A privacy preservation process may retain an indication that the region annotation corresponds to a person, but may remove the person's name.

[0182] The next step is step 850.

[0183] In step 850, it is determined whether there are more region annotations to process. If it is determined that there are more region annotations to process, the next step is step 860. If not, the next step is step 870.

[0184] In step 860, another region annotation associated with the volumetric media item is retrieved. The next step is step 820.

[0185] The process ends in step 870.

[0186] 9a and 9b show a fourth embodiment of the present invention. In this embodiment, the content of the annotation may be split into several chunks. For example, the annotation may be a burst of CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) images associated with a volumetric media item. In this embodiment, a 3D region annotation item 210 may be associated with an entity group 930 that describes the content of the region. This association may be achieved using an item reference of type "eroi" from the 3D region annotation item 210 to the entity group 930. The entity group 930 may be of type "sbst" to indicate a spatial burst of images. An entity group of type "sbst" called spatial burst of images groups entities related via spatial location. This entity group includes image items that are part of the spatial burst, including image items 931, 932, and 933.

[0187] A spatial burst of images does not have any specific timing information about the entities contained in the group. For example, a spatial burst of images may be cross-sections of an object captured at regular intervals, such as CT or MRI images. A spatial burst of images may also be images taken at regular distance intervals on a path traveled by a sensor. A spatial burst of images may also be a video of a cross-section of an object captured at regular intervals.

[0188] The geometry of a 3D region corresponding to 3D region annotation item 210 may be composed of a set of geometries, where each geometry in the set corresponds to a chunk of annotation content. For example, in Figure 9b, the geometry of the region is represented as a set of plane 961, plane 962, ..., plane 963. Each image item included in entity group 930 corresponds to a chunk of annotation content. Plane 961 may be associated with image item 931, plane 962 may be associated with image item 932, and so on.

[0189] The syntax of the content of the 3D area annotation item may be expressed as an extension of the syntax used in the first embodiment. aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(8) geometry_type; … if (geometry_type == 5) { / / Volume as a discrete set of planes unsigned int(8) plane_count; unsigned int(field_size) plane_gap; signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) normal_x; signed int(field_size) normal_y; signed int(field_size) normal_z; } } }

[0190] The semantics of the 3D region annotation item are: x, y, z are fixed-point numbers that specify the coordinates of a point in the reference plane. normal_x, normal_y, normal_z are fixed-point numbers that specify the normal vector of the reference plane. plane_count is the number of planes in the set of planes. plane_gap is the distance between two planes in the set.

[0191] The other fields have the same meaning as in the first embodiment.

[0192] For a set of planes, the first plane is the reference plane. Then each plane is a distance of "plane_gap" from the previous plane in the direction of the normal vector.

[0193] In some cases, the x, y, z fields may specify a point not on a reference plane, but at the centre of a set of faces.

[0194] In this embodiment, each plane is associated with an image item from entity group 930. The planes may be ordered within the set of planes starting from the reference plane. The image items included in entity group 930 may be ordered within the entity group according to their enumeration order within the entity group definition. Planes ordered by index i may be associated with image items ordered by index i.

[0195] In this embodiment, other sets of geometries may be used, for example a set of circles or a set of cuboids. Other types of items or entities may be associated with a set of geometries, for example an entity group that groups volumetric media items, an entity group that groups audio tracks, an entity group that groups auto exposure bracketing groups. In some cases, a set of geometries may be associated with a set of properties, for example a 3D region annotation item may have several associated "udes" item properties, each corresponding to a geometry inside the set of geometries.

[0196] In a variation of this embodiment, spatial burst entity group 930 is directly associated with entity 200, for example using an item reference of type "3drg" or type "eroi". In this variation, the geometry of a region from entity 200 associated with an item included in entity group 930 may be specified in an item property associated with that item. The geometry of a region from entity 200 associated with an item included in entity group 930 may be specified in an item associated with the entity group or in an item property associated with the entity group.

[0197] In some cases, annotations of any type may be directly associated with entity 200, for example using item references of type "3drg" or type "eroi." In this variation, the geometry of an area from entity 200 associated with an item included in entity group 930 may be specified in item properties associated with that item.

[0198] 10 shows a fifth embodiment for annotating regions in a volumetric media track. In this embodiment, the annotations are stored in the "meta" portion of the media file 1001 while the track description is in the "moov" portion of the media file 1000. In this embodiment, the media file is extended with region annotations where the annotations are stored in the "meta" box 1001 at the file level.

[0199] In this embodiment, the volumetric media is stored as track 1000-1. The geometry of the region may be stored as samples 1000x in timed metadata track 1000-2. The timed metadata track 1000-2 may be associated with the volumetric media track 1000-1 using a track reference type set to "cdsc" (for content description). This timed metadata track may be called the region annotation track.

[0200] Optionally, the timed metadata track may also be associated with another volumetric media track 1000-3, or a video or image sequence track, or an audio track that provides a description of the regions of interest described in this timed metadata track. This may be indicated by an "eroi" track reference type.

[0201] Timed metadata tracks containing samples containing the geometry of the region may be identified with a sample entry having a specific four-letter code, for example "3dan". As an alternative or in addition to a new sample entry type, a specific track reference may be used between the timed metadata track 1000-2 and the volumetric media track 1000-1. For example, instead of using a generic "cdsc" track reference, a track reference type set to "3dan" for "3D region annotation" may be used between the timed metadata track 1000-2 and the volumetric media track 1000-1 from which the region was identified or extracted.

[0202] The particular track reference type can help a media file reader interpret the timed metadata track 1000-2.

[0203] The geometry of the region may be stored inside the samples of the timed metadata track 1000-2 using the same syntax as in any of the previous embodiments. In some cases, fields corresponding to the reference_size of the region annotation may be stored in a sample entry box instead of being stored inside the sample itself. They may also be defined as data structures such as 3DPoint or Vector3D providing x, y, z coordinates. The syntax of such a sample entry is: aligned(8) class 3DRegionTrackConfigBox () extends FullBox ('3rgC', version=0, flags=0) { unsigned int (7) reserved = 0; unsigned int (1) field_length_size; unsigned int((field_length_size + 1) * 16) reference_size_x; unsigned int((field_length_size + 1) * 16) reference_size_y; unsigned int((field_length_size + 1) * 16) reference_size_z; } aligned(8) class RegionSampleEntry extends MetadataSampleEntry ('3dan'){ 3DRegionTrackConfigBox config; / / Required } may be also possible.

[0204] As with the third embodiment, volumetric media defining the geometry of the region may be stored in a track and associated with the region annotation track 1000-2 using a track reference of type "3drg". Volumetric media may also be stored in an item (not shown) and associated with the region annotation track 1000-2 using a sample group providing SampleToMetadataItemEntry (from the ISOBMFF specification).

[0205] Note that the timed metadata track 1000-2 may contain fewer samples than the volumetric media track 1000-1 it describes. For example, a sample 1000x corresponding to a volumetric media sample may not exist when the position and size of the region of interest remain stable over time, or when the position or size may be interpolated.

[0206] In a first variant, the region annotation is declared as an item property of type "udes" (e.g. in an ItemPropertyContainerBox "ipco" 1002) and the track 1000-2 providing the region geometry contains a sample group 1040 providing a SampleToMetadataItemEntry (from the ISOBMFF specification). In other words, a group of 2dcc samples from the track 1000-2 may be associated with one or more item properties of type "udes", as indicated by the arrows 1020 or 1030. The item_ID in the SampleToMetadataItemEntry is set to the implicit ID of the property in the "ipco" container box 1002. In fact, the "ipco" box implicitly defines an identifier that corresponds to the location of the item property in the "ipco" box. Several groups of samples may be linked to the same item property providing annotations for the region. Some item properties that provide annotations or user descriptions may not be referenced by samples from the timed metadata track 1000-2 (e.g., because they are also used for other volumetric media items declared in the media file.) The sample group 1040 may be a default group when all samples that describe the geometry of a region have the same annotation.

[0207] In a second variant, a new grouping type is defined to explicitly indicate that the ID used in the sample group entry 1040 corresponds to an identifier of an item property, indicating that a sample is explicitly associated with an item property, not with an item. The syntax of this new grouping type may be as follows (4cc and the name of the sample group entry are taken as examples here): class SampleToItemPropertyEntry() extends SampleGroupDescriptionEntry('stip') { unsigned int(32) property_type; unsigned int(32) meta_box_handler_type; unsigned int(32) num_properties counts; for(i = 0; i < num_properties counts; i++) { unsigned int(32) property_index[i]; } }

[0208] property_type is an optional parameter indicating the 4cc corresponding to the type of property the sample is associated with. If property_index is present, only properties of the specified type may be counted. If not present (or by default), property_index is the 1-based index of the associated property box in the ItemPropertyContainerBox 1002 contained in the same ItemPropertiesBox (counting all boxes including FreeSpace boxes). The value 0 may for example be reserved to indicate no association with any property. This may be signaled by not mapping samples or NAL units to the property.

[0209] The meta_box_handler_type may specify the type of metadata schema used by MetaBoxes referenced by items in this sample group.

[0210] If there are multiple MetaBoxes with the same handler type, the MetaBox referenced in this sample group entry is the first one that satisfies one of the following ordering constraints: - A MetaBoxhandler_type contained in the current track that has handler_type equal to meta_box_handler_type. - A MetaBox contained in a MovieBox with handler_type equal to meta_box_handler_type. - A MetaBox contained at the root level of the file with handler_type equal to meta_box_handler_type.

[0211] num_properties counts the number of item properties that this sample group references.

[0212] property_index[i] specifies the 1-based index (counting all boxes including FreeSpace boxes) of the item property box within the ItemPropertyContainerBox contained in the ItemPropertiesBox that applies to or is valid for the samples mapped to this sample group description entry.

[0213] In a third variant, the annotations may be stored in another media stored in a track (e.g. the ROI volumetric media track 1000-3), which may be a volumetric media track, a video track, a metadata track or an audio track. The relationship between this track 1000-3 and the timed metadata track 1000-2 describing the geometry of the region may use a track reference of type "eroi".

[0214] All the embodiments and variants described in relation to the items may also be applied to a truck based on this fifth embodiment.

[0215] Annotations may correspond to objects detected in the volumetric media by an object detection tool. This may be represented using user-described item properties, for example using specific values ​​for the name and / or tag fields and / or using a more descriptive value for the description field. For example, the name field is "Building" and the description field is "House, Building, Monument". This may be represented with a new item property.

[0216] The annotation may correspond to a particular object instance detected in the volumetric media by an object detection tool. This may be represented using user-described item properties, for example using a specific value to describe the generic type of object in a tag field, using a more precise value corresponding to the object instance in a name field, and / or using a descriptive value for the object instance in a description field. For example, the tag field may be "Church", the name field may be "Notre Dame" and the description field may be "Notre Dame de Paris". This may be represented with new item properties.

[0217] An annotation may be the GPS location of an object in the volumetric media. This may be represented with a new item property, for example: aligned(8) class ItemGPSLocationProperty extends ItemFullProperty('igps', version = 0, flags = 0) { signed int(32) viewpoint_gpspos_longitude; signed int(32) viewpoint_gpspos_latitude; signed int(32) viewpoint_gpspos_altitude; } where viewpoint_gpspos_longitude gives the longitude of the geographic location of the viewpoint in units of 2-23 degrees. viewpoint_gpspos_longitude is in the range -180*223 to 180*223-1 inclusive. Positive values ​​represent east longitudes and negative values ​​represent west longitudes.

[0218] viewpoint_gpspos_latitude indicates the latitude of the geographic location of the viewpoint, in 2-23 degrees. viewpoint_gpspos_latitude is in the range -90*223 to 90*223-1, inclusive. Positive values ​​represent latitudes north, negative values ​​represent latitudes south.

[0219] viewpoint_gpspos_altitude indicates the altitude of the geographic position of the viewpoint in millimeters above the WGS 84 reference ellipsoid specified in the EPSG:4326 database.

[0220] Note that this property may be associated with an image or volumetric item and may indicate where the image or volumetric data was captured.

[0221] Annotations may describe edits or modifications applied to a region of volumetric media. Annotations may be represented by user-written item properties. Annotations may be represented by new item properties.

[0222] The annotation may be stored in an item, which may be associated with the 3D region annotation item by a reference of type "cdsc". The item may be associated with the 3D region annotation item property via a new item property associated with and referencing the 3D region annotation item, which has a box type of "rgcd". For example, the annotation may be stored in an item of type "Exif". As another example, the annotation may be stored in an XMP document with content type "application / rdf+xml" contained in an item of type "mime".

[0223] An annotation may be another media stored in an item. An annotation may be another volumetric media item or an image item. An annotation may be an entity group. An annotation may be another media stored in a track. An annotation may be a volumetric media track, a video track, or an audio track. The relationship between a region and other media stored in an item or track may be specified via an association between the 3D region annotation item and the other item or track. For example, the relationship between a region and an item or entity group may use an item reference of type "eroi".

[0224] In some cases, when a user-written item property is used, the language field is set to an appropriate value. In some cases, when a user-written item property is used, several instances of this item property are used with different language field values.

[0225] In all embodiments, some fields may be removed or renamed, some fields may be added, some parameters, e.g. x, y, z, may be collected in a data structure and replaced by this data structure, e.g. 3DPoint or Vector3D.

[0226] In all embodiments, the encoding of one or more fields may be changed, in particular the size of the field and / or its type may be changed.

[0227] In all embodiments, the 4cc used may have different names, or may reuse existing names where appropriate.

[0228] The coordinates of the region may be expressed using different reference systems: they may use the external coordinates of the volumetric media with which the region is associated, they may use the cryptographic coordinates of the volumetric media with which the region is associated, or they may use another coordinate system associated with the volumetric media.

[0229] The coordinates of the region may use another coordinate system defined with reference to one of those systems. This coordinate system may be specified using a transform from or to one of the coordinate systems associated with the volumetric media. This transform may include a translation, rotation, scaling, or any other affine transformation.

[0230] In some cases, the 3D region annotation item may include several fields to indicate which coordinate system is used and / or which transformation is used from or to this coordinate system.

[0231] In some cases, the same coordinate system may be used for all regions described in a 3D region annotation item. In some cases, different coordinate systems may be used for different regions described in a 3D region annotation item.

[0232] In some cases, the coordinates of a region may be specified relative to a portion of the volumetric media it is associated with, for example it may be specified relative to a tile or slice of the volumetric media.

[0233] In some cases, the coordinates of the region may be specified relative to a portion of another volumetric media.

[0234] For this variation, the syntax of the 3D region annotation item may be updated as follows: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(8) geometry_type; unsigned int(1) entity_relative; unsigned int(1) slice_relative; unsigned int(1) tile_relative; unsigned int(5) reserved; if (entity_relative == 1) { unsigned int(8) entity_id; } if (slice_relative == 1) { unsigned int(8) slice_id; } if (tile_relative == 1) { unsigned int(8) tile_id; } … } }

[0235] The semantics of this variant of the 3D region annotation item are: item_relative set to true indicates that the region's coordinates are relative to the coordinates of the entity indicated by entity_id. Otherwise, the region's coordinates are expressed using the global coordinate system. slice_relative set to true indicates that the region's coordinates are relative to the bounding box of the slice indicated by slice_id. Otherwise, the region's coordinates are expressed using the global coordinate system. tile_relative set to true indicates that the region's coordinates are relative to the bounding box of the tile indicated by tile_id. Otherwise, the region's coordinates are expressed using the global coordinate system.

[0236] In some cases only slice-relative or tile-relative coordinates may be included in a 3D region annotation item.

[0237] In some cases, slice and tile related fields may be common to all regions of a 3D region annotation item.

[0238] For the third embodiment, the same coordinate system may be used for both the annotated item 310 and the volumetric media that defines the annotated region 511 or 512. In some cases, a different coordinate system may be used and may be specified within the 3D region annotation item 320 or 325. In some cases, a transformation between the coordinate system of the annotated item 310 and the coordinate system of the volumetric media item 511 or 512 may be specified within the 3D region annotation item 320 or 325, respectively.

[0239] In some cases, the coordinates of the region may be specified as integer, floating point, or fixed point. In some cases, the type used for the coordinates may be specified within the 3D region annotation item.

[0240] The reference_size parameters are optional. They may provide the original size of the edited volumetric media. They can aid in the use of region annotations associated with the edited volumetric media.

[0241] The field_size parameter is optional. The size of the field depending on the field_size parameter may be predefined or it may only depend on the version of the box describing the area annotation.

[0242] The orientation of the region may be specified as a quaternion. In some cases, only the x, y, and z values ​​of the quaternion may be specified, and the w value of the quaternion is calculated as follows:

number

[0243] In some cases, the orientation of a region may be specified as a vector and an angle. In some cases, the orientation of a region may be specified as two vectors. In some cases, the orientation of a region may be specified as a matrix. In some cases, the orientation of a region may be specified as three angles corresponding to roll, pitch, and yaw.

[0244] In some cases, the orientation may not be defined. In some cases, the orientation may be defined in a simpler way.

[0245] When possible, the orientation of a region may be specified by listing at least four points that correspond to the vertices of a rectangular prism or the vertices of the bounding box of an ellipsoid.

[0246] In some cases, a rectangular solid or an ellipsoid may be specified as follows: The point that defines the center of a rectangular prism or ellipsoid. The first vector defines the "width" direction and dimension of the cuboid or ellipsoid. A second vector defining the "length" direction and dimension of the cuboid or ellipsoid. - The "height" dimension of a rectangular prism or ellipsoid.

[0247] In some cases, the plane may be bounded. The dimensions may be specified by vectors defining a first direction, a length along this direction, and a width along a perpendicular direction. The dimensions may also be specified by vectors defining a first direction, a length along this direction, and a width along a perpendicular direction.

[0248] Other geometries for regions may be defined, such as polygon-based, vertexed pyramids, cylinders, flat shape extrusions, triangular meshes, Platonic solids, etc. The geometry of a region may be any two-dimensional shape.

[0249] In some cases, the geometry of a region may be defined as a combination of geometries using constructive solid geometries. Possible combinations may include the union, difference, or intersection of two shapes.

[0250] In a variation of the first embodiment that uses constructive geometry operations, the syntax of the 3D region annotation item is: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(1) is_region unsigned int(7) reserved; unsigned int(8) geometry_type; if (geometry_type == 0) { / / Point … } if (geometry_type == 1) { / / Polyline … } } if (geometry_type == 2) { / / plane … } if (geometry_type == 3) { / / Cuboid … } if (geometry_type == 4) { / / Ellipsoid … } if (geometry_type == 5) { / / join unsigned int(8) first_geometry_index; unsigned int(8) second_geometry_index; } if (geometry_type == 6) { / / intersection unsigned int(8) first_geometry_index; unsigned int(8) second_geometry_index; } if (geometry_type == 7) { / / difference unsigned int(8) first_geometry_index; unsigned int(8) second_geometry_index; } } } may be also possible.

[0251] The semantics of this 3D region annotation item are: · is_region indicates whether the geometry corresponds to a region of volumetric media or whether it is only used to construct a region. · first_geometry_index and second_geometry_index indicate the index in the list of regions of the geometries that were joined by the constructive solid geometry operation. In the case of a union, the resulting geometry is the union of the two specified geometries. In the case of an intersection, the resulting geometry is the intersection of the two specified geometries. In the case of a difference, the resulting geometry is the difference between the first and second geometries. In some cases, the first, second, or both specified geometries correspond to the result of another constructive solid geometry operation.

[0252] In this variation, different operations may combine three or more geometries.

[0253] This variation may be used in conjunction with other embodiments.

[0254] In a variation of the third embodiment, entity groups may be defined to combine volumetric media using constructive geometry operations. For example, a geometry union entity group of type "geou" may be used to define the geometry of a region as a union of two or more other geometries that are either volumetric media items or constructive geometry operation entity groups. Similarly, a geometry intersection entity group of type "geoi" may be defined and a geometry difference entity group of type "geod" may be defined.

[0255] Where possible, two or more of the above embodiments may be combined, which may be achieved by indicating the use of the second or third embodiment via a particular value of the geometry_type field.

[0256] In some cases, the first and second embodiments may be combined to define a region using geometry and attribute values ​​together, which may be defined as all points whose region identifier value matches the value defined for the region and that are inside the region's geometry.

[0257] For example, the syntax of the content of a 3D region annotation item or entity combining the first and second embodiments may be: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(8) geometry_type; if (geometry_type == 0) { / / ポイント signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; } if (geometry_type == 1) { / / ポリライン unsigned int(field_size) point_count; for (i=0; i < point_count; i++) { signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; } } if (geometry_type == 2) { / / plane unsigned int(field_size) point_count; for (i=0; i < point_count; i++) { signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) normal_x; signed int(field_size) normal_y; signed int(field_size) normal_z; } } if (geometry_type == 3) { / / Cuboid signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) size_x; signed int(field_size) size_y; signed int(field_size) size_z; signed int(field_size) quaternion_w; signed int(field_size) quaternion_x; signed int(field_size) quaternion_y; signed int(field_size) quaternion_z; } if (geometry_type == 4) { / / Ellipsoid signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) size_x; signed int(field_size) size_y; signed int(field_size) size_z; signed int(field_size) quaternion_w; signed int(field_size) quaternion_x; signed int(field_size) quaternion_y; signed int(field_size) quaternion_z; } if (geometry_type == 5) { / / Region defined by attributes signed int(field_size)x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) size_x; signed int(field_size) size_y; signed int(field_size) size_z; unsigned int(8) region_identifier_value; } } } may be also possible.

[0258] The semantics of the 3D region annotation item are: For -geometry_type 5, the x, y, z and size_x, size_y, size_z fields define the bounding box of the region as a rectangular prism aligned on the axes of the coordinate system. The region_identifier_value is the value of the region identifier attribute of the volumetric media points contained in the region. Optionally, rotation information may be added to the bounding box description of the region in case of geometry_type5, for example using quaternions.

[0259] In some cases, the first and third embodiments may be combined to restrict the geometry defined by the volumetric media to a geometry specified for a region, which may be considered as a crop of the volumetric media.

[0260] For example, the syntax of the content of a 3D region annotation item or entity combining the first and second embodiments may be: aligned (8) class 3DRegionItem { unsigned int(8) version = 0; unsigned int(8) flags; unsigned int field_size = ((flags & 1) + 1) * 16; / / This is a temporary, unparseable variable: unsigned int(field_size)reference_size_x. unsigned int(field_size) reference_size_y; unsigned int(field_size) reference_size_z; unsigned int(8) region_count; for (r=0; r < region_count; r++) { unsigned int(8) geometry_type; if (geometry_type == 0) { / / Point signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; } if (geometry_type == 1) { / / Polyline unsigned int(field_size) point_count; for (i = 0; i < point_count; i++) { signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; } } if (geometry_type == 2) { / / Plane unsigned int(field_size) point_count; for (i = 0; i < point_count; i++) { signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) normal_x; signed int(field_size) normal_y; signed int(field_size) normal_z; } } if (geometry_type == 3) { / / Cuboid signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) size_x; signed int(field_size) size_y; signed int(field_size) size_z; signed int(field_size) quaternion_w; signed int(field_size) quaternion_x; signed int(field_size) quaternion_y; signed int(field_size) quaternion_z; } if (geometry_type == 4) { / / Ellipsoid signed int(field_size) x; signed int(field_size) y; signed int(field_size) z; signed int(field_size) size_x; signed int(field_size) size_y; signed int(field_size) size_z; signed int(field_size) quaternion_w; signed int(field_size) quaternion_x; signed int(field_size) quaternion_y; signed int(field_size) quaternion_z; } if (geometry_type == 5) { / / Region defined by another item unsigned int(field_size) offset_x; unsigned int(field_size) offset_y; unsigned int(field_size) offset_z; unsigned int(field_size) tolerance; } } } may be also possible.

[0261] The semantics of the 3D region annotation items or entities may be similar to those described in the previous embodiment.

[0262] In some cases, the second and third embodiments may be combined to allow a single volumetric media to define several regions. A region may be defined by points of the volumetric media associated with a 3D region annotation item whose region identifier values ​​match values ​​defined for regions inside the 3D region annotation item.

[0263] 11 is a schematic block diagram of a computing device 110 for implementing one or more embodiments of the present invention. Computing device 110 may be a device such as a microcomputer, a workstation, or a light portable device. Computing device 110 includes a communication bus connected to the following: a central processing unit 111, such as a microprocessor, referred to as a CPU; a random access memory 112, called RAM, for storing the executable code of the method of an embodiment of the invention, as well as registers adapted to record variables and parameters necessary for the implementation of the method according to an embodiment of the invention, the memory capacity of which can be expanded, for example by an optional RAM connected to an expansion port; a read-only memory 113, called ROM, for storing a computer program for implementing the embodiments of the invention; The network interface 114 is typically connected to a communication network over which the digital data to be processed is transmitted or received. The network interface 114 may be a single network interface or may be composed of a set of different network interfaces (e.g., wired and wireless interfaces, or different types of wired or wireless interfaces). Data packets are written to the network interface for transmission or read from the network interface for reception under the control of a software application executing in the CPU 111. A user interface 115 may be used to receive input from a user or to display information to a user. A hard disk 116, designated -HD, may be provided as mass storage device. - I / O module 117 may be used for receiving / transmitting data from / to external devices such as video sources or displays.

[0264] The executable code may be stored either in the read-only memory 113, in the hard disk 116 or on a removable digital medium, such as a disk. According to a variant, the executable code of the program may be received by the communications network, via the network interface 114, to be stored in one of the storage means of the communications device 110, such as the hard disk 116, before being executed.

[0265] The central processing unit 111 is adapted to control and direct the execution of instructions of a program or part of a software code according to an embodiment of the present invention, which instructions are stored in one of the aforementioned storage means. After power-on, the CPU 801 is capable of executing instructions from the main RAM memory 112 relating to a software application after they have been read from the program ROM 803 or the hard disk 116. Such a software application, when executed by the CPU 801, causes the steps of the flowcharts of the present invention to be performed.

[0266] Any step of an algorithm of the present invention may be implemented in software by execution of a set of instructions or programs by a programmable computing machine such as a PC ("Personal Computer"), a DSP ("Digital Signal Processor") or a microcontroller, or else in hardware by mechanical or dedicated components such as an FPGA ("Field-Programmable Gate Array") or an ASIC ("Application-Specific Integrated Circuit").

[0267] While the present invention has been described above based on specific embodiments, the present invention is not limited to the above embodiments, and various modifications are possible without departing from the gist of the present invention.

[0268] Furthermore, many further modifications and changes will be suggested to those skilled in the art with reference to the exemplary embodiments described above, which are given by way of example only and are not intended to limit the scope of the invention, which is determined solely by the appended claims. In particular, different features from the various embodiments may be interchanged where appropriate.

[0269] It should be noted that each of the above-described embodiments of the present invention may be implemented alone or in combination with other embodiments, and features from the various embodiments may be combined where necessary or where a combination of elements or features from the individual embodiments in a single embodiment is beneficial.

[0270] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. 1. A method of encapsulating volumetric media data in a file, the method comprising: generating a volumetric media entity (200) describing the volumetric media data; generating a 3D region annotation entity (210) related to a region of the volumetric media data, the 3D region annotation entity being associated with the volumetric media entity; generating geometry data (220) associated with the 3D region annotation entity to describe a geometry of the region of the volumetric media data; generating at least one annotation data structure (230, 231, 232, 931, 932, 933) associated with said 3D region annotation entity; embedding the volumetric media entities, the 3D region annotation entities, the geometry data, and the at least one annotation data structure in a file; A method comprising:

2. the volumetric media entity is a first item (310); the 3D region annotation entity is a second item (320, 325); the geometry data (321, 326, 411, 511, 512) being associated with the second item; The method of claim 1.

3. The method of claim 2 , wherein the at least one annotation data structure is a property of the second item.

4. The method of claim 2 , wherein the at least one annotation data structure is an item associated with the second item.

5. The method of claim 2 , wherein the geometry data (321, 326) is included in the second item.

6. The method of claim 2 , wherein the geometry data (411) is contained within the first item.

7. The method of claim 6 , wherein the geometric data is determined from identifiers associated with points of the volumetric media data.

8. The method of claim 2, wherein the geometry data (511, 512) is a third item.

9. The method of claim 2 , wherein the at least one annotation data structure is a group (930) of at least one item (931, 932, 933).

10. The method of claim 9 , wherein the at least one item represents a plane within the volumetric media data.

11. the volumetric media entity is a volumetric media track (1000-1), the volumetric media data being included in samples of the volumetric media track; the 3D region annotation entity is a 3D region annotation track (1000-2); said geometry data being contained in samples (1000x) of said 3D region annotation track (1000-2); The method of claim 1.

12. the volumetric media entity is a volumetric media track (1000-1), the volumetric media data being included in samples of the volumetric media track; the 3D region annotation entity is a 3D region annotation track (1000-2); the geometry data is included in samples (1000-2) of another track associated with the 3D region annotation track; The method of claim 1.

13. The method according to claim 11 or 12, wherein said at least one annotation is an item property associated with a group of samples of said 3D region annotation track (1000-2).

14. The method according to claim 11 or 12, wherein the 3D region annotation track (1000-2) is associated with a track (1000-3) providing a representation of the region of interest described in the 3D region annotation track (1000-2).

15. 1. A method for reading a file containing volumetric media data, comprising: reading a volumetric media entity (200) describing the volumetric media data to obtain the volumetric media data; reading a 3D region annotation entity (210) associated with a region of the volumetric media data, the 3D region annotation entity being associated with the volumetric media entity; reading geometry data (220) associated with the 3D region annotation entity to describe a geometry of a region of the volumetric media data; reading at least one annotation data structure (230, 231, 232, 931, 932, 933) associated with said 3D region annotation entity to obtain annotation data; processing the acquired volumetric media data and the acquired at least one annotation data according to the geometry of the region of the volumetric media data; A method comprising:

16. A computer program product for a programmable device comprising sequences of instructions for implementing the method of any one of claims 1 to 15, when said computer program product is loaded and executed on said programmable device.

17. A non-transitory computer readable storage medium storing computer program instructions for implementing the method according to any one of claims 1 to 15.

18. A computer program which, when executed, causes a method to be performed according to any one of claims 1 to 15.

19. 1. A device for encapsulating volumetric media data in a file, the device comprising: generating a volumetric media entity (200) describing the volumetric media data; generating a 3D region annotation entity (210) related to a region of the volumetric media data, the 3D region annotation entity being associated with the volumetric media entity; generating geometry data (220) associated with the 3D region annotation entity to describe a geometry of the region of the volumetric media data; generating at least one annotation data structure (230, 231, 232, 931, 932, 933) associated with said 3D region annotation entity; embedding the volumetric media entities, the 3D region annotation entities, the geometry data, and the at least one annotation data structure in a file; A device comprising a processor configured for:

20. 1. A device for reading a file containing volumetric media data, the device comprising: reading a volumetric media entity (200) describing the volumetric media data to obtain the volumetric media data; reading a 3D region annotation entity (210) associated with a region of the volumetric media data, the 3D region annotation entity being associated with the volumetric media entity; reading geometry data (220) associated with the 3D region annotation entity to describe a geometry of a region of the volumetric media data; reading at least one annotation data structure (230, 231, 232, 931, 932, 933) associated with said 3D region annotation entity to obtain annotation data; processing the acquired volumetric media data and the acquired at least one annotation data according to the geometry of the region of the volumetric media data; A device comprising a processor configured for:

Images