Method and apparatus for encapsulating annotation regions in an ISOBMFF track

Abstract

The present invention relates to a method for generating a media file, the method comprising: generating a first track of a media data sample; generating at least one annotation area, the annotation area being associated with a geometry of the annotation area, the annotation area being associated with at least one area annotation describing the annotation area, wherein the annotation area is associated with the first track of the media data sample; and generating a media file comprising the first track and the at least one annotation area, the associated geometry and the associated area annotation.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for encapsulating annotation areas in media tracks. Background Art

[0002] Modern cameras and image analysis services are able to generate metadata for an image or image sequence (e.g., annotated regions) associated with a portion, sub-portion, or even an object within the image or image sequence. For example, a camera can generate a focus region of a video or detect faces while recording the video. As another example, a deep learning system can detect objects within a moving image sequence.

[0003] The images or videos captured by the camera or processed by the image analysis service are stored on a storage device such as a memory card. The recorded video is usually compressed to reduce the size of the data on the storage device. Many video compression standards can be used, such as those defined by the MPEG committee: AVC, HEVC or VVC.

[0004] Once compressed, the video can be stored or transmitted over a communication network for consumption by a media player. For example, a compressed video sequence can be encapsulated using the ISO Base Media File Format (ISOBMFF) to produce an ISO Base Media File or a collection of ISOBMFF segments. Some video sequences (e.g., bursts of images) can also be stored as image sequences or image tracks using image file formats (e.g., HEIF or MIAF).

[0005] The International Organization for Standardization Base Media File Format (ISOBMFF, ISO / IEC 14496-12) is a well-known flexible and extensible file format that encapsulates and describes encoded timed or non-timed media data for local storage or for transmission via a network or via another bitstream delivery mechanism. An example of an extension is ISO / IEC 14496-15, which describes encapsulation tools for various NAL (network abstraction layer) unit-based video coding formats. Examples of such coding formats are AVC (Advanced Video Coding), SVC (Scalable Video Coding), HEVC (High Efficiency Video Coding), L-HEVC (Layered HEVC), and VVC (Versatile Video Coding). Another example of a file format extension is ISO / IEC 23008-12, which describes encapsulation tools for still images or still image sequences (such as HEVC still images, etc.). Another example of a file format extension is ISO / IEC 23090-2, which defines the Omnidirectional Media Application Format (OMAF). The ISO base media file format is object-oriented. It consists of building blocks called boxes, corresponding to a data structure characterized by a unique type identifier (usually a four-character code, also called FourCC or 4CC). A full box is a data structure similar to a box that additionally contains version and flag value attributes. In the following, the term box can specify both a full box or a box. These boxes or full boxes are organized hierarchically or sequentially in the ISOBMFF file and define parameters that describe the encoded timed or non-timed media data, its structure and timing (if any). All data in the encapsulated media file (media data and metadata describing the media data) is contained in the box. There is no other data in the file. File-level boxes are boxes that are not contained in other boxes.

[0006] In the file format, the entire media presentation is called a movie. The movie is described by a movie box (with the four-character code 'moov') at the top level of the file. The movie box represents an initialization information container containing a collection of various boxes that describe the media presentation. It is logically divided into tracks represented by track boxes (with the four-character code 'trak'). Each track (uniquely identified by a track identifier (track_ID)) represents a timed sequence of media data belonging to the presentation (e.g., frames of video or audio samples). Within each track, each timed unit of data is called a sample; this can be a frame of video, audio, or timed metadata. Samples are implicitly numbered in a decoding order sequence. Each track box contains a hierarchy of boxes that describe the samples of the track, for example, the sample table box ('stbl') contains all the time and data indexes of the media samples in the track. The actual sample data is stored in a box called a media data box (with the four-character code 'mdat') or an identified media data box (with the four-character code 'imda', similar to the media data box but containing additional identifiers) at the same level as the movie box. Animations can also be segmented and organized in time as an animation box, which contains information for the entire presentation, followed by a list of media fragments, i.e., a list of several animation fragments and media data boxes ('mdat' or 'imda'). Within an animation fragment (a box with the four-character code 'moof'), there is a collection of track fragments (boxes with the four-character code 'traf') that describe the tracks within the media fragment, with zero or more track fragments per animation fragment. A track fragment in turn contains zero or more track run boxes ('trun'), each of which records a consecutive run of samples for that track fragment.

[0007] An ISOBMFF file may contain multiple coded timed media data or sub-parts of coded timed media data forming multiple tracks. When a sub-part corresponds to one or continuous spatial portion of a video source (taken over time (e.g., at least one rectangular region taken over time, sometimes referred to as a 'tile' or 'sub-picture')), the corresponding multiple tracks may be referred to as tile tracks or sub-picture tracks.

[0008] ISOBMFF and its extensions include several grouping mechanisms for grouping tracks, static items or samples together and associating group descriptions with them. Groups typically share common semantics and / or properties.

[0009] These encapsulation standards do not provide an easy means of associating descriptions or annotations with regions within an image or within a sequence of images in an interoperable manner. "Easy means" should be interpreted here as being processable by a generic ISOBMFF reader or a standard reader based on ISOBMFF. For example, ISOBMFF does not provide a simple box or data structure for user annotations or text descriptions. There is a box for user data, but it is a container box for vendor-specific metadata and is therefore not dedicated to interoperability.

[0010] The Web Video Text track format (WebVTT), a candidate recommendation from the W3C, provides captions or subtitles for video content, and also provides text video descriptions, chapters for content navigation, and more generally any form of metadata that is time-aligned with the audio or video content. The association between a video file or media file and a WebVTT file can be signaled in HTML, but there is no specific relationship between WebVTT tracks and media tracks within the ISOBMFF file. Summary of the invention

[0011] The present invention has been designed to solve one or more of the aforementioned problems. The present invention more particularly relates to a solution for describing an annotation region (possibly a moving region) in a video track. According to an embodiment, the description of the annotation region may be stored in its own track and associated with the media track it describes. The description of the annotation region may be stored in its own track and associated with a metadata track describing the media track. The description of the annotation region may be stored in a specific part of the media file, such as a 'meta' box or a sample description. The description of the annotation region may be associated with a track, a sample group or a group of sub-samples.

[0012] According to a first aspect of the present invention, a method for generating a media file is provided, the method comprising:

[0013] - generating a first track of media data samples;

[0014] - generating at least one annotation region, the annotation region being associated with a geometry of the annotation region, the annotation region being associated with at least one region annotation describing the annotation region; wherein the annotation region is associated with a first track of a media data sample; and

[0015] - generating a media file comprising said first track and said at least one annotation region, associated geometry and associated region annotations.

[0016] In the examples:

[0017] - define the annotation area as an item associated with the first track using the item reference frame;

[0018] - defining said region annotation as an item attribute associated with an item defining said annotated region.

[0019] In the example:

[0020] - define the annotation area as an item associated with the first track using the item reference frame;

[0021] - said region annotation is defined as an item property directly or indirectly associated with said first track.

[0022] In an embodiment, the method further comprises:

[0023] - generating a second track of timed metadata, the second track of timed metadata comprising annotation regions stored as samples, the second track being associated with the first track using a track reference.

[0024] In the example:

[0025] - Region annotations are stored as properties of the sample group of the second track.

[0026] In an embodiment, the method further comprises:

[0027] - generating a third track of samples, each sample being a region annotation associated with an annotation region of the second track, the third track being associated with the second track using a track reference.

[0028] In the example:

[0029] -Store region annotations as properties of a property container in the metadata section of a media file.

[0030] In the example:

[0031] - Annotation regions are associated with region annotations via sample groups in the second track.

[0032] In the example:

[0033] - an association between the region annotation and the annotation region is stored in a sample group description entry indicating an index of the region annotation in the attribute container, the sample group description entry being associated with the sample group in the second track.

[0034] In an embodiment, the first track includes a comment region stored in the sample group description entry, and includes a region comment stored as an attribute of a sample storing the comment region.

[0035] In an embodiment, the method further comprises:

[0036] - generating a first sample grouping indicating an annotation region; and

[0037] - generating a second sample grouping directly or indirectly indicating the region annotation.

[0038] In an embodiment, the second sample grouping associates a sample or a group of samples with a region annotation stored as a property in a property container box in a metadata portion of the media file.

[0039] In an embodiment, the second sample grouping is a default sample group that associates all samples with a region annotation stored as a property in a property container box in the metadata portion of the media file.

[0040] In an embodiment, the region annotation comprises a text annotation.

[0041] According to another aspect of the invention, there is provided a computer program product for a programmable device, the computer program product comprising a sequence of instructions for implementing the method according to the invention when loaded into and executed by the programmable device.

[0042] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores instructions of a computer program for implementing the method according to the present invention.

[0043] According to another aspect of the present invention, there is provided a computer program which, when executed, causes the method of the present invention to be performed.

[0044] According to another aspect of the present invention, there is provided an apparatus for generating a media file, the apparatus comprising a processor, the processor being configured to:

[0045] - generating a first track of media data samples;

[0046] - generating at least one annotation region, the annotation region being associated with a geometry of the annotation region, the annotation region being associated with at least one region annotation describing the annotation region; wherein the annotation region is associated with a first track of a media data sample; and

[0047] - generating a media file comprising said first track and said at least one annotation region, associated geometry and associated region annotations.

[0048] At least a portion of the method according to the invention may be computer-implemented. Thus, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may be collectively referred to herein as "circuits," "modules," or "systems." Furthermore, the invention may take the form of a computer program product embodied in any tangible medium of expression having a computer usable program code embodied in the medium.

[0049] Since the present invention can be implemented in software, the present invention can be embodied as computer readable code for providing to a programmable device on any suitable carrier medium. Tangible non-transitory carrier media may include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices, or solid-state memory devices. For example, transient carrier media may include signals such as electrical signals, electronic signals, optical signals, acoustic signals, magnetic signals, or electromagnetic signals (e.g., microwave or RF signals). BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0051] FIG1 shows modules of an exemplary apparatus for implementing the present invention;

[0052] Figure 2 A high level view showing the invention for associating annotations with regions of an image or sequence of images;

[0053] Figure 3 A first embodiment is shown in which a note area is described as an item associated with a track;

[0054] Figure 4 A first variation of the second embodiment is shown in which the annotation area is described within a metadata track describing the area location or size;

[0055] Figure 5 A second embodiment of the present invention is shown in which a comment area is described in its own track;

[0056] Figure 6 A third embodiment showing that the annotation area is stored in the 'meta' section of the media file;

[0057] Figure 7 A fourth embodiment showing that the annotation area is stored in the media track;

[0058] Figure 8 A variation of the fourth embodiment is shown where the media file has only a 'moov' part.

[0059] Fig. 9 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. DETAILED DESCRIPTION

[0060] FIG. 1 shows a module showing an exemplary device of the present invention. It should be noted that the present invention can be implemented in any type of computing device capable of manipulating video files, and is not limited to the device described herein. The device can be a recording device such as a camera. The device can have communication capabilities, such as a network camera with a wired or wireless connection. The device can be used as a media server using the communication module. For example, device 105 is configured to encapsulate a video stream with an annotation area using the present invention. As shown in the figure, device 105 records a video source through camera 100, generating an image sequence that can be compressed by encoding module 105. Object recognition module 110 can be used to identify and locate one or more objects that can be tracked through an image sequence to define an area of ​​interest. These objects can be determined by image analysis after or before encoding (as shown by the dotted lines). The detected objects can move in the image over time, and therefore potentially overlap, appear or disappear at a certain point in time. It should be noted that before transmitting the video over a communication network, an object recognition or detection step can be performed offline, or during encoding an image initiated by a video source and transmitting the encoded image, an object recognition or detection step can be performed online. The detection may be handled by an external device, with which the device 105 then communicates through the communication module, which is responsible for the content analysis for object identification. The identified or detected objects on the recorded image may be associated with the entire image or with a partition within the image (e.g., NAL unit, slice, block, sub-picture, ...). As shown, the encoded image is encapsulated in a media file or in a media segment in the encapsulation module 115 before possible transmission using the communication module 120. Transmission may occur when a client requests the device server 105 to download a media file or requests the device server to adaptively stream a file.

[0061] Figure 1b are examples of objects detected on the image 150. Each object may be described with a text annotation, for example: flower, star, person, ...

[0062] Figure 1c 155 are examples of objects detected on an image 155 with corresponding picture partitions (e.g., strips, blocks, or sub-pictures represented by dashed lines). For some detected objects, the corresponding images or image sequences are available as media tracks (155-1, 155-2, or 155-3). The images in the media track consist of one or several parts of a complete image sequence.

[0063] Figure 1d is an example of a moving region of interest (165-1, 165-5, and 165-10) in the image sequence 160 (eg, a recognized face). Figure 1dAbove, picture partitions 170 are also depicted. The moving region consists of image portions that overlap the object. The image portions that make up the moving region of interest and their number may vary over time to follow the object in the image.

[0064] Figure 2 A high-level view of the invention for associating annotations with regions of an image or image sequence is shown. An annotation region 210 is linked to an entity 200. An entity here is, for example, an image item describing an image. The entity can also be an item of another class, an item group, a track or a sample within a track, or even a subsample within a sample of a track. The link between the annotation region and the entity can depend on the entity and be an item reference, an item attribute, a track reference, a sample description, or a subsample description according to different embodiments.

[0065] The annotation area 210 may be defined by geometries 220, such as anchor points and shapes. Several kinds of geometries may be defined: points, rectangles, circles, polygons, binary masks, ... Other geometries may also be used, such as 3D boxes projected onto the image plane, etc. The geometry may change over time: anchor points or shapes or both.

[0066] The annotation area 210 may also contain or may be linked to one or more annotations 230, 231, and 232. These annotations are used to store information describing or corresponding to the area 210 of the entity 200. The annotation may be, for example, the focus position of the picture, a face detected in the image, an object detected in the image, a GPS location corresponding to an object in the image, a description of a portion of the image, text contained in an area of ​​the image... The annotation may be user-generated or machine-generated, preferably generated as one or more null-terminated strings. A text annotation is defined as an annotation comprising one or more strings. The text annotation may be human-readable. One of the strings may define the text language in another or other string. As for geometry, a given area may have annotations that change over time depending on the media content. For example, the text annotation may be described using the item attribute of the user description 'udes' defined in HEIF.

[0067] Figure 3A first embodiment is shown in which an annotation region is described as an item (region item 320) associated with a track. When receiving or extracting object information, the encapsulation module may create a 'meta' box in the file while encapsulating a picture track or a video track 310 to declare the region item 320 in an ItemInfoEntry box. The track 310 may be a picture track (e.g., identified by a 'pict' handler type, such as a HEIF image sequence track) or a video track (e.g., identified by a 'vide' handler type, such as a HEVC or VVC track stored in an ISOBMFF or ISOBMFF-based specification such as HEIF, MIAF, or any application format). The region item 320 has a specific item type and an item_ID that does not conflict with any other entity in the ID of other items or tracks or media files. The region item 320 is associated with the track using an item reference box. The item reference type may be a generic 'cdsc' item reference type indicating that the region item provides some description of the track 310. Alternatively, the item reference type may be a more specific type, such as 'rgan', to indicate that the region item provides annotation region information. Using a more specific item reference type, a more precise indication of the relationship between the track and region items is provided. This is an indication to the parser or media player on how to handle the item. The region item can be associated with one or more item attributes 340 (such as any HEIF item) (for example, using the 'ipma' box 345). For example, an item attribute can provide an anchor point. Another attribute can provide the geometry of the region. A single item attribute can provide both position and shape. Alternatively, the anchor point or geometry is located in the data portion of the region item, in the data portion of the file (for example, an 'mdat' or 'idat' box) and indexed by the ItemLocationBox. Another item attribute representing a text annotation can be a 'udes' item attribute that provides a title or description or label of the region. Optionally, a metadata item 350 can provide a further description of the region item 320. For example, XMP metadata or MPEG-7 descriptors or vendor-specific metadata (for example, in the 'uuid' box) can provide more detailed information about the region. The detailed information may not be human-readable. The format of these additional metadata may depend on the format generated by the object recognition module 110. This is indicated by the dashed arrow 355. Furthermore, when video or image content is available for a region item 320, the item may contain an item reference 335 to that video or image item (referred to as an ROI image item 330), thereby providing a representation of the region. For this purpose, an item reference of type 'eroi' (for "encoded region of interest") may be used. It should be noted that 330 may be a single image item, for example a picture of a person detected on a video track. 330 may also be a collection of alternative image items (different pictures of that detected person, grouped into 'alter' entities for grouping).330 may also be an image sequence track or a video track showing faces of people along the sequence in track 310. ROI image item 330 and track 310 may have some relationship (not shown) indicated in the media file.

[0068] For this embodiment, the item reference 315 and the optional item reference 335 require that the 'to_item_ID' field of the ItemReferenceBox also reference a track or track group (e.g., an alternative track set) in addition to the item. The semantics of the to_item_ID becomes: the to_item_ID contains the item_ID or group_id or the track_ID or track_group_id of the entity to be grouped of the item, entity group, track or track group, respectively, being referenced. This is relevant in media files that mix items and tracks and use unique IDs (as can be indicated by the 'unif' class in the file's ftyp box). This can be a simple way to associate item properties with tracks. The extended semantics of the ItemReferenceBox allows any item in the 'meta' box of a media file to be associated with a track in the 'moov' section of the media file.

[0069] In a variation, to provide an even more direct association of item attributes to tracks, the 'ipma' box may be extended to allow reference to a track ID in addition to an item ID or entity ID. This may be useful for media files containing picture or video tracks and image items: some attributes associated with an image item may be associated with this mechanism for all samples of a video or picture track. Such an extended 'ipma' box may be indicated by a new version or flag value of the box, or when a particular class (e.g., the 'unif' class) is used. Although the annotation area is described as an item allowing simple annotation areas for an image sequence or video track, it is limited to "static annotation areas". A static annotation area is an annotation area that does not change (e.g., is unchanging) along the entire track 310. The following embodiments allow dynamic annotation areas.

[0070] Figure 4A first variation of the second embodiment is shown, in which region annotations are stored in a track different from the media track (describing a video or image sequence). In this embodiment, a first video or image sequence track 410 containing samples is described. A second track 420, a timed metadata track 420, is dedicated to storing the annotation region of the video track as samples 420x. The timed metadata track is associated with the video track, where the track reference type is set to 'cdsc' (415, for content description). As an example of a timed metadata track, the '2dcc' track from MPEG-B Part-10 can be used. The samples of the '2dcc' track provide the anchor points and sizes (e.g., having the same decoding time) of the rectangular region of interest in the corresponding samples of the first track 410. Optionally, the timed metadata track can also be associated with another video or image sequence track that provides a representation of the region of interest described in the timed metadata track. This can be indicated by the 'eroi' track reference type as defined in MPEG-B Part-10. In order to distinguish the timed metadata track defined in MPEG-B Part-10 from the timed metadata track according to the present embodiment, a specific track reference may be used between the timed metadata track having the annotation 420 and the media track 410. For example, instead of a general 'cdsc' track reference type, a track reference type set to 'rgan' for 'region annotation' may be set between the timed metadata track 420 and the video track that identifies or extracts the region.

[0071] It should be noted that the timed metadata track 420 may not contain as many samples as the video track 410 it describes. For example, when the position and size of the region of interest remain stable over time (e.g., Figure 1d 421-1, 421-2, ...) or when the position or size can be interpolated, there may not be a sample 420x corresponding to a video sample. In this embodiment, the timed metadata track 420 is extended with a new sample group that provides annotations (421-1, 421-2, ...) for the area described in the sample 420x. In a first variant, a new grouping_type 'udes' 421 is proposed to associate a sample group with a title, description or tag. A special UserDescriptionSampleGroupEntry is defined to inherit from SampleGroupDescriptionEntry (e.g., inherit from TextSampleGroupEntry) to allow the following parameters to be indicated:

[0072]

[0073] has the following semantics:

[0074] lang (language) is a string containing an RFC5646-compliant language tag string, such as "en-US", "fr-FR", or "zh-CN", etc., indicating the language of the text contained in the name, description, and tags. When lang is empty, the language is unknown / undefined.

[0075] name is a null-terminated UTF-8 string containing a human-readable name for the item or entity group. If absent (the empty string is supplied), no name is provided.

[0076] description is a null-terminated UTF-8 string containing a human-readable description of the item or entity group. If not present (the empty string is supplied), no description is provided.

[0077] tags is a null-terminated UTF-8 string containing comma-separated user-defined tags associated with the item. If not present (the empty string is supplied), no tags are provided.

[0078] It should be noted that in this variant, since the user description sample group entry provides the same parameters as the 'udes' item attribute from HEIF, the same 4cc is used for the grouping type. This allows the player to reuse the same parsing code for this type of sample group or item attribute. In the alternative, especially if only a single text parameter is preferred for describing the comment area ('rdes'), a different 4cc may be used, for example as follows:

[0079]

[0080]

[0081] Same semantics as previous variant.

[0082] For these variants, a new sample group entry for region annotations ('rdes' or 'udes' or any other dedicated 4cc) allows a sample group to be associated with a user-defined name (e.g., title, name, description, or tag); there may be multiple such attributes, which should have different language codes (different values ​​of their lang parameter). When several instances of a new sample group entry for region annotations ('rdes' or 'udes' or any other dedicated 4cc) are associated with the same sample group, they represent alternatives that may be expressed in different languages, and the reader should choose the most appropriate. For different attributes in the track selection box, these alternatives can be distinguished by setting the grouping_type_parameter to, for example, the value 'lang' or reusing the 'mela' four-character code for "Media language". At most one new sample group entry for region annotations ('rdes' or 'udes' or any other dedicated 4cc) with the same lang value should apply to the same sample group.

[0083] This embodiment can even be applied to static areas by defining the 'udes' or 'rdes' sample groups as static.

[0084] In a variation of this embodiment (where region annotations are stored in tracks providing the location and size of the region), the annotations (420-1, 420-2, ...) may be stored in newly defined samples 420x, rather than being declared as sample groupings. For example, a new sample entry type '2dca' may be defined as follows:

[0085] 2D Cartesian Coordinates with annotation "2dca" sample entry provides spatial information related to the referenced track expressed in a two-dimensional Cartesian coordinate system, optionally with a textual description of the region (human readable). Such a sample indicates the region geometry and an optional annotation, for example as an additional string parameter which may be an empty string. The length of the string is determined by the sample size indicated in the appropriate ISOBMFF box: A 2D Cartesian Coordinate sample may be defined as follows:

[0086] aligned(8)class 2DCartesianCoordinatesSample{

[0087] unsigned int(16)top_left_x;

[0088] unsigned int(16)top_left_y;

[0089] unsigned int(16)width;

[0090] unsigned int(16)height;

[0091] unsigned int(1)interpolate;

[0092] unsigned int(7)reserved;

[0093] utf8string annotation;

[0094] }

[0095] top_left_x and top_left_y give the horizontal and vertical coordinates, respectively, of the top left corner of the rectangular area associated with the media samples of the referenced track. The width and height give the width and height, respectively, of the rectangular area associated with the media samples of the referenced track. Interpolation indicates the temporal continuity of consecutive samples. When true, the application can linearly interpolate the values ​​of the ROI coordinates between the previous sample and the current sample. When false, there will be no interpolation of the values ​​between the previous sample and the current sample, and the annotation is a null-terminated UTF-8 string containing a human-readable description of the ROI. If not present (an empty string is supplied), no description is provided. When the interpolation flag is set, the interpolated sample is expected to match the presentation time of the sample in the referenced track. For example, for each video sample of the video track, an interpolated 2D Cartesian coordinate sample is calculated. For annotations, the same annotation is repeated in the interpolated sample.

[0096] Figure 5 A second embodiment of the present invention is shown. In this embodiment, the region annotation is in a different (metadata) track than the media track. The first video or image sequence track 510 contains samples. These samples can be samples based on NAL units. The second track 520 (timed metadata track) is dedicated to storing the region position and size of the video track 510 as samples 520x. The timed metadata track is associated with the video track, where the track reference type is set to 'cdsc' (515, for content description). The second track 520 can be associated with the 'eroi' track reference type for another media track (not shown) that provides an encoded version of the region of interest. The third track 530 is created by the encapsulation module to provide regional annotations as samples 535x. The track 530 is associated with the second track 520 that provides the region position and size using a new track reference type dedicated to the indication of the region annotation 'rgan' 545. When parsing the track reference 345, the parser or media player is notified that the third track 530 provides annotations for the region defined in the second track 520.

[0097] The new track reference type allows the encapsulation module and the parser to describe and identify the track as a track dedicated to storing annotations for regions, respectively.

[0098] The handler type for such a third track 530 can be set to 'text' to indicate that the sample contains a text annotation and the associated decoder will only process text data. The text annotation can be user-generated or machine-generated. It describes an area identified in or extracted from a video or image sequence track. The text annotation in the sample of such a track can be accompanied by a parameter indicating the language of the text description. As an alternative, the language is indicated in the media header box (language field). When a text description in an alternative language is provided for a region, the former is preferred. The samples for the text description can be aligned or unaligned with the samples of the second track or the first (media) track. When not aligned, the text description is applied until a new sample providing a new text description is defined in the third track. The sample duration is set accordingly to indicate these durations. For segmented files, it is recommended to provide a text description sample for the first sample of the fragment. The payload of the sample is a null-terminated string (when there is no language) or includes two null-terminated strings (when the language is provided within the sample). The sample size in the appropriate ISOBMFF box (e.g., 'stsz' or 'stz2') provides the length of these strings.

[0099] According to this embodiment, a media file may contain (or be referenced by a data reference box 'dref') multiple "third tracks" that provide annotations for regions described in a second track. Each of these third tracks has a track reference of type 'rgan', which indicates that each track provides annotations for regions described in the referenced second track. A set of third tracks may be declared as an alternative track group. In this case, only one track should be selected and used to present the region annotations on the media player interface. This may be the case, for example, when the third track provides an alternative language for the region annotations. This may be further indicated in a track selection box with a distinguishing attribute set to the media language 'mela'. There may be cases where a set of third tracks that provide annotations for regions described in a second track is quite complementary rather than alternative. For example, there may be a first third track that provides user annotations (such as a title), a second third track that provides machine-generated tags, and a third track that provides even more detailed information, such as a WebVTT track (providing annotations but also providing rendering instructions) or an MPEG-7 track. Then, some third tracks (e.g., containing structured data) may have a handler type set to 'meta' instead of 'text'. In this case, the parser or media player can select and present different annotations on the client interface. The client can provide means in the GUI for the user to adapt or select the level of annotation he or she wants to see. In a variation, the specific track reference type between the track that defines the region position and size and the track that provides the annotation can be from track 520 to track 530. This is relevant for parsers because if they decide not to process the track that describes the region position and size, they may not be interested in the annotation track either. Instead, when processing track 520 following a specific track reference 545, the track or track group that provides the annotation will be obtained. When there are multiple alternative or supplementary "third" tracks, it is of interest to have a track reference 545 in this direction, the track reference directly refers to the alternative track group in the track_ID parameter.

[0100] Figure 6 A third embodiment is shown where annotation regions are stored in a 'meta' section 601 of a media file 601 while the track description is located within a 'moov' section of the media file 600. In this embodiment, region annotations stored within the file level 'meta' box 601 are used to enhance the media file.

[0101] In this embodiment, the locations and sizes of the regions are stored in the timed metadata track 600-2. Optionally, a video or picture track 600-3 provides an encoded representation of some regions.

[0102] In a first variant, the region annotation is declared as an item attribute of type 'udes' (e.g., in an 'ipco' box), and track 600-2 provides the region position and size (e.g., the geometry of the region), containing a sample grouping 640 that provides a SampleToMetadataItemEntry (from the ISOBMFF specification). In other words, a group of 2dcc samples from track 600-2 can be associated with one or more item attributes of type 'udes', as indicated by arrows 620 or 630. The item_ID in the SampleToMetadataItemEntry is set to the implicit ID of the attribute in the 'ipco' container box. In fact, the 'ipco' box implicitly defines an identifier corresponding to the position of the item attribute in the 'ipco' box. Several sample groups can be linked to the same item attribute that provides an annotated region. Some item attributes that provide annotations or user descriptions may not be referenced by samples from the timed metadata track 600-2 (e.g., because they are used for other image items also declared in the media file). The sample grouping 640 can be the default grouping when all samples that describe the position and size of the region have the same annotation.

[0103] In the second variation, in order to explicitly indicate that the ID used in the sample group entry 640 corresponds to the identifier of the item attribute, a new grouping type is defined to indicate that the sample is not actually associated with the item but is explicitly associated with the item attribute. The syntax of the new grouping type may be as follows (here, 4cc and the name of the sample group entry are taken as examples):

[0104]

[0105]

[0106] property_type is an optional parameter corresponding to 4cc indicating the type of property associated with the sample. When present, property_index can count only properties of the specified type. When not present (or default), property_index is a 1-based index of the associated property box in the ItemPropertyContainerBox contained in the same ItemPropertiesBox (counting all boxes, including FreeSpace boxes). The value 0 can be reserved as it will indicate no association with any property. This can be signaled by not mapping samples or NAL units to properties.

[0107] According to a fourth embodiment, the video or picture track itself contains or refers to the annotation area.

[0108] exist Figure 7In the first variant depicted, the media file has a 'moov' section describing a video or picture track 701 and a 'meta' section 710 providing a region annotation description (e.g., as an item attribute (711-1 or 711-4) declared in an 'ipco' box 711). The media track 701 contains NALU-based samples, each sample representing an image (or a partition of an image, such as a strip, tile, or sub-picture). The track can be segmented over time. The video track 701 contains two sample groups 702 and 703 in the sample table box. The first sample group 702 provides the location and size of the region of interest identified or extracted by the object recognition module 110. For example, the sample group description entry 702-1 provides a description of the region identified within the sample group 701-1. For this purpose, the encapsulation module can use the existing 'trif' sample group or any sample group that provides the location and shape of the image region. The second sample group 703 provides annotations for the region described by the first sample group 702. As an example, the sample group description entry 703-1 provides a reference (720) to the annotation of the sample group 701-1. The grouping type may be a 'stmi' or 'stpi' sample grouping on a video or picture sample as described in the previous embodiments. The second sample group associates the samples or sample groups of the video or picture track to one or more item attributes in the 'meta' box. As an example, the second sample group 703 associates the first sample group 701-1 with an item attribute of type 'udes' 711-1, and the second sample group 703 also associates the sample group 701-3 with the item attribute 711-4. It should be noted that for the region annotation, the sample group is mapped to the 'udes' item attribute, but the same applies when mapping the samples of the media track to any kind of item attribute. NAL units (with NALU mapping mechanism), the samples or sample groups of the media track can be mapped to the item attributes via the 'stmi' or 'stpi' sample group (such as 703). In this case, the first sample group 702 becomes optional because it is dedicated to the region. Furthermore, in the case of the default sample grouping, this can be an easy way to associate an item property with all samples of a track. This can be Figure 3 Embodiments or ItemReferenceBox are also alternatives to extensions of the reference track. The two sample groups between 702 and 703 may or may not be aligned. When aligned, this means that each region described in the first sample group is associated with an annotation attribute. When not aligned, this means that one annotation attribute may be applied to different regions described in the first sample group or one region in the first sample group may have several annotations. When grouping_type allows, the two sample groups may use the attributes of the sample group, such as default, static, or virtual sample grouping. The first sample group and the second sample group may also be used for samples, but may also be used for NAL units through the NALU mapping mechanism.

[0109] In a variant, the video track contains sample groups to associate groups of samples with region items such as 320. This means that both the geometry and annotations of the regions detected or extracted from the media track 701 are described in the meta part of the media file 710. This would avoid stating that the first sample group 702 provides the positions and sizes of these detected or extracted regions of interest.

[0110] exist Figure 8 In another variation depicted, the media file 800 has only a 'moov' section. The media track is self-contained or self-describing, meaning that it has no dependencies on item properties or additional tracks. In this variation, the media track 800 has a sample group mechanism for describing region annotations. Figure 8 An example of two different sample groups 802 and 803 is depicted, but there may be a single sample group that merges the attributes of the two sample group description entries from 802 and 803. This merge may be of interest when aligning the sample groups in 802 and 803, thereby saving the description bytes for one SampleToGroupBox and one SampleToGroupDescriptionBox. This is the case when the same sample group has the same region annotation and the same region description (e.g. position, size, ...) for the duration of the sample group. Return to Figure 8, a media track 801 contains a first sample group of a grouping_type providing the location and size of a region (e.g., a 'trif' sample group 802) and a second sample group 803 of a grouping_type providing a textual description of the sample group (e.g., 'udes' or 'rdes' from the previous embodiments). As an example, sample group 801-1 may be associated with a first sample group description entry 802-1 and a second sample group description entry 803-1. Although the figure shows the same sample group mapped to two different sample group description entries, the sample groups mapped to 802-1 and 803-1 may be different. This keeps the annotation region simple, as all information is stored in a single track, especially for media streaming use cases, thereby avoiding transmission delays and synchronization issues in receiving data from different tracks. The limitation is that while new sgpds of type trif (or type of location and size of a region) may be defined, the roi or region location may not be based on sample movement, but only on fragment movement. This embodiment also allows annotations based on subsamples (e.g., ranges of NAL units, slices, tiles, or sub-pictures) when the first sample-to-group box first maps samples to NAL unit patterns via a NALUMapEntry, and the NALU MapEntry maps NAL units to a sample group description box containing a new sample group entry (e.g., 'rdes' or 'udes' from the previous embodiment, or any other dedicated 4cc). In a variant where the annotation relates to a full image, the first sample group 802 describing the region becomes optional, and only the second sample group 803 can be present in the file. In addition, when used with the NALU mapping mechanism, and when the annotation corresponds to a sample or a partition in an image (e.g., a slice, tile, or sub-picture) and these partitions correspond to one or more NAL units, the first sample group can be optional. If the media track 801 contains samples corresponding to partitions of an image (e.g., an HEVC tile track or a VVC sub-picture track), the annotation can be associated with samples or groups of samples of such a track via the second sample group 803, making the first sample group 802 optional.

[0111] Fig. 9 is a schematic block diagram of a computing device 900 for implementing one or more embodiments of the present invention. The computing device 900 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 900 includes a communication bus connected to:

[0112] - a central processing unit 901 denoted as CPU, such as a microprocessor or the like;

[0113] - A random access memory 902 marked as RAM, used to store executable code of the method according to the embodiment of the present invention and registers suitable for recording variables and parameters, which are necessary for implementing the method according to the embodiment of the present invention. The memory capacity of the random access memory can be expanded, for example, by connecting to an optional RAM of the expansion port;

[0114] - a read-only memory 903 marked as ROM, used to store a computer program for implementing an embodiment of the present invention;

[0115] - A network interface 904, which is usually connected to a communication network through which digital data to be processed is sent or received. The network interface 904 may be a single network interface, or may be composed of a collection of different network interfaces (e.g., wired and wireless interfaces, or different kinds of wired or wireless interfaces). Under the control of a software application running in the CPU 901, data packets are written to the network interface for sending, or data are read from the network interface for receiving;

[0116] - User interface 905 may be used to receive input from a user or display information to a user;

[0117] - A hard disk 906 labeled as HD can be provided as a mass storage device;

[0118] - The I / O module 907 may be used to receive / send data from / to an external device (such as a video source or a display, etc.).

[0119] The executable code may be stored in the read-only memory 903, may be stored on the hard disk 906, or may be stored, for example, on a removable digital medium such as a disk, etc. According to a variant, the executable code of the program may be received via the network interface 904 by means of a communication network so that the executable code of the program is stored in one of the storage components of the communication device 900 (such as the hard disk 906, etc.) before being executed.

[0120] The central processing unit 901 is adapted to control and direct the execution of instructions or parts of software code of one or more programs according to embodiments of the present invention, which instructions are stored in one of the aforementioned storage components. After being powered on, the CPU 901 is able to execute instructions from the main RAM memory 902 related to software applications after loading them, for example, from the program ROM 903 or the hard disk (HD) 906. Such software applications, when executed by the CPU 701, cause the steps of the flowchart of the present invention to be performed.

[0121] Any step of the algorithm of the present invention may be implemented in software by executing a collection of instructions or programs by a programmable computing machine (such as a PC ("personal computer"), a DSP ("digital signal processor") or a microcontroller, etc.); or may be implemented in hardware by a machine or dedicated component (such as an FPGA ("field programmable gate array") or an ASIC ("application-specific integrated circuit"), etc.).

[0122] Although the present invention has been described above with reference to specific embodiments, the present invention is not limited to the specific embodiments, and modifications within the scope of the present invention will be apparent to those skilled in the art.

[0123] Many further modifications and variations will occur to those skilled in the art when referring to the foregoing illustrative embodiments, which are given as examples only and are not intended to limit the scope of the invention, which is determined solely by the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.

[0124] The above-mentioned embodiments of the present invention can be implemented separately or as a combination of multiple embodiments. In addition, features from different embodiments can be combined when necessary or a combination of elements or features from separate embodiments can be combined in a single embodiment when it is beneficial.

[0125] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. A method for generating a media file, the method comprising: Generate a first track of media data samples; generating a second track of timed metadata, the second track being associated with the first track using a track reference and comprising at least one region geometry stored in a sample, wherein the at least one region geometry describes a geometry of a region in a media data sample of the first track, wherein the at least one region geometry is associated with at least one region annotation describing the region in the media data sample of the first track; as well as A media file is generated that includes the first track, the second track, and associated region annotations.

2. The method according to claim 1, wherein: The region geometry includes the location of anchor points and the shape of the region.

3. The method according to claim 1, wherein: The region annotation is stored as an attribute of a group of samples of the second track.

4. The method according to claim 1, wherein: The method further comprises: A third track of samples is generated, each sample being a region annotation associated with the region geometry of the second track, wherein the third track is associated with the second track using a track reference.

5. The method according to claim 1, wherein: The region annotations are stored as attributes in a property container in the metadata portion of the media file.

6. The method according to claim 5, wherein: The at least one region geometry is associated with the region annotation by means of a group of samples in the second track.

7. The method according to claim 5, wherein: An association between a region annotation and the at least one region geometry is stored in a sample group description entry indicating an index of the region annotation in the attribute container, wherein the sample group description entry is associated with a group of samples in the second track.

8. The method according to any one of claims 1 to 7, wherein: Region annotations include text annotations.

9. The method according to any one of claims 1 to 7, wherein: Region annotations include alternative representations of the region.

10. A computer program product for a programmable device, the computer program product comprising a sequence of instructions which, when loaded into and executed by the programmable device, implements the method according to any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium storing instructions of a computer program for implementing the method according to any one of claims 1 to 9.

12. An apparatus for generating a media file, the apparatus comprising a processor, the processor being configured to: Generate a first track of media data samples; generating a second track of timed metadata, the second track being associated with the first track using a track reference and comprising at least one region geometry stored in a sample, wherein the at least one region geometry describes a geometry of a region in a media data sample of the first track, wherein the at least one region geometry is associated with at least one region annotation describing the region in the media data sample of the first track; and A media file is generated that includes the first track, the second track, and associated region annotations.

13. A method for processing a media file, the method comprising: Obtain a media file, the media file comprising: the first track of media data samples, and a second track of timed metadata, the second track being associated with the first track using a track reference and comprising at least one region geometry stored in a sample, wherein the at least one region geometry describes a geometry of a region in the media data samples of the first track, wherein the at least one region geometry is associated with at least one region annotation describing the region in the media data samples of the first track; obtaining a media data sample from the media file; obtaining a region geometry associated with a media data sample, an associated geometry, and an associated region annotation; The obtained media data and information are processed to output a video based on the obtained media data and information.

14. An apparatus for processing a media file, the apparatus comprising a processor, the processor being configured to: Obtain a media file, the media file comprising: the first track of media data samples, and a second track of timed metadata, the second track being associated with the first track using a track reference and comprising at least one region geometry stored in a sample, wherein the at least one region geometry describes a geometry of a region in the media data samples of the first track, wherein the at least one region geometry is associated with at least one region annotation describing the region in the media data samples of the first track; obtaining a media data sample from the media file; obtaining a region geometry associated with a media data sample, an associated geometry, and an associated region annotation; The obtained media data and information are processed to output a video based on the obtained media data and information.

Images