Information processing apparatus, information processing method, and computer program

Abstract

At least one piece of region information for describing at least one piece of shape information for identifying an object included in the image is generated, and association information for associating the image with the region information is generated. Combination type information for combining the area information is generated. Combination area information for identifying the area combined according to the combination type information is generated. At least one piece of area information, association information, combination type information and combination area information are stored in the image file.

Description

Technical Field

[0001] The present invention relates to an information processing program, an information processing method, and a computer program. Background Art

[0002] Recent cameras, smartphones, and image analysis services can generate annotations (metadata that can be read by humans or computers and enables identification of, for example, what the object is, or metadata related to the image) associated with a portion or object in an image. For example, it is possible to detect a person, face, or object captured in an image. This can be done manually, but thanks to advances in deep learning technology, AI-based recognition is now available. In this case, information for identifying the area or annotation information can be added as metadata indicating the identified object.

[0003] Images captured by a camera or smartphone or processed by an image analysis service are stored in a storage device such as a memory card. Images are usually encoded to reduce their data size on the storage device. In encoding, many codec standards such as JPEG, H.264 (AVC), H.265 (HEVC), H.266 (VVC) and AV1 can be used. In order to store these encoded images in a file, a specification structure including metadata is defined, and a method for associating stored images and a metadata configuration in a specific format are defined. As a file format of this type, the ISO Base Media File Format (ISOBMFF, ISO / IEC 14496-12) is defined.

[0004] ISOBMFF is used for transmission via local storage, network or another streaming distribution mechanism. ISOBMFF is a well-known flexible and extensible file format for encapsulating and describing temporally or non-temporally coded media data or bitstreams. The file format has several extensions. For example, ISO / IEC 14496-15 defines encapsulation tools for various NAL (Network Abstraction Layer) unit-based video coding formats. Examples of such coding formats are AVC (Advanced Video Coding), SVC (Scalable Video Coding), HEVC (High Efficiency Video Coding), L-HEVC (Layered HEVC) and VVC (Versatile Video Coding).

[0005] Another example of a file format extension is ISO / IEC 23090-2, which defines the Omnidirectional Media Application Format (OMAF). Still other examples of file format extensions are ISO / IEC 23090-10 and ISO / IEC 23090-18, which define the transmission of Visual Volumetric Video Coding (V3C) media data and Geometry-based Point Cloud Compression (G-PCC) media data.

[0006] Other examples of file format extensions are the High Efficiency Image File Format (ISO / IEC 23008-12, HEIF), which defines encapsulation tools for still image sequences, such as still images or HEVC still images.

[0007] These file formats are standards developed by the Moving Picture Experts Group (MPEG) to store and share images and image sequences, and define structures with object orientation geometry-based point cloud compression (G-PCC).

[0008] Patent Document 1 discloses a method of identifying a region in an image stored in HEIF as a Region item, making the in-image region identifiable in association with the stored image, and adding annotation information to the identified in-image region.

[0009] Patent Document 2 discloses a method for extracting multiple spatial portions of an image to be stored in HEIF, storing these portions, and reconstructing single image data by combining these extracted portions. This method can record a single image representation formed by multiple still images, called derived images, as an image file.

[0010] Prior art literature

[0011] Patent Literature

[0012] Patent Document 1: International Publication No. 2021 / 204526

[0013] Patent Document 2: Japanese Patent Application Laid-Open No. 2021-119713 Summary of the Invention

[0014] Problems to be solved by the invention

[0015] In the method of patent document 1, the Region item identified as a region uses a predetermined shape to define the region, so the Region item cannot flexibly define the region. As a flexible shape description, a Mask shape is defined. However, the description data size of Mask data is large. That is, the flexible description using a predetermined shape increases the description data size, or the shape needs to be expressed using a defined specific shape, and flexible shape specification (that is, the expression obtained by combining regions) is not considered.

[0016] Furthermore, while the framework can add annotation information to identified regions, it cannot show associations between regions or combine regions to add another annotation. It is possible to use metadata described as annotations to identify groups. However, this cannot be identified without analyzing all the large amounts of annotation information. Furthermore, it is not possible to identify whether a common annotation is added for grouping purposes or whether the same annotation is added to regions that do not belong to a group.

[0017] On the other hand, the method described in Patent Document 2 allows for the representation of a combination of image data units, but does not represent a combination of identification areas corresponding to subjects in the image. The present invention provides a technique for generating an image file having a structure that enables flexible specification of areas in an image and identification of associations between the areas.

[0018] Solutions for solving problems

[0019] As an aspect of the present invention, an information processing device is an information processing device for generating a media file, characterized in that it includes: an acquisition component for acquiring an image; a first acquisition component for acquiring at least one area information, the at least one area information being used to describe at least one shape information for identifying an object included in the image; a second acquisition component for acquiring association information for associating the image with the area information; a third acquisition component for acquiring combination type information for combining area information; a fourth acquisition component for acquiring combination area information for identifying areas combined according to the combination type information; and a storage component for storing the at least one area information, the association information, the combination type information, the combined area information and the image in a media file.

[0020] Effects of the Invention

[0021] According to the present invention, it is possible to generate an image file having a structure capable of flexibly specifying regions in an image and identifying the relationship between the regions.

[0022] Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings. Note that throughout the drawings, the same reference numerals represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0024] Figure 1 is a block diagram showing an example of the hardware configuration of the image file storage device 100;

[0025] Figure 2 is a diagram showing the structure of a HEIF file;

[0026] Figure 3A is a diagram showing an example of definition of a structure of area information;

[0027] Figure 3B is a diagram showing an example of definition of a structure of area information;

[0028] Figure 4 is a diagram showing the structure of EntityToGroupBox (entity to group box);

[0029] Figure 5 is a diagram showing an example of definition of a structure of annotation information;

[0030] Figure 6 is a diagram showing the structure of a CombinedRegionEntityGroupBox;

[0031] Figure 7 is a diagram showing the structure of a MultiplyRegionEntityGroupBox;

[0032] Figure 8 is a diagram showing the structure of an InverseRegionEntityGroupBox;

[0033] Figure 9 is a diagram showing the structure of a LogicalOperateRegionEntityGroupBox (logical operation region entity group box);

[0034] Figure 10 is a diagram showing the structure of an InclusionRelationRegionEntityGroupBox (including a relationship region entity group box);

[0035] Figure 11 is a diagram showing the structure of a TouchRelationRegionEntityGroupBox (touch relationship region entity group box);

[0036] Figure 12 is a diagram showing the structure of a DepthOrderRelationRegionEntityGroupBox;

[0037] Figure 13 is a diagram showing the structure of a ForbidRelationRegionEntityGroupBox;

[0038] Figure 14 is a diagram showing the structure of InvereRegionProperty (inverse region property);

[0039] Figure 15 is a diagram showing the structure of LogicalOperateRegionProperty (logical operation region property);

[0040] Figure 16A is a diagram showing an example of the configuration of an image file;

[0041] Figure 16B is a diagram showing an example of the configuration of an image file;

[0042] Figure 16C is a diagram showing an example of the configuration of an image file;

[0043] Figure 17 It is a flowchart of the HEIF file generation process;

[0044] Figure 18 is a flowchart of a reproduction process related to reproduction (display) of a HEIF file;

[0045] Figure 19 is a diagram showing an example of display of area information and area annotation information;

[0046] Figure 20 is a diagram showing an example of display of area information and area annotation information;

[0047] Figure 21 is a diagram showing an example of display of area information and area annotation information;

[0048] Figure 22 is a diagram showing the structure of a HEIF file;

[0049] Figure 23 is a diagram showing the structure of RegionLogicalDerivation;

[0050] Figure 24A is a diagram showing an example of the configuration of an image file;

[0051] Figure 24B is a diagram showing an example of the configuration of an image file;

[0052] Figure 24C is a diagram showing an example of the configuration of an image file;

[0053] Figure 25 is a diagram showing the data structure of a derived region item;

[0054] Figure 26 is a diagram showing the definition of CombinedRegionItem;

[0055] Figure 27 is a diagram showing the definition of CompoundEntityToGroupBox;

[0056] Figure 28 is a diagram showing an example of a presentation including region information stored in a media file;

[0057] Figure 29 is a diagram showing an example of a regional track 2920;

[0058] Figure 30 is a diagram showing another example in the embodiment;

[0059] Figure 31A is a diagram showing an example of a regional track 3110 (eg, an "rgan" track); and

[0060] Figure 31B is an example showing that a description of the combined area 3155 is included in the metadata portion of the combined area track 3150 . DETAILED DESCRIPTION

[0061] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the claimed invention. While various features are described in the embodiments, this does not limit the invention to requiring all of these features, and multiple such features may be appropriately combined. Furthermore, in the accompanying drawings, the same reference numerals are assigned to identical or similar configurations, and redundant descriptions thereof are omitted.

[0062] [First embodiment]

[0063] First, refer to Figure 1 An example of the hardware configuration of the image file storage device 100 serving as the information processing device according to the present embodiment is explained with reference to a block diagram of FIG. Figure 1 As shown, the functional units of the image file storage device 100 are configured to communicate with each other via the system bus 109. Note that in this embodiment, Figure 1 The functional units shown are implemented by hardware, but some functional units or functional units that perform some of the functions of some functional units may be implemented by software (computer programs). In this case, the computer program is executed by CPU 101, and the corresponding functions are implemented as a result.

[0064] The CPU 101 executes various processes using computer programs and data stored in the RAM 103 or the ROM 102. Thus, the CPU 101 controls the operation of the entire image file storage device 100 and executes or controls various processes to be described as processes to be performed by the image file storage device 100.

[0065] The ROM 102 is an example of a nonvolatile storage device capable of permanently storing information. The ROM 102 stores setting data for the image file storage device 100, computer programs and data associated with starting the image file storage device 100, and computer programs and data associated with the basic operation of the image file storage device 100. The data stored in the ROM 102 includes, for example, parameters required for the operation of each functional unit and display data.

[0066] The RAM 103 is an example of a volatile storage device capable of temporarily storing information. The RAM 103 includes an area configured to store computer programs and data loaded from the ROM 102 or the nonvolatile memory 110, and an area configured to store captured images input from the imaging unit 104. Furthermore, the RAM 103 includes an area used when the image processing unit 105 performs various processes, an area for storing data received from the outside by the communication unit 108, and a work area used when the CPU 101 performs various processes. Thus, the RAM 103 can provide various areas as appropriate.

[0067] For example, the RAM 103 is used not only as an area for expanding computer programs but also as a storage area (output buffer) for temporarily storing data or the like output in the operations of various functional units.

[0068] The imaging unit 104 performs photoelectric conversion on an optical image formed on an imaging surface of an imaging element (e.g., an imaging element such as a CMOS sensor or CCD) via an optical system (not shown), and performs various image processing (such as noise removal and gain processing) on ​​the analog signal obtained by the photoelectric conversion. The imaging unit 104A then A / D-converts the analog signal that has undergone various image processing into a digital signal and outputs the digital signal as a captured image.

[0069] The image processing unit 105 performs various image processing on the image. Image processing includes, for example, processing related to development, such as gamma conversion, color space conversion, white balance processing, and exposure correction. In addition, the image processing unit 105 can perform image analysis processing or synthesis processing of two or more images.

[0070] The image processing unit 105 includes an encoding / decoding unit 111, a metadata processing unit 112, a generation unit 113, and an identification processing unit 114. In this embodiment, for ease of understanding, the description will be made assuming that the hardware of a single image processing unit 105 performs the processing of the functional units (encoding / decoding unit 111, metadata processing unit 112, generation unit 113, and identification processing unit 114). However, the processing of the functional units (encoding / decoding unit 111, metadata processing unit 112, generation unit 113, and identification processing unit 114) may be performed by multiple hardware units.

[0071] The encoding / decoding unit 111 is a codec for moving images or still images compliant with H.265 (HEVC), H.264 (AVC), H.266 (VVC), AV1, JPEG, etc. The encoding / decoding unit 111 performs encoding or decoding of images (still images or moving images (image sequences)) handled by the image file storage device 100.

[0072] The metadata processing unit 112 acquires the image data (encoded data) encoded by the image encoding / decoding unit 111. The metadata processing unit 112 generates an image file that conforms to a predetermined file format (e.g., HEIF) and includes the encoded data and metadata associated with the encoded data. More specifically, the metadata processing unit 112 analyzes the encoded data stored in an image file, such as a still image or an image sequence, generates information related to the still image or image sequence, and acquires parameters associated with the encoded data. The metadata processing unit 112 then stores this information as metadata in the image file along with the encoded data. Note that the metadata processing unit 112 can generate not only HEIF files but also image files conforming to other moving image file formats defined by MPEG or formats such as JPEG. Note that the encoded data to be acquired can be data previously stored in the ROM 102 or non-volatile memory 110, or data acquired via the communication unit 108 and stored in the RAM 103. Furthermore, the metadata processing unit 112 generates and stores metadata corresponding to the region information and combined region information generated by the generation unit 113. Furthermore, the metadata processing unit 112 analyzes metadata stored in image files and performs metadata processing when reproducing still images or image sequences. Furthermore, the metadata processing unit 112 generates metadata associated with information identifying regions of images to be stored in image files based on the region information and combined region information generated by the generation unit 113, and generates metadata associating the region information with the images. The metadata processing unit 112 also generates metadata related to annotation information to be associated with the images or region information. The metadata processing unit 112 also analyzes this metadata during image reproduction.

[0073] The generation unit 113 generates region information capable of identifying a subject based on the subject information detected from the image by the recognition processing unit 114, and also generates type information indicating a combination of region information. Furthermore, the generation unit 113 generates annotation information to be added to the region information or to combined information corresponding to the combination of region information. Note that when generating region information, the subject detection results of the recognition processing unit 114 may not be used, and instead, information on a region specified by a user operating the operation input unit 107 may be used.

[0074] The recognition processing unit 114 performs object recognition processing using, for example, AI on the image stored in the image file, and performs processing to detect the subject object from the image. Note that in the object recognition processing, the subject object can be detected through content analysis, and this processing can be performed by, for example, an external device capable of communicating with the image file storage device 100 via the communication unit 108. In this case, the image file storage device 100 obtains the subject object detection result from the external device. The recognition processing unit 114 obtains information such as the position of the subject object in the image or the area of ​​the subject object in the image. Note that in the object recognition processing, various detection processes such as multiple objects and the association between these objects can be performed according to the recognition algorithm.

[0075] The display unit 106 is a display device including a liquid crystal display (LCD) or a touch panel screen, and may be a display device that is detachably connected to the image file storage device 100 or may be a display device that is integrated with the image file storage device 100. The display unit 106 displays an image (still image or moving image) being captured by the imaging unit 104 (live view display), or displays information such as various settings / GUI (graphical user interface). In addition, the display unit 106 displays an image when a generated image file is reproduced.

[0076] The operation input unit 107 is a user interface including operation buttons, switches, a mouse, a keyboard, and the like, and a user can input various instructions to the CPU 101 by operating the operation input unit 107. Note that in a mode in which the display unit 106 is a touch panel screen, the operation input unit 107 may include a touch panel sensor.

[0077] The communication unit 108 is a communication interface configured to perform data communication with an external device. The communication unit 108 may be, for example, a network interface that is connected to a network and sends / receives transmission frames. In this case, the communication unit 108 may be, for example, a network interface that can perform data communication via Ethernet. Alternatively, if the communication unit 108 can be connected to a wireless LAN, the communication unit 108 may include a controller that performs wireless LAN control such as IEEE802.11a / b / g / n / ac / ax, an RF circuit, and an antenna.

[0078] The nonvolatile memory 110 is a nonvolatile information storage device having a large storage capacity, for example, an SD card, a compact flash memory, or a memory card. For example, in the nonvolatile memory 110 , an image file generated in the present embodiment may be stored, or an image file acquired from an external device via the communication unit 108 may be stored.

[0079] Notice, Figure 1 The hardware configuration shown is merely an example of a configuration capable of realizing the operation of the image file storage device 100 to be described below, and can be changed / modified as appropriate. Figure 1 The camera unit 104 is integrated with the image file storage device 100, but can be detachably connected to the image file storage device 100. In addition, the image processing unit 105 can be a device detachably attached to the image file storage device 100, or can be an external device capable of communicating with the image file storage device 100 via the communication unit 108.

[0080] Next, the generation of an image file using the image file storage device 100 will be described. The image file generated by the image file storage device 100 can store a plurality of images and is configured to include information attached to these images. The following will describe a mode in which HEIF is used as the file format of the image file, necessary information is exported to generate an image file (HEIF file) that conforms to the information, and metadata to be attached is formed. However, the present invention is not limited thereto, and for example, the file format adopted for the image file to be generated may be, for example, other motion image file formats defined by MPEG, omnidirectional media application file formats, file formats for handling 3D data such as point group data, or formats such as JPEG.

[0081] Next, refer to Figure 2 To illustrate the file structure of HEIF files. Figure 2 As shown, the HEIF file 200 generally includes the following four boxes (storage areas).

[0082] The first box 201 is a FileTypeBox (“ftyp”), and a brand name used by a reader (Reader) of the HEIF file 200 to identify the specifications of the HEIF file 200 is stored in the box 201 .

[0083] The second box 202 is an ExtendedTypeBox ("etyp"), and in the box 202, an extended brand name used by a reader (Reader) of the HEIF file 200 to identify the specifications of the HEIF file 200 is stored. Here, a "unif" brand definition configured to enable a track ID, item ID, entity group ID, etc. to be used as a uniquely identifiable value at the file level is stored.

[0084] The third box 203 is a MetaBox ("meta"). Figure 2 As shown, in block 203, as will be described in detail below, multiple types of information associated with the image are stored in different blocks.

[0085] The fourth box 204 is a MediaDataBox ("mdat"), and in box 204, coded data (image) 241 and 242 are stored as a coded bit stream. Also stored in box 204 are region information 243 and 244 indicating regions within the image. Furthermore, box 204 stores an Exif data block 245, which stores information related to image capture by the image capture unit 104. In this embodiment, a mode in which box 204 is used as a region for storing coded data 241 and 242, region information 243 and 244, and Exif data block 245 will be described. However, a frame structure such as "idat" or "imda" may also be used as the region. Note that in the following description, the coded data 241 and 242 stored in box 204 will be referred to as "image" or "coded data," using different terms, as appropriate.

[0086] Note that when storing moving images, audio, or image sequences, a MovieBox ("moov") (not shown) can be stored separately. This box stores metadata that describes various information related to the presentation of the moving images, audio, etc. stored in the image file. Note that if the stored data is an image sequence, the metadata is stored using a mechanism that describes various information related to the moving images. However, unlike moving images, time limit information is optional.

[0087] Box 211 is a HandlerReferenceBox ("hdlr") and stores a declaration of a handler type for analyzing the structure of box 203. In the HEIF file 200 generated by the image file storage device 100 according to this embodiment, the target of the metadata describing the non-timing data stored in box 203 is a still image. For this reason, the handler type name "pict" is set in box 211 to identify that a still image is the target.

[0088] Box 212 is a PrimaryItemBox ("pitm"), and specifies an identifier (item ID) of image data that is a representative item of image items stored in the HEIF file 200. It is assumed that the image item designated as the first priority item is an image that is normally displayed for reproduction display.

[0089] Box 213 is an ItemLocationBox ("iloc"), and stores information indicating the storage location of information items including image items in the HEIF file 200. Box 213 generally describes the storage location of the image item as a byte offset relative to the top of the HEIF file 200 and a data length relative to the top. That is, through the information in box 213, the location of the encoded data, region information, and Exif data block stored in box 204 can be specified. For derived items, the information in box 213 indicates that there is no data in box 204. In this case, it is indicated that there is no data structure, or that the data of the derived item is stored in box 217 in box 203.

[0090] Box 214 is an ItemInfoBox ("iinf"), and defines basic information (item information) such as an item ID and an item type indicating the type of the item for all items included in the HEIF file 200. In the item information, not only image items such as a coded image item and a derived image item are specified, but also area items indicating areas in an image and Exif information items indicating Exif data blocks are specified.

[0091] Box 215 is an ItemReferenceBox ("iref"), and stores information describing the association between items stored in the HEIF file 200. In a mode where the image item is a captured image, box 215 is used to describe the association between the image item and the items of its camera information (Exif data, etc.). In a mode where multiple image items have a relationship of derived images, box 215 is used to describe the association between image items. In a mode indicating an area in an image item, box 215 is used to describe the association between the image item and the area item. In order to associate items, the item reference type is specified, and the type of the item reference can be identified. In addition, the reference relationship between the items is described by describing the item ID specified in from_item_ID and to_item_ID in box 214.

[0092] Box 216 is an ItemPropertiesBox ("iprp") and stores various attribute information (item properties) of the information items included in the HEIF file 200. More specifically, box 216 includes an ItemPropertyContainerBox ("ipco") as a box 221 describing attribute information and an ItemPropertyAssociation ("ipma") as a box 222 indicating the association between attribute information and each item. Box 221 can store, for example, attribute information such as entry data indicating an HEVC parameter set required for decoding an HEVC image item and entry data indicating the width and height of the image item in pixels. Attribute information that can specify arbitrary text information as item properties can also be specified. Figure 5 The illustrated UserDescriptionProperty ("udes") is an example of attribute information that can store arbitrary text information. Figure 5 The UserDescriptionProperty shown includes the 4-character code "udes" shown in definition 501, and the 4-character code is used to identify the UserDescriptionProperty. The udes property includes user-defined name information (name) 503, comment description information (description) 504, and tag information (tag) 505, and can be associated with an item or entity group. The udes property also includes lang 502, and the language used in the udes property is indicated by a language tag string based on IETF RFC5646. If multiple udes properties are associated with the same item or entity group, these multiple udes properties indicate options specified by different languages, and the reader can select the appropriate option. Here, the name information 503 is information for indicating the name of the item or entity group associated with the udes property in human-readable format. In addition, the comment description information 504 is information for indicating text presented as an explanation or comment related to the item or entity group associated with the udes property in human-readable format. The tag information 505 is information for indicating a tag that is user-defined or automatically assigned and is associated with an item or entity group associated with the udes property, and is configured to include multiple tags separated by, for example, commas. When the udes property is associated with an area item, the annotation information can be associated as a text description with the specific area indicated by the area item associated with the image item.

[0093] A Transformative Property (transformation property) for converting an image when outputting it and displaying it can also be specified as attribute information that can be specified as an item property. This can store, for example, data indicating rotation information for rotating an image and displaying it, or data indicating cutout information for cropping (cutting out) an image and displaying it.

[0094] Next, block ("ipma") 222 stores, for each item, entry data indicating the association with the attribute information stored in block 221 using the ID of the information item (item ID). Note that if there is no attribute information to be associated with the item (as in the case of an Exif data block information item), entry data indicating the association is not stored.

[0095] Box 217 is an ItemDataBox ("idat") and stores data related to items included in the HEIF file 200. In box 217, for example, a data structure for describing a derived image item is stored. Here, for example, if the item type indicated by box 214 is a "grid" item, a data structure of a grid derived image item that reconstructs an input image in a predetermined grid order and defines the input image is specified. For the input image of the derived image item, box 215 is used to specify an item reference of the dimg reference type. Note that if the derived item is, for example, an identity derived image item "iden" that has no data structure, no data structure is stored in box 217.

[0096] Box 218 is a GroupListBox ("grpl") and stores metadata configured to group and store entities such as items or tracks included in the HEIF file 200. In box 218, the metadata is stored for each grouping type parameter extension and definition. Figure 44. The EntityToGroupBox is a box of the EntityToGroupBox shown. The EntityToGroupBox includes a grouping_type indicated by definition 401. The grouping_type includes a 4-character code defined for each grouping type, and the 4-character code is used to identify the grouping type of the EntityToGroupBox. The grouping type is a concept that defines the relationship between multiple entities included in the group. The EntityToGroupBox includes a group_id 402 for uniquely identifying the entity group itself, and a num_entities_in_group 403 indicating the number of entities included in the entity group. In addition, the EntityToGroupBox includes as many entity_ids 404 as the number specified by num_entities_in_group. In the entity_id 404, an item ID identifying an item defined in the box 214 or a track ID identifying a single track of the presentation included in the MovieBox (not shown) can be specified. In an entity group of a specific group type, a group_id identifying other entity groups can be specified. Furthermore, EntityToGroupBox is configured so that it can be extended and defined for each grouping type, and is defined as a structure capable of defining extended parameters corresponding to the grouping type in section 405. As described above, in EntityToGroupBox, entities such as image items or tracks included in a group can be handled as meaningful group units by grouping type. Figure 2 In the illustrated frame configuration, frames 231 and 232 (CombinedRegionEntityGroupBox ("cbrg")) obtained by extending EntityToGroupBox to a grouping type "cbrg" as one of the grouping types for grouping region information are stored. As described above, frames 231 and 232 are extended definitions of EntityToGroupBox and therefore include definitions 402 to 404 included in EntityToGroupBox, and grouping_type 401 includes "cbrg" as a 4-character code (4CC) for identifying CombinedRegionEntityGroupBox.

[0097] Furthermore, in entity_id, the item ID indicating the region item is specified, and the regions are grouped. Note that in entity_id, the group ID of the entity group that groups individually defined regions can also be specified. This allows combining region items and region item groups, or combining region item groups. Note that detailed definitions for the "cbrg" entity group or other grouping types will be described later.

[0098] Next, we will refer to the Figure 3A and Figure 3B The structure of the region information 243 to 244 stored in the box 204 is explained below. The item type "rgan" shown in the iinf box defines a region item for identifying a region in the image. As for the association between the region information and the coded data, the iref box is used to specify an item reference of the cdsc reference type and defines the region in the image. Figure 3A and Figure 3B As shown, each of the region information 243 and 244 includes version information 301 of the data structure and flags 302 for selectively handling the process or structure according to the value. In this embodiment, version information 301 is designated as 0. The region information also includes data size information 303 indicating the size (field_size) of the parameter handled in the data structure. In this embodiment, the size handled in the data structure of region information 243 or region information 244 is configured to be switchable between 16 bits and 32 bits, and one of these data sizes is determined based on the value of flags.

[0099] In addition, if Figure 3A and Figure 3B As shown, the spatial size information is included, and the spatial size information is used to indicate the two-dimensional size of the reference space used to define the area associated with the area information 243 or 244. Figure 3A and Figure 3BAs shown, the space size information includes reference_width 304, which indicates the width of the reference space, and reference_height 305, which indicates the height of the reference space. Given that HEIF files can store images of multiple sizes and that image sizes can be changed through editing, storing region information 243 or 244 for each image size of the stored images is inefficient. Therefore, in this embodiment, various information is introduced that allocates the image size handled by region information 243 or 244 to the entire reference space and defines regions for that reference space, thereby enabling region definition to be relatively independent of the image size. For example, in a mode where the reference space has a size of 1024 pixels × 512 pixels, the region designated for an image of the same image size is defined using values ​​equal to the position, width, and height indicated by region shape information 307, described later. On the other hand, the region designated for an image of 2048 pixels × 1024 pixels is defined using values ​​obtained by multiplying the position, width, and height indicated by region shape information 307, described later, by 2. That is, the region information 243 and 244 each defines a region as a relative position / shape in the entire image, and the space size information defines the two-dimensional size of the reference space allocated to the whole.

[0100] In addition, if Figure 3A and Figure 3B As shown, the region count information 306 indicates the number of regions (region_count) defined by the region information 243 or 244, and the region shape information 307 indicates the shape (geometry_type) of each defined region. In the region information 243 or 244 according to this embodiment, one of a point, rectangle, ellipse, polygon, polyline, reference mask, and inline mask can be selected as the region shape, and the shape is specified by the value of geometry_type. Note that in this embodiment, it will be assumed that seven types of two-dimensional shapes can be selected as regions for explanation, but the present invention is not limited to this. That is, it can be easily understood that if any space in the image can be specified, for example, a line, triangle, free curve, or 3D shape can be used as the region shape information 307.

[0101] Here, in area shape information 307, the method for describing the specific parameters used to specify an area varies depending on the shape. If the area shape is a point (geometry_type is 0), the area can be specified by position information (x, y) 308, which indicates the coordinates of the point in the reference space. If the area shape is a rectangle (geometry_type is 1), the area is specified by position information (x, y) 309, which indicates the coordinates of the point (reference point) at the upper left corner of the rectangle in the reference space, and shape definition information 310, which indicates the width and height of the rectangle. If the area shape is an ellipse (geometry_type is 2), the area is specified by position information (x, y) 311, which indicates the coordinates of the center of the ellipse in the reference space, and shape definition information 312, which indicates the length of the radius (radius_x) in the x-axis direction and the length of the radius (radius_y) in the y-axis direction of the ellipse. If the shape of the region is a polygon (geometry_type is 3) or a polyline (geometry_type is 6), the region is specified by vertex count information 313 indicating the number of vertices of the polygon in the reference space, and position information (px, py) 314 indicating the coordinates of the vertices. Regarding the shape difference between a polygon (geometry_type is 3) and a polyline (geometry_type is 6), a shape formed by connecting the start and end points of the position information indicating the coordinates of each vertex is a polygon (geometry_type is 3), and a shape where the start and end points are not connected is a polyline (geometry_type is 6). If the shape of the region is a reference mask (geometry_type is 4), the reference type mask in the iref box is used to associate another item defined as a mask image item with the encoded image. The area is specified by position information (x, y) 315 and shape definition information 316, wherein the position information (x, y) 315 is used to indicate the coordinates of the point (reference point) at the upper left corner of the reference mask in the reference space, and the shape definition information 316 is used to indicate the width and height of the reference mask. Note that if 0 is specified as the width and height, the size information is associated with the reference mask image item as an item property. If the shape of the area is an inline mask (geometry_type is 5), the image portion corresponding to the pixel whose value in the mask image stored in data 321 is 1 indicates the area. The area is specified by position information (x, y) 317 for indicating the coordinates of the point (reference point) at the upper left corner of the area (rectangle) in the reference space, and shape definition information 318 for indicating the width and height of the area. In addition, as. Figure 3A and Figure 3B As shown, mask_coding_method 319 is included, which indicates the encoding method applied to the mask included in the data. If the value of mask_coding_method 319 is 0, the encoding is not suitable for the mask. If the value of mask_coding_method 319 is 1, this indicates adaptation to deflate compression defined by RFC1951. Other values ​​are reserved and undefined. mask_coding_parameters 320 indicate additional encoding parameters required for normal processing of the encoded mask data. If the value of mask_coding_method 319 is 1, mask_coding_parameters 320 indicates the number of bytes of the coded mask array data. When the value of mask_coding_method is greater than 1, the value of mask_coding_parameters is retained. In data 321, a coded representation or uncompressed representation of the mask including pixels of the inline mask is included in raster scan order. Each pixel is represented by 1 bit, and 8 pixels are packed into 1 byte. Byte packing is performed in big-endian order. If the mask width is not a multiple of 8 pixels, no padding is done at the end of each row. Only the last data byte is padded with bits set to 0.

[0102] Next, a definition will be described that uses region information 243 or 244 formed according to the definition to set a region in one of the image items stored in the HEIF file 200, and stores a region annotation in association with the region. The region annotation is formed by metadata or other image items associated with one or more regions in the image item. When the region items are grouped into entity groups corresponding to the grouping type in box 218, the regions can be treated as region entity groups combined according to the grouping type. The regions placed in the group can be uniquely identified by the group ID.

[0103] In the image file storage device 100 according to the present embodiment, information associated with the region information 243 to 244 and the frames 231 and 232 is added to the following frames in the frame 203 , thereby associating region annotations with regions and combined regions in a specific image item.

[0104] First, in the HEIF file 200 according to the present embodiment, since the encoded data 241 to 242 and the region information 243 to 244 are stored in the box 204, the item type is set for each data in the box 214 to distinguish them. Regarding the item type, in the case of the encoded data 241, for example, "hvc1" is set, and in the case of the region information 243, "rgan" is set. As with the encoded data 241 to 242, the storage location of the region information 243 to 244 stored in the box 204 in the HEIF file 200 is stored in the box 213. In addition, since the region information 243 to 244 is information for relatively specifying a region in the reference space as described above, it is necessary to specify the image item for setting the region. Information for indicating for which image item the region is set for one region information 243 is stored in the box 215. That is, since box 215 describes the association between the items set in box 214, for example, information identifying an image item associated with encoded data 241 is stored in this box. For this encoded data 241, region information 243 is used to set regions for region items associated with this encoded data 241. Furthermore, when a region item is associated with a derived image item formed as a derived image, regions within the derived image can be set. When a region item is treated as a region entity group, the grouping type used to group regions is determined to determine grouping_type. To treat regions assigned to a region item as logically added regions, cbrg is specified as the group type. The region item ID is specified as the entity that forms the entity group. Thus, logically added regions can be designated as a region group. Furthermore, to associate image items with grouped region groups, box 215 stores information indicating for which image item a region group sets a region. Note that since each region item included in an entity group is associated with an image item, a region group does not necessarily need to be associated with an image item.

[0105] On the other hand, for a region set for a specific image item, a region annotation for the region may be set by associating one or all of the following data with a region item or a region entity group.

[0106] First, there is a regional annotation of the usage item property. The udes property stored in box 221 in box 216 is associated with the regional item or regional entity group through box 222. When the udes property is associated with the regional item or regional entity group, the annotation information can be associated as a text description.

[0107] Secondly, there is a region annotation using metadata items. Metadata items such as Exif etc. that store metadata in mdat are associated with region items or region entity groups through the reference type cdsc in box 215.

[0108] Finally, there are region annotations that use other image items or entity groups. Region items or region entity groups are associated with other image items or entity groups using the reference type eroi. This can provide different image representations for specific regions, for example. By associating, region annotations can be set for specific regions in the image.

[0109] Next, the definition of a method of specifying a combined area of ​​region items that can be stored in the HEIF file 200 will be described. Figure 6 is a diagram showing the data structure of a CombinedRegionEntityGroupBox, which is an entity group for grouping regions that can be stored in box 218 of a HEIF file 200. CombinedRegionEntityGroupBox is an extension of EntityToGroupBox and has a grouping_type "cbrg" 601. No additional parameters specific to the entity group type are defined. The CombinedRegionEntityGroup can be defined as follows. Note that it can be easily understood that the name of the entity group and the 4CC used to identify the entity group can be another name and a different 4CC. For example, UnionOfRegionsEntityGroupBox can be used as another name to indicate a combination and can be identified by "unrg" or the like.

[0110] CombinedRegionEntityGroup is identified by "cbrg" grouping_type and indicates a set of regions used to create a combined region formed by combining the regions indicated by the included entities by OR (or). For entity_id, if the region item, region metadata track or unif brand uses a unique ID, specify EntityToGroupBox to indicate other combined regions. More flexible region specification can be made by specifying other combined regions. Regions are formed by OR of these regions. That is, all regions indicated by one of the regions indicated by entity_id are identified as the region indicated by CombinedRegionEntityGroup. Multiple CombiendRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0111] The number of regions indicated by an entity specified by entity_id is recommended to be one (for example, region_count in the region item is 1). In this case, the number of specified entities (num_entities_in_group) is recommended to be two or more. On the other hand, if the number of regions indicated by an entity is two or more (for example, region_count in the region item is 2 or more), these regions are combined by ORing all regions together with the regions indicated by other entities. In this case, the number of specified entities (num_entities_in_group) can be one. As another method, if the number of regions indicated by the entities specified in the group is equal (region_count in all entities is equal), these regions can be combined in the order in which they are indicated. Various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of CombinedRegionEntityGroupBox. The regions indicated by CombinedRegionEntityGroup and the regions indicated by the entities included in CombinedRegionEntityGroup can be individually associated with image items, and regions in the image can be identified. In addition, different annotation information can be associated by the method defined in the region annotation.

[0112] Figure 7 701. Parameters specific to the entity group type are not defined. The MultiplyRegionEntityGroup can be defined as follows. Note that it can be easily understood that the name of the entity group and the 4CC used to identify the entity group can be another name and another 4CC. For example, RegionsIntersectionEntityGroupBox can be used as another name to indicate a combination and can be identified by "isrg" or the like.

[0113] MultiplyRegionEntityGroup is identified by "mprg" grouping_type and indicates a set of regions to be created by combining the regions indicated by the included entities via AND. For entity_id, if a region item, region metadata track, or unif brand uses a unique ID, specify EntityToGroupBox to indicate other combined regions. More flexible region specification can be made by specifying other combined regions. Regions are formed by AND of these regions. That is, the common region included in all regions indicated by entity_id is identified as the region indicated by MultiplyRegionEntityGroup. Multiple MultiplyRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0114] The number of regions indicated by an entity specified by entity_id is recommended to be one (for example, region_count in the region item is 1). In this case, the number of entities specified (num_entities_in_group) is recommended to be two or more. On the other hand, if the number of regions indicated by an entity is two or more (for example, region_count in the region item is 2 or more), these regions are combined by ANDing all regions with the regions indicated by other entities. In this case, the number of entities specified (num_entities_in_group) can be one. As another method, if the number of regions indicated by an entity is two or more, these regions can be treated as one region and combined by ANDing with the regions indicated by other entities. That is, the common region between different region entities is designated as the combined region. As yet another method, if the number of regions indicated by the entities specified in the group is equal (region_count in all entities is equal), these regions can be combined in the order in which they are indicated. Various region combination methods corresponding to the number of regions can be individually controlled by the value of flags included in the definition of MultiplyRegionEntityGroupBox. The region indicated by the MultiplyRegionEntityGroup and the region indicated by the entities included in the MultiplyRegionEntityGroup can be individually associated with the image item, and the region in the image can be identified. In addition, different annotation information can be associated by the method defined in the region annotation.

[0115] Figure 8is a diagram showing the data structure of an InverseRegionEntityGroupBox, which is an entity group for grouping regions that can be stored in box 218 of a HEIF file 200. InverseRegionEntityGroupBox is an extension of EntityToGroupBox and has a grouping_type "ivrg" 801. Additional parameters specific to the entity group type are not defined. The InverseRegionEntityGroup can be defined as follows. Note that it can be easily understood that the name of the entity group and the 4CC used to identify the entity group can be another name and another 4CC. For example, NegationRegionEntityGroupBox can be used as another name to indicate a combination and can be identified by "ngrg" or the like.

[0116] InverseRegionEntityGroup is identified by "ivrg" grouping_type and indicates a set of regions used to create a combined region formed by combining the regions indicated by the included entities via NOT or NOR. For entity_id, if the region item, region metadata track, or unif brand uses a unique ID, specify EntityToGroupBox to indicate other combined regions. More flexible region specification can be made by specifying other combined regions. Regions are formed by NOT or NOR of these regions. That is, a region not included in any region indicated by entity_id is identified as a region indicated by InverseRegionEntityGroup. Multiple InverseRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0117] It is recommended that the number of regions indicated by one entity specified by entity_id is one (for example, region_count in the region item is 1). In this case, if the number of specified entities (num_entities_in_group) is one, the region obtained by NOT of these regions is designated as a combined region. That is, the region other than the region indicated by the included entities is the target region. On the other hand, if the number of specified entities is two or more, the region obtained by NOR of the regions indicated by the included entities is designated as a combined region. That is, the region indicated by all entities and not specified in any region is the target region.

[0118] On the other hand, if the number of regions indicated by one entity is two or more (for example, region_count in the region item is 2 or more), the regions are combined by NOR of all regions with the regions indicated by other entities. As another method, if the number of regions indicated by the entities specified in the group is equal (region_count in all entities is equal), the regions can be combined in the order in which the regions are indicated. Various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of InverseRegionEntityGroupBox. The regions indicated by the InverseRegionEntityGroup and the regions indicated by the entities included in the InverseRegionEntityGroup can be individually associated with the image item, and the regions in the image can be identified. In addition, different annotation information can be associated by the method defined in the region annotation.

[0119] Similarly, to specify the combination of region information through other logical operations, you can define the grouping type of each EntityToGroupBox. For example, you can specify exclusive OR, NAND, or exclusive NOR.

[0120] Figure 9 is a diagram showing the data structure of a LogicalOperateRegionEntityGroupBox, which is an entity group for grouping regions that can be stored in a box 218 of a HEIF file 200. The LogicalOperateRegionEntityGroupBox is an extension of the EntityToGroupBox and has a grouping_type "lorg" 901. The LogicalOperateRegionEntityGroupBox includes a logical_operation indicated by a definition 902 as an additional parameter specific to the group type. The LogicalOperateEntityGroup can be defined as follows. Note that it can be easily understood that the name of the entity group and the 4CC used to identify the entity group can be another name and another 4CC.

[0121] The LogicalOperateEntityGroup is identified by the "lorg" grouping_type and indicates a set of regions used to create a combined region formed by combining the regions indicated by the included entities through a logical operation. For entity_id, if the region item, region metadata track or unif brand uses a unique ID, specify EntityToGroupBox to indicate other combined regions. More flexible region specification can be made by specifying other combined regions. Regions are formed by logical operations on these regions. The operator of the logical operation is provided by the logical_operation parameter. That is, the region obtained as a result of a logical operation on the region indicated by the entity_id using the operation method indicated by the logical_operation parameter is identified as the region indicated by the LogicalOperateRegionEntityGroup. Multiple LogicalOperateRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0122] logical_operation indicates the type of logical operation and is an unsigned integer indicating the logical operator applicable to the region entities included in the group. A value of 0 indicates a logical combination of regions using an OR operation. A value of 1 indicates a logical combination of regions using an AND operation. A value of 2 indicates a logical combination of regions using a NOT or NOR operation. A value of 3 indicates a logical combination of regions using an XOR operation. A value of 4 indicates a logical combination of regions using a NAND operation. A value of 5 indicates a logical combination of regions using an XOR operation. Other values ​​are reserved.

[0123] The number of regions indicated by an entity specified by entity_id and the number of entities are handled in the same manner as above, and methods can be defined. In addition, various region combination methods corresponding to the number of regions can be individually controlled by the value of flags included in the definition of LogicalOperateRegionEntityGroupBox. This behavior can be determined by the combination of the value of logical_operation and the value of flags.

[0124] The region indicated by the LogicalOperateRegionEntityGroup and the region indicated by the entity included in the LogicalOperateRegionEntityGroup can be individually associated with the image item and the region in the image can be identified. In addition, different annotation information can be associated by the method defined in the region annotation.

[0125] Note that in this embodiment, the operator of the logical operation is included in the LogicalOperateRegionEntityGroupBox and the method provided by the logical_operation parameter is described, but this can be controlled using flags. In addition, the operator can be associated with the group of LogicalOperateRegionEntityGroup as a property using the above-mentioned association structure of the item property. Figure 15 To illustrate the data structure in this case. Figure 15 The illustrated LogicalOperateRegionProperty ("lorg") is an example of property information capable of storing logical operator information. Figure 15 The LogicalOperateRegionProperty shown includes the 4-character code "lorg" shown in definition 1501, and this 4-character code is used to identify the LogicalOperateRegionProperty. The lorg property includes logical_operation 1502, which provides a logical operation method, and can be associated with a region item or region entity group. The definition of logical_operation 1502 is the same as the definition described for LogicalOperateRegionEntityGroupBox. The region item or region entity group associated with the lorg property uses the logical operation specified by logical_operation. Note that the region item can be a derived region item. In this embodiment, the logical operation adapted to the region item or region entity group is described. However, the logical operation can be adapted to different separately defined item or entity types. For example, NOT can be adapted to the ster entity group indicating a stereo pair, thereby adapting to the operation of replacing the left and right sides of the stereo pair. In addition, by defining the logical operation as an item property, the logical operation can be directly associated with the region item.

[0126] The above describes a group type that logically combines areas designated as entity groups. Next, a group type for showing associations between entities included in a group will be described.

[0127] Figure 10218 ] is a diagram showing the data structure of the InclusionRelationRegionEntityGroupBox, which is an entity group for grouping regions and can be stored in the box 218 of the HEIF file 200. The InclusionRelationRegionEntityGroupBox is an extension of the EntityToGroupBox and has a grouping_type "irrg" 1001. No additional parameters specific to the entity group type are defined. The InclusionRelationRegionEntityGroup (Inclusion Relation Region Entity Group) can be defined as follows. Note that it can be easily understood that the name of the entity group and the 4CC used to identify the entity group can be another name and another 4CC. For example, the CompoundRegionEntityGroupBox (Compound Region Entity Group Box) can be used as another name to indicate a combination and can be identified by "corg" or the like.

[0128] The InclusionRelationRegionEntityGroup is identified by the "irrg" grouping_type and indicates a set of regions used to create a composite region relationship that indicates an association between regions showing that the included entities maintain an inclusion relationship. For entity_id, if the region item, region metadata track, or unif brand uses a unique ID, specify an EntityToGroupBox to indicate additional composite regions. More flexible region specification can be made by specifying additional composite regions. The additional composite region is identified as a region that includes the second or subsequently specified entity_id in the region indicated by the first entity_id. Multiple InclusionRelationRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0129] The number of regions indicated by one entity specified by entity_id is recommended to be one (for example, region_count in the region item is 1). In this case, the number of specified entities (num_entities_in_group) is recommended to be two or more. On the other hand, if the number of regions indicated by one entity is two or more (for example, region_count in the region item is 2 or more) and it is the first entity_id, the following relationship can be defined: the first region specified as region information includes all subsequent regions and the regions indicated by the second and subsequent entity_ids. In this case, the number of specified entities (num_entities_in_group) can be one. Alternatively, it can be defined so that the region indicated by the first entity_id is handled as one region and includes the second and subsequent entity_ids. In this case, the number of specified entities (num_entities_in_group) is recommended to be two or more. In addition, if the number of regions indicated by the entities specified in the group is equal (region_count is equal in all entities), each region can be defined so that each region specified by the first entity_id includes the regions specified by the second and subsequent entity_ids in the order in which the regions are indicated. Various region combination methods corresponding to the number of regions can be individually controlled by the value of flags included in the definition of InclusionRelationRegionEntityGroupBox. The regions indicated by the InclusionRelationRegionEntityGroup and the regions indicated by the entities included in the InclusionRelationRegionEntityGroup can be individually associated with the image item. In addition, different annotation information can be associated by the method defined in the region annotation. Note that since the InclusionRelationRegionEntityGroup is a combination type that indicates the association between regions, if the regions indicated by the entities included in the InclusionRelationRegionEntityGroup are associated with the image, the InclusionRelationRegionEntityGroup may not be associated with the image.

[0130] Furthermore, entity groups can be defined as follows. A CompoundRegionEntityGroup ("corg") associates a primary region item with one or more region items. This indicates a logical configuration relationship between a primary object covered by the region of the primary entity and other objects covered by the regions described by one or more other entities, and that the primary object is logically formed from the other objects. For example, a compound region entity group can be used to associate a primary region corresponding to a vehicle with a region corresponding to a tire (wheel), thereby indicating that the vehicle is logically formed from the wheels.

[0131] Entities in a composite region entity group are region items. The number of entities in a composite region entity group must be at least two. The first entity_id value indicates the primary region item. The regions described by the second and subsequent entity_id values ​​indicate regions for the primary object logically formed by the target object.

[0132] This logical configuration relationship does not transmit information at the geometric level. The composite entity group informs that the main area formed by other areas sometimes includes other areas geometrically or sometimes does not include other areas.

[0133] As an adaptation example of InclusionRelationRegionEntityGroup (or CompoundRegionEntityGroup), adaptation can be performed in the following case: a tree is included as a subject in an image, the area of ​​the subject is identified, its fruit is included as a subject in the same image, and the area of ​​the subject is identified as another area. This can indicate which tree includes the area identified as the fruit. In addition, as the entire area for identifying a person (the first area item specified in InclusionRelationRegionEntityGroup), and the area included in the entire area (the second or subsequent area items specified in InclusionRelationRegionEntityGroup), the shapes of the hands, legs, and head, etc. are defined as area items, indicating the skeleton, and area items indicating the point shapes of the connection points between the joints are defined and included in the entity group. This makes it possible to describe information such as person detection, skeleton estimation, posture estimation, and action prediction detection, and also describe the relationship between areas. For example, for the area specified first, the person is simply detected, and the name, etc. are associated as annotations. Even for the connection points between bones or joints, the positions of each joint or body part are indicated, and annotations related to actions such as running or stretching a hand can be added to the group as annotations for the relationships indicated using EntityGroup. By combining this information, information detected by different AI algorithms, etc. can be added as separate annotations.

[0134] Figure 11 is a diagram showing the data structure of a TouchRelationRegionEntityGroupBox, which is an entity group for grouping regions that can be stored in a box 218 of a HEIF file 200. The TouchRelationRegionEntityGroupBox is an extension of the EntityToGroupBox and has a grouping_type "tcrg" 1101. No additional parameters specific to the entity group type are defined. The TouchRelationRegionEntityGroup (touch relationship region entity group) can be defined as follows. Note that it can be easily understood that the name of the entity group and the 4CC used to identify the entity group can be another name and another 4CC.

[0135] A TouchRelationRegionEntityGroup is identified by a "tcrg" grouping_type and indicates a set of regions used to create a combined region relationship that indicates an association between regions showing that the included entities maintain a contact relationship. The number of entity_ids is two, and if a region item, region metadata track, or unif brand uses a unique ID, an EntityToGroupBox is specified to indicate additional combined regions. More flexible region specification can be made by specifying additional combined regions. The two regions in the image associated with the entity group indicate that the subject objects maintain a contact relationship. This does not simply indicate that the regions are in contact or overlap with each other, but rather indicates that the subject objects in the image maintain a contact relationship. Multiple TouchRelationRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0136] The number of regions indicated by one entity specified by entity_id is recommended to be one (for example, region_count in the region item is 1). On the other hand, if the number of regions included in entity_id is two, the number of entities specified (num_entities_in_group) may be one, and in this case, this indicates that the two regions included in one entity maintain a contact relationship.

[0137] If the number of regions indicated by an entity is two or more (for example, region_count in the region item is 2 or more), or if the number of entities is two or more, it can be defined that all regions of the first entity and all regions indicated by the second region maintain a contact relationship. In addition, if the number of regions indicated by the entities specified in the group is equal (region_count in all entities is equal), it can be defined so that each region indicates that the region specified by the first entity_id and the region indicated by the second entity_id or subsequent entity_id maintain a contact relationship in the order of the indicated regions. Various region combination methods corresponding to the number of regions can be individually controlled by the value of flags included in the definition of TouchRelationRegionEntityGroupBox. The region indicated by TouchRelationRegionEntityGroup and the region indicated by the entity included in TouchRelationRegionEntityGroup can be individually associated with the image item. In addition, different annotation information can be associated by the method defined in the region annotation. Note that since TouchRelationRegionEntityGroup is a combination type indicating an association between regions, if each region indicated by entities included in the TouchRelationRegionEntityGroup is associated with an image, the TouchRelationRegionEntityGroup may not be associated with the image.

[0138] As an adaptation example of TouchRelationRegionEntityGroup, adaptation can be performed in the following cases: a person is included as a subject in an image, the area of ​​the subject is identified, a ball held by the person is included as a subject in the same image, and the area of ​​the subject is identified as another area. The area entities indicating the person and the ball are stored as a group. This makes it possible to identify that the person is holding the ball. For example, it can be identified whether the ball overlaps as an area after being thrown, or whether the ball is held in the hand before being thrown. In this embodiment, a grouping type indicating a contact relationship is described. However, a grouping type indicating non-contact can be used. In addition, both contact and non-contact relationships can be defined by defining flags or additional parameters in TouchRelationRegionEntityGroupBox. Figure 12218 , which is an entity group for grouping regions that can be stored in box 218 of HEIF file 200. DepthOrderRelationRegionEntityGroupBox is an extension of EntityToGroupBox and has grouping_type "dorg" 1201. DepthOrderRelationRegionEntityGroupBox includes depth_direction indicated by definition 1202 as an additional parameter specific to the group type. DepthOrderRelationRegionEntityGroup (depth order relationship region entity group) can be defined as follows. Note that it can be easily understood that the name of the entity group and the 4CC used to identify the entity group can be another name and another 4CC.

[0139] The DepthOrderRelationRegionEntityGroup is identified by the "dorg" grouping_type and indicates a collection of regions used to create a composite region relationship that is used to indicate the association between regions using the front / back relationship of the foreground / background of the included entities. For entity_id, if the region item, region metadata track, or unif brand uses a unique ID, specify an EntityToGroupBox to indicate additional composite regions. More flexible region specification can be made by specifying additional composite regions. The entity_id values ​​indicating that the regions in the DepthOrderRelationRegionEntityGroup are listed in depth_direction order. Multiple DepthOrderRelationRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0140] depth_direction indicates the direction of depth (foreground / background) and is an unsigned integer indicating the direction in the depth direction of the region entities included in the group. A value of 0 indicates that the region entities are listed from the foreground to the background. A value of 1 indicates that the region entities are listed from the background to the foreground. Other values ​​are reserved.

[0141] The number of regions indicated by one entity specified by entity_id is recommended to be one (for example, region_count in the region item is 1). In addition, the number of specified entities (num_entities_in_group) is recommended to be two or more. On the other hand, if the number of regions indicated by one entity is two or more (for example, region_count in the region item is 2 or more), this indicates that the multiple regions included in one entity are sequentially specified in the direction indicated by depth_direction in the storage order. As another method, the multiple regions included in one entity can be defined as regions at the same position (that is, at the same depth). If multiple regions are included in one entity, the number of entities included in the group (num_entities_in_group) can be one. Alternatively, if the number of regions indicated by the entities specified in the group is equal (region_count in all entities is equal), it can be defined so that these regions are arranged sequentially in the direction indicated by depth_direction for each region specified by entity_id in region count order. The various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of DepthOrderRelationRegionEntityGroupBox. In addition, additional parameters can be defined and specified to indicate whether the regions have the same depth or the next depth. The regions indicated by the DepthOrderRelationRegionEntityGroup and the regions indicated by the entities included in the DepthOrderRelationRegionEntityGroup can be individually associated with image items. In addition, different annotation information can be associated by the method defined in the region annotation. Note that since the DepthOrderRelationRegionEntityGroup is a combination type that indicates the association between regions, if the regions indicated by the entities included in the DepthOrderRelationRegionEntityGroup are associated with the image, the DepthOrderRelationRegionEntityGroup may not be associated with the image.

[0142] Figure 13is a diagram showing the data structure of a ForbidRelationRegionEntityGroupBox, which is an entity group for grouping regions that can be stored in a box 218 of a HEIF file 200. ForbidRelationRegionEntityGroupBox is an extension of EntityToGroupBox and has a grouping_type "fbrg" 1301. Additional parameters specific to the entity group type are not defined. The ForbidRelationRegionEntityGroup (forbidden relationship region entity group) can be defined as follows. Note that it can be easily understood that the name of the entity group and the 4CC used to identify the entity group can be another name and another 4CC. For example, OclusionRegionEntityGroupBox (occlusion region entity group box) can be used as another name to indicate a combination and can be identified by "ocrg" or the like.

[0143] The ForbidRelationRegionEntityGroup is identified by the "fbrg" grouping_type and indicates a set of regions used to create a combined region relationship, which is used to indicate an association between regions showing a relationship in which the second specified entity is partially or completely hidden by the first specified entity. The number of entity_ids is two, and if the region item, region metadata track, or unif brand uses a unique ID, an EntityToGroupBox is specified to indicate other combined regions. More flexible region specification can be achieved by specifying other combined regions. It indicates that two regions in the image associated with the entity group have a relationship in which one region hides the object indicated by the other region. In this case, for example, if the object indicated by the other region is completely hidden by the object indicated by the one region, the hidden object does not appear as a subject in the image. That is, this indicates that a hidden region is set based on information that can be estimated from another image, etc., and that the region is hidden by the object indicated by the other region. Multiple ForbidRelationRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0144] The recommended number of regions indicated by an entity specified by entity_id is one (for example, region_count in the region entry is 1). On the other hand, if the number of regions included in entity_id is two, the number of entities specified (num_entities_in_group) can be one. In this case, this indicates that, of the two regions included in the entity, the object in the image region indicated by the first region hides the object in the image region indicated by the second region. Furthermore, if the number of regions indicated by an entity is two or more (for example, region_count in the region entry is 2 or more), or if the number of entities is two or more, it can be defined that the region indicated by the first entity and all regions indicated by the second and subsequent regions are hidden. Furthermore, if the number of regions indicated by the entities specified in the group is equal (region_count in all entities is equal), it can be defined to indicate that the regions have the following relationship: in the order in which the regions are indicated, the region specified by the first entity_id hides the regions specified by the second and subsequent entity_ids. Various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of ForbidRelationRegionEntityGroupBox. The regions indicated by a ForbidRelationRegionEntityGroup and the regions indicated by the entities included in the ForbidRelationRegionEntityGroup can be individually associated with an image item. Furthermore, different annotation information can be associated using methods defined in region annotations. Note that since the ForbidRelationRegionEntityGroup is a composite type that indicates an association between regions, if the regions indicated by the entities included in the ForbidRelationRegionEntityGroup are associated with an image, the ForbidRelationRegionEntityGroup may not be associated with the image.

[0145] In the above definitions of entity groups corresponding to each combination type, a method of using flags in each definition to define the number of areas held by the area item as the input area is described. As another combination area description method, refer to Figure 27 The following method is described for realizing this by defining data structures of different entity groups without using control using flags values ​​even when multiple areas are indicated by an entity group in one area item or a combined area designated as an input area. Figure 27In the example shown, a combination area indicating an inclusion relationship between areas is described as an example. Even other combination types can be defined by the same mechanism. Figure 27 The definition of CompoundEntityToGroupBox shown illustrates a data structure that may be stored in a GroupListBox ("grpl") box in box 218 as a data structure for a composite region entity group.

[0146] Figure 27 The data structure shown is an example of an extension of a CompoundRegionEntityGroup ("corg"). The extended CompoundRegionEntityGroup ("corg") has a configuration that enables the description of relationships between multiple compound region groups in one compound region entity group. In a region item, when multiple regions are described in one region item, these regions can be described by specifying a value of 2 or more in region_count. Like this method, this definition enables the description of multiple regions even in relationships between regions. In order to identify the first input region (main region) for a relationship of as many regions as the number (group_count) defined in the extended compound region entity group, it is specified by the index (compound_region_item_index) of each input region item and the index (compound_region_index) in the list of regions defined by the input region item. Furthermore, in order to identify the second and subsequent input regions for each region relationship, the number of input regions (comprised_count) is specified for each region relationship. In addition, the region input for each relationship is specified by the index of each input region item (region_item_index) and the index of the region defined by the input region item in the list (region_index).

[0147] Here, version is defined as 0 or 1. In future expansions, new version values ​​may be specified for versions. Here, flags is set to 0, and all flag values ​​are reserved. Here, region_count is the number of region relationships defined by the composite region entity group. Here, group_count is the number of region groups defined by the composite region entity group. Here, compound_region_item_index is the index of a region item in the list of region items specified by the composite region entity group, and indicates the region item that includes one of the first input regions (primary regions) of the composite region. Here, compound_region_index is the index of a region in the region item indicated by the compound_region_item_index field that is one of the composite regions. Here, comprised_count is the number of regions that form one composite region. Here, region_item_index is the index of a region item in the list of region items specified by the composite region entity group, and indicates the region item that includes one of the second and subsequent input regions of the composite region. Here, region_index is the index of a region in the region item indicated by the region_item_index field that is one of the composite regions.

[0148] By defining the data structure of a combined region entity group in this manner, a combined region entity group can be easily formed regardless of the number of regions in the region item used as the input region. Furthermore, if the same annotation needs to be added to multiple combined regions, a common annotation can be added even if annotations are not added for each combined region entity group, as with existing region items. The data structure can also be adapted to other combination types. In a combined region entity group with a parameter like RegionLogicalDerivation, the same definition can be performed by adding this parameter to the data structure.

[0149] Here, we will refer to 16A to 16C An example of an image file output by the image file storage device 100 according to this embodiment is described. Note that in this embodiment, an image file is configured such that a region obtained by combining region entries is stored in the file as a file data structure by two CombinedRegionEntityGroups.

[0150] exist Figures 16A to 16CIn the example shown, as shown by the description 1604 corresponding to the block 204, descriptions 1630 and 1631 corresponding to the HEVC encoded data (HEVC image data) 241 to 242 are stored. In addition, an image file is exemplarily shown in which descriptions 1632 and 1633 corresponding to the region information (region item data) 243 to 244 and a description 1634 corresponding to the Exif data block 245 are stored. As shown by the descriptions 1632 and 1633, the region information 243 to 244 conforms to the Figure 3A and Figure 3B The definitions shown in FIG. 1 and FIG. 2 each specify a rectangular region for a reference space having an image size of 4032 pixels × 3024 pixels. In addition, the coordinates (x0, y0) of the reference point in the reference space of the region and the image size (w0 × h0) and the coordinates (x7, y7) of the reference point and the image size (w7 × h7) are specified for the region.

[0151] Description 1601 corresponds to block 201. As the type value major-brand of the brand definition conforming to the HEIF file, "mif1" is stored, and as the type value compatible-brands of the compatible brand definition, "heic" is stored.

[0152] Description 1602 corresponds to block 202 and stores “unif” as a type value of compatible-brands that conforms to the extended brand definition of the HEIF file. This indicates that the ID value at the file level is a uniquely identifiable value.

[0153] Next, description 1603 corresponding to block 203 shows various metadata information describing the non-timing data stored in the output file example.

[0154] Description 1610 corresponds to box 211 and specifies that the handler type of MetaDataBox (meta) is “pict.” Description 1611 corresponds to box 212 , stores 5 as item_ID, and specifies the ID of an image to be displayed as a first priority image.

[0155] Description 1612 corresponds to box 214 and indicates item information (item ID (item_ID) and item type (item_type)) for each item. Each item can be identified by item_ID, and it shows what kind of item is identified by item_ID. 16A to 16CIn the example shown, since 14 items are stored, entry_count is 14, 14 types of information are arranged in description 1612, and the item ID and item type are specified for each of these 14 types of information. In the image files shown in these figures, the first to fourth information corresponding to descriptions 1640 to 1641 are HEVC coded image items of type hvc1. In addition, the fifth information indicated by description 1642 is a grid-derived image item of item type grid. The sixth information indicated by description 1643 is an HEVC coded image item of item type hvc1 used as a thumbnail image. The seventh to thirteenth information corresponding to descriptions 1644 to 1645 are region items of item type rgan indicating a region. The fourteenth information corresponding to description 1646 is a metadata item of item type Exif indicating an Exif data block.

[0156] Description 1613 corresponds to box 213 and specifies the storage location and data size of each item in the HEIF file. For example, a coded image item with item_ID 1 indicates that the data is located at offset O1 in the file and is L1 bytes in size, and specifies the location of the data in mdatBox.

[0157] Description 1614 corresponds to box 215 and indicates a reference relationship (association) between items. The item reference indicated by description 1650 specifies a dimg indicating that the reference type is a derived image. In addition, it is indicated that the HEVC coded image items of item_ID 1 to 4 specified by to_item_ID are referenced by the grid derived image item of item_ID 5 specified by from_item_ID. Thus, the coded image items of item_ID 1 to 4 are designated as input images to the derived image items. In addition, the item reference indicated by description 1651 specifies a thmb indicating a thumbnail relationship as a reference type. It is indicated that the grid derived image item of item_ID 5 specified by to_item_ID is referenced by the HEVC coded image item of item_ID 6 specified by from_item_ID. This indicates that the HEVC coded image item of item_ID 6 is a thumbnail of the grid derived image item of item_ID 5. The item references indicated by descriptions 1652, 1653, 1654, 1655, and 1656 specify a cdsc, indicating a content description relationship, as the reference type. Description 1652 indicates that the grid-derived image item with item_ID 5 specified by to_item_ID is referenced by the region information item with item_ID 50 specified by from_item_ID. Thus, the region information item with item_ID 50 indicates a partial region within the grid-derived image item with item_ID 5. Similarly, description 1653 indicates that the grid-derived image item with item_ID 5 specified by to_item_ID is referenced by the region information item with item_ID 56 specified by from_item_ID. Thus, the region information item with item_ID 56 indicates a partial region within the grid-derived image item with item_ID 5. Description 1654 indicates that the grid-derived image item of item_ID 5, specified by to_item_ID, is referenced by the Exif data block item of item_ID 1000, specified by from_item_ID. This indicates that the Exif data block item of item_ID 1000 describes information about the image capture time of the grid-derived image item of item_ID 5. In description 1655, item_ID 100, specified in from_item_ID, is interpreted as the group ID of an entity group by the UNIF brand designation in the etyp box of description 1602. This indicates that the grid-derived image item of item_ID 5, specified by to_item_ID, is referenced by the region information entity group of group_ID 100, specified by from_item_ID. Therefore, the combined region indicated by the region information entity group of group_ID 100 indicates a partial region within the grid-derived image item of item_ID 5. Note that a configuration that associates entity groups with image items is not essential.That is, if an image item is associated with a region item that forms an entity group, a combined region in the image can be identified without associating the entity group with the image item. Similarly, in description 1656, item_ID 101 specified in from_item_ID is interpreted as the group ID of the entity group by the unif brand designation in the etyp box of description 1602. It is indicated that the grid-derived image item of item_ID 5 specified by to_item_ID is referenced by the region information entity group of group_ID 101 specified by from_item_ID. Therefore, the combined region indicated by the region information entity group of group_ID 101 indicates a partial region in the grid-derived image item of item_ID 5.

[0158] Description 1615 corresponds to box 217, and specifies the data structure of the grid-derived image item of item ID 5. In the HEIF file according to the present embodiment, 2 is specified in rows_minus_one of the grid-derived image, and 2 is specified in columns_minus_one. This indicates that 2×2 input images are formed as grid-derived images. Note that, as the number of input images referenced by the item indicated by description 1650, values ​​consistent with each other are specified with respect to the item count "4" specified in ref_count and the number of grids (that is, 2×2 (4)). 4032 is specified in the number or output_width, and 3024 is specified in output_height. The image is formed as an image with a width of 4032 pixels and a height of 3024 pixels.

[0159] Description 1616 and description 1617 correspond to box 218 and specify an entity group. In the HEIF file according to the present embodiment, two CombinedRegionEntityGroups are specified, and description 1616 and description 1617 correspond to boxes 231 and 232, respectively. In description 1616, 100 is specified in group_id, and the IDs of the five region information items of item_id 50, 51, 52, 54, and 55 are specified as entity_id. Therefore, the regions indicated by item_id 50, 51, 52, 54, and 55 are combined into a combination type cbrg, and all regions included in any one of these five regions are identified as the region indicated by CombinedRegionEntityGroup. In description 1617, 101 is specified in group_id, and the IDs of the two region information items of item_id 53 and 56 are specified as entity_id. Therefore, the regions indicated by item_ids 53 and 56 are combined as a combination type of cbrg, and all regions included in either of the two regions are identified as a region indicated by CombinedRegionEntityGroup.

[0160] Description 1618 corresponds to block 216 and includes description 1620 corresponding to block 221 and description 1621 corresponding to block 222 .

[0161] Description 1620 lists the attribute information available in the item or entity group as entry data. As shown in the figure, description 1620 includes not only the first and second entries indicating the encoding parameters and the third, fourth, and fifth entries each indicating the display pixel size of the item, but also the sixth to fourteenth entries indicating annotation descriptive information. As annotation descriptive information, lang is American English (en-US) in all entries, "human" is set as name in the sixth to twelfth entries, and "family" is set in the thirteenth and fourteenth entries. As description, "Hanako," "Taro," "Shohei," "Mike," "Naomi," "Ginnji," and "Susan" are set in sequence to represent the names of the people indicated by the area in the sixth to twelfth entries. In addition, "Taro family" and "Mike family" are set as descriptions in the thirteenth and fourteenth entries to indicate the family. In tags, "AutoRecognition" is set to add a tag indicating that the area is an automatically recognized area. In the entry data of the description 1621 corresponding to the box 222, the attribute information listed in the description 1620 is associated with the item or entity group stored in the HEIF file. 16A to 16C In the example shown, a common "ispe" (property_index is 4) is associated with the image items of item_IDs 1 to 4, and this indicates that the images have the same image size of 2016 pixels × 1512 pixels. Similarly, a common "hvcC" (property_index is 1) is associated with the image items of item_IDs 1 to 4, and this indicates that the same encoding parameters are used. On the other hand, different item properties are associated with the items or entity groups of item_IDs 5 to 15. "ispe" (property_index is 3) is associated with the derived image item of item_ID 5, and this indicates that the image size is 4032 pixels × 3024 pixels.

[0162] "ispe" (property_index is 5) is associated with the image item of item_ID 6, indicating that it is an image with an image size of 768 pixels x 576 pixels. Similarly, "hvcC" (property_index is 2) is associated with the image item of item_ID 6, indicating encoding parameters. "udes" (property_index is 6) is associated with the region information item of item_ID 50, indicating annotation information for the region. "udes" (property_index is 12) is associated with the region information item of item_ID 56, indicating annotation information for the region. Thus, adding annotation information to a region is achieved by associating the "udes" property. Furthermore, "udes" (property_index is 13) is associated with the region information entity group of item_ID (group_id) 100, indicating annotation information for a combined region. "udes" (property_index is 14) is associated with the region information entity group of item_ID (group_id) 101, indicating annotation information for a combined region. Thus, adding annotation information to the combined area is achieved by associating the "udes" property with the entity group. Note that since no item property is associated with the Exif data block item of item_ID 1000, the corresponding entry information is not stored.

[0163] Next, refer to Figure 17 The flowchart of FIG. 1 is used to describe the generation process of a HEIF file having a file structure in which regions in an image are identified by combining region information for identifying the regions, and annotation information can be stored in association with each region.

[0164] Note that according to Figure 17 The processing of the flowchart is a processing executed by the CPU 101 executing various control processing using the computer program and data read out from the ROM 102 or the nonvolatile memory 110 to the RAM 103. Note that according to Figure 17 The generation process of the flowchart is started, for example, in response to the CPU 101 detecting that the user has input an instruction associated with imaging by operating the operation input unit 107. However, the process for starting the image capturing process according to Figure 17 The event that triggers the processing of the flowchart is not limited to a specific event.

[0165] In step S1701, the CPU 101 controls the imaging unit 104 and the image processing unit 105, and acquires an image (image data) to be stored in the HEIF file. Note that the image acquisition method is not limited to a specific acquisition method. For example, the image can be acquired from the non-volatile memory 110, or can be acquired from an external device via the communication unit 108.

[0166] In step S1702, under the control of the CPU 101, the recognition processing unit 114 performs detection processing for detecting the object captured as the subject in the image acquired in step S1701. Detection processing is processing for detecting an object, such as a person or an object, from an image. Note that the method for detecting an object from an image is not limited to a specific method.

[0167] For example, by using matching processing of an image of a specific person registered in advance, processing for checking whether the person is captured in the image acquired in step S1701 and detecting the person may be performed.

[0168] Furthermore, according to the pre-registered contents, not only a specific person but also the attribute information of the person may be included, and detection processing of detecting an object from an image may be performed by further using the attribute information.

[0169] Furthermore, recognition processing unit 114 can detect associations based on multiple detection results, or can perform detection processing for combined areas. For example, when detecting a human body, recognition processing unit 114 can individually detect the head, body, right hand, left hand, right leg, and left leg, and then detect the human body by combining these. The detection results vary depending on the recognition processing algorithm executed by recognition processing unit 114. Furthermore, instead of simply performing detection using recognition processing unit 114, the user can detect the areas of each object while confirming the image using image editing software, etc.

[0170] In step S1703 , under the control of the CPU 101 , the generation unit 113 generates region information for the region of the object detected by the recognition processing unit 114 , and stores the generated region information in the output buffer provided in the RAM 103 .

[0171] In step S1704, under the control of the CPU 101, the encoding / decoding unit 111 performs encoding processing on the image acquired in step S1701 to generate encoded image data, and stores the encoded data in the output buffer. Furthermore, under the control of the CPU 101, the metadata processing unit 112 generates metadata to be stored in the image file, and stores the generated metadata in the output buffer.

[0172] In step S1705, under the control of the CPU 101, the metadata processing unit 112 generates metadata when storing the region information generated by the generation unit 113 for the region of the object detected by the recognition processing unit 114. At this time, the metadata processing unit 112 determines whether it is necessary to group the region items of the detection region (region where the object was detected) and store them as a combined region. This may include data conversion processing such as optimization of the region information generated in step S1703. For example, if it is determined that the region items can be stored efficiently by storing them as a combined region for NOT, the region information is changed and region information indicating the reversed region is generated.

[0173] As a result of the determination, if it is determined that the area items of the detection area need to be grouped and stored as a combined area, the process proceeds to step S1706. On the other hand, if it is not determined that the area items of the detection area (area where the object is detected) need to be grouped and stored as a combined area, the process proceeds to step S1709.

[0174] In step S1706, the CPU 101 determines the grouping type when combining regions, for example, the above-mentioned EntityToGroupBox.

[0175] In step S1707 , the CPU 101 generates group information for grouping the area information corresponding to the grouping type decided in step S1706 .

[0176] In step S1708, the CPU 101 generates metadata for adding area annotation information (annotation) to the group information generated in step S1707 and associates the metadata with the group information. As described above, this is achieved by using the udes property to generate text description information and associating the text description information as an item property with the entity group.

[0177] In step S1709 , the CPU 101 generates metadata for adding region annotation information to the region detected as the individual region, and associates the metadata with the region information.

[0178] In step S1710, the CPU 101 generates a HEIF file for storing the generated metadata and encoded data. More specifically, the final metadata of the HEIF file is configured based on the information stored in the output buffer by the metadata processing unit 112. Then, the information of the box 201 associated with the HEIF file, the information of the box 203 storing the final metadata, and the information of the box 204 storing the image and area information, etc. are combined. The CPU 101 writes the HEIF file generated by the combination from the RAM 103 to the non-volatile memory 110 and stores the HEIF file. Note that the output destination of the HEIF file is not limited to the non-volatile memory 110, and the HEIF file can be sent to an external device via the communication unit 108, for example.

[0179] As described above, the image file storage device 100 according to this embodiment performs object detection processing on an image to be stored in an image file and generates region information for identifying an object as a subject in the image. The image file storage device 100 then generates metadata for identifying the generated region information as a combined region, thereby making individual regions and combined regions identifiable. Annotation information is added to each identifiable region as metadata, allowing region annotations to be added to each region and combined region.

[0180] Note that in this embodiment, a mode in which the images to be stored in the image file are obtained by imaging using the imaging unit 104 is described, but the present invention is not limited to this. Of course, the series of images may be images pre-stored in the ROM 102 or the non-volatile memory 110, or images received via the communication unit 108. In this case, the images may include a HEIF file storing a single still image, or may include a still image file such as a JPEG file. Alternatively, the images may be images encoded in a HEIF file storing a plurality of still images, or may be unencoded RAW images.

[0181] Next, the mode for reproducing a HEIF file will be described. It will be readily understood that a HEIF file can be used in any device other than the image file storage device 100 that generated the HEIF file. When used, a processor such as a CPU of the device reads metadata of the HEIF file being processed, thereby allowing the encoded data stored in the HEIF file to be reproduced / changed.

[0182] As a mode of image file reproduction processing, reference will be made to Figure 18 The reproduction process associated with the reproduction (display) of the HEIF file in the image file storage device 100 is explained with reference to the flowchart of FIG.

[0183] Note that according to Figure 18The processing of the flowchart is a processing executed by the CPU 101 executing various control processing using the computer program and data read out from the ROM 102 or the nonvolatile memory 110 to the RAM 103. Figure 18 The reproduction process of the flowchart is started, for example, in response to the CPU 101 detecting that the user has performed an operation input associated with a HEIF file reproduction instruction by operating the operation input unit 107 in a state where the image file storage device 100 is set in the reproduction mode. Figure 18 The event that triggers the processing of the flowchart is not limited to a specific event.

[0184] In step S1801, the CPU 101 acquires the image file (HEIF file) that is the target of the reproduction instruction. The image file acquisition method is not limited to a specific acquisition method. For example, the image file can be acquired from the non-volatile memory 110, or can be acquired from an external device via the communication unit 108.

[0185] In step S1802 , under the control of the CPU 101 , the metadata processing unit 112 acquires metadata and encoded data from the HEIF file acquired in step S1801 , and analyzes the metadata, thereby acquiring the configuration of the HEIF file.

[0186] In step S1803, the CPU 101 specifies the representative item based on the information of the frame 212 of the metadata acquired in step S1802, and causes the encoding / decoding unit 111 to decode the encoded data 241 indicated by the representative item. Thus, the encoding / decoding unit 111 acquires the corresponding encoded data (encoded data 241) from the metadata associated with the image item designated as the representative image, and stores the decoded image obtained by decoding the encoded data in a buffer on the RAM 103.

[0187] In step S1804 , under the control of the CPU 101 , the metadata processing unit 112 acquires region metadata associated with the reproduction target image designated as the representative image.

[0188] In step S1805, the metadata processing unit 112 determines whether the regional metadata acquired in step S1804 are associated as a group under the control of the CPU 101. Whether the regional metadata acquired in step S1804 are associated as a group can be determined by analyzing the blocks 218 and 215.

[0189] As a result of the determination, if the region metadata acquired in step S1804 is associated as a group (if grouped region information associated with the reproduction image data is stored), the process proceeds to step S1806.

[0190] On the other hand, if the region metadata acquired in step S1804 is not associated as a group (if the region metadata associated with the reproduction target image is not grouped), the process advances to step S1807.

[0191] In step S1806 , under the control of the CPU 101 , the generation unit 113 generates region information corresponding to the grouping type of the grouped region information, and stores the generated region information in a buffer on the RAM 103 .

[0192] In step S1807 , the CPU 101 superimposes the associated region information and the combined region information generated in step S1806 on the decoded image (representative image) stored in the buffer on the RAM 103 , and displays on the display unit 106 .

[0193] In step S1808, if region annotation information corresponding to each region has been added, the CPU 101 displays the region annotation information (annotation) so that the association with the region information can be recognized. Note that in this embodiment, a method for displaying the region annotation information and the region information in a superimposed manner along with the representative image is described. However, the display of the region annotation information and the region information can be optional. In other words, whether or not to display these can be determined based on instructions from a UI operation, etc.

[0194] Figure 19 、 Figure 20 and Figure 21 An example of display of the region information and the region comment information displayed together with the representative image in steps S1807 and S1808 is shown. Figure 19 、 Figure 20 and Figure 21 The illustrated images indicate examples of UI screens displayed in steps S1807 and S1808 .

[0195] The shaded area indicated by 1901 indicates an area detected as a head area of ​​a human body. Also, 1911 indicates area annotation information associated with the area 1901. The area shape is an ellipse, and "Head" is associated as the area annotation information.

[0196] The dotted polyline indicated by 1902 indicates the region detected as the right arm region of the human body. In addition, 1912 indicates region annotation information associated with the region 1902. The region shape is a polyline, and "right arm" is associated as the region annotation information.

[0197] The dotted polyline indicated by 1903 indicates a region detected as the left arm region of the human body. Also, 1913 indicates region annotation information associated with the region 1903. The region shape is Polyline, and "Left arm" is associated as the region annotation information.

[0198] The mesh area indicated by 1904 indicates an area detected as a body area of ​​a human body. In addition, 1914 indicates area annotation information associated with the area 1904. The area shape is rectangular (Rectangular), and Body is associated as the area annotation information.

[0199] The dotted polyline indicated by 1905 indicates a region detected as the right leg region of the human body. Also, 1915 indicates region annotation information associated with the region 1905. The region shape is Polyline, and Right leg is associated as the region annotation information.

[0200] The dotted polyline indicated by 1906 indicates a region detected as the left leg region of the human body. In addition, 1916 indicates region annotation information associated with the region 1906. The region shape is Polyline, and "Left leg" is associated as the region annotation information.

[0201] Regions 1901 to 1906 are combined into a combined region. In this case, the combined region can be identified as one region by grouping using the cbrg region entity group. The region annotation information associated with the combined region is indicated by 1921, and "Human" is associated with cbrgRegionGroup7 (cbrg region group 7) as the region annotation information. cbrgRegionGroup7 is the region obtained by combining Region1 to Region6. This makes it possible to detect regions as parts individually. And combined region representation is achieved without associating the combined region of these regions as a single region. In addition, since the combined region can be identified, the region annotation can also be associated with the combined region.

[0202] Then, in Figure 20 In the UI screen shown, the areas (individual areas) 2001 to 2006 indicating the parts are Figure 19 The areas 1901 to 1906 shown are identical. Figure 20Rather than illustrating a method for generating a combined region by combining parts, an example is shown in which region data 2007 is generated for separately identifying a region indicating a human body as a combined region. Region annotation information is associated with the human body region, and as indicated by 2017, the region shape is Rectangular, and "Human" is associated as the region annotation information. Thus, while the regions of parts and the region indicating the entire body can be identified as separate regions and individually associated with region annotation information, the association between the parts and the entire body region cannot be shown. For this reason, the InclusionRelationRegionEntityGroup, which indicates an inclusion relationship between regions, allows the association between regions to be identified as a combined region. This inclusion relationship is shown as a combined region indicating that regions 2001 to 2006 are included in region 2007. This inclusion relationship can be identified as a combined region by grouping using the irrg region entity group, and the region annotation information associated with the combined region is indicated by 2021. Regarding the combined region, "Parts of Human" is associated as region annotation information with irrgRegionGroup10, which is a region that includes Regions 1 to 6 in Region 7. Therefore, it is possible to detect regions of each part individually, detect regions indicating the whole individually, and identify combined regions indicating the association between regions obtained by combining these regions. Region annotation information can also be associated with the combined region.

[0203] Figure 21 is a diagram showing another example of adapting a cbrg entity group. Figure 21 The UI screen shown shows the 16A to 16C Examples of adaptation examples are shown.

[0204] Area 2101 is composed of 16A to 16C 2111 indicates the area annotation information associated with the area 2101 and shows the description "Hanako" indicated by the udes item property. 16A to 16C The sixth entry in the ipco box associated with item_ID 50.

[0205] Area 2102 is composed of 16A to 16C 2112 indicates the area annotation information associated with the area 2102 and shows the description "Taro" indicated by the udes item property associated with the item_ID 51. 16A to 16C The seventh entry in the ipco box.

[0206] Area 2103 is composed of 16A to 16C 2113 indicates the area annotation information associated with the area 2103 and shows the description "Shohei" indicated by the udes item property associated with the item_ID 52. 16A to 16C The eighth entry in the ipco box.

[0207] Area 2104 is composed of 16A to 16C 2114 indicates the area annotation information associated with the area 2104 and shows the description "Mike" indicated by the udes item property, which is associated with the item_ID 53. 16A to 16C The ninth entry in the ipco box.

[0208] Area 2105 is composed of 16A to 16C 2115 indicates the area annotation information associated with the area 2105 and shows the description "Naomi" indicated by the udes item property associated with the item_ID 54. 16A to 16C The tenth entry in the ipco box in .

[0209] Area 2106 is composed of 16A to 16C 2116 indicates the area annotation information associated with the area 2106 and shows the description "Ginnji" indicated by the udes item property, which is associated with the item_ID 55. 16A to 16C The eleventh entry in the ipco box.

[0210] Area 2107 is composed of 16A to 16C 2117 indicates the area annotation information associated with the area 2107 and shows the description "Susan" indicated by the udes item property. 16A to 16C The twelfth entry in the ipco box associated with item_ID 56.

[0211] Furthermore, as a cbrg combined region formed by combining these regions, in 16A to 16C Two cbrg entity groups in the grpl box are stored, and region annotation information associated with the combined region indicated by the group ID 100 of the cbrg entity group is indicated by 2122.

[0212] Regarding the area annotation information 2121, the areas 2101, 2102, 2103, 2105, and 2106 are combined to be identified as one area, and the area annotation information is associated with the one area. On the UI screen, these areas are shown as solid line areas, and the area annotation information is displayed by 2121. 16A to 16C As shown in the fourteenth entry in the ipma box, the udes property, which is the thirteenth entry in the ipco box, is associated with the cbrg entity group, which is the entity group of group ID 100. As the area comment information, the description "Taro family" indicated by the udes property is displayed.

[0213] Similarly, regarding the area annotation information 2122, the areas 2104 and 2107 are combined and identified as one area, and the area annotation information is associated with the one area. On the UI screen information, these areas are shown as dotted areas, and the area annotation information is displayed by 2122. 16A to 16C As shown in the fifteenth entry in the ipma box, the udes property, which is the fourteenth entry in the ipco box, is associated with the cbrg entity group, which is the entity group of group ID 101. As the area comment information, the description "Mike family" indicated by the udes property is displayed.

[0214] According to the above embodiment, when one or more area information for identifying an object associated with a part of an image and captured in the image is stored in an image file in association with the image, combination type information for combining the area information and combination area information for identifying the combined area corresponding to the combination type information are stored, thereby efficiently and flexibly specifying areas. That is, to date, in order to represent complex shapes as area information, each area needs to be represented as bit mask data, and the data size of the area information is large. However, according to the present embodiment, by combining multiple area information, complex areas can be specified as lightweight area information without using bit mask data. In addition, since areas can be combined and identified according to type, the association between separately specified areas can be shown. Accordingly, information between areas whose relationship has been difficult to show so far, or the association between areas that cannot be obtained without analyzing a large amount of area annotation information, can be efficiently shown.

[0215] Note that in this embodiment, regarding the combined area information corresponding to the combination type stored in the image file, grouping type information or parameters are used to specify the combination type when the area is used as an entity group. When a grouped entity group is associated with an image, the subject in the image can be identified by combining multiple areas. In addition, area annotation information can be associated with the combined area. However, the grouping type is merely an example. Other combination types can also be defined, and any combination type configured to combine areas for processing can be used. When grouping areas, other grouped combined areas can also be specified in the group. This enables more flexible area specification. In addition, when treating individual areas as groups, each area can also belong to different groups. Thus, a subject object can belong to multiple area groups and be identified, and area annotations can be added to each area. Note that the four-character code used to identify the grouping type of the combined area shown in the embodiment is merely an example, and any grouping type with an arbitrary four-character code reserved for each grouping type can be used.

[0216] [Second embodiment]

[0217] In the following embodiments, including this embodiment, differences from the first embodiment will be described, and unless otherwise specified, the configuration is assumed to be the same as in the first embodiment. In the first embodiment, as a method for identifying area information as a combined area, a method of grouping into entity groups along with grouping type information is described. However, this method is not limited to this.

[0218] In this embodiment, the structure of a derived item that handles derived information defined as an item will be used to illustrate a method for identifying a combined area. A derived item is a mechanism configured to perform other representations by performing operations on an item using metadata. For example, a derived image item achieves different image representations by performing operations on one or more input images such as the grid image shown in the first embodiment. A grid image is a derived image that forms an image composited into one image by arranging multiple input images as sub-images on a grid. In addition, image overlays and identity derived images are defined. In ISO / IEC23008-12, second edition FDIS, in a region item, an identity derived region item is defined as a region item after adapting the properties of a transformation item. An identity derived region item is defined as a mechanism configured to perform a conversion to a region before being associated with an image item. Since a conversion operation using a transformation property is defined as an operation after being associated with a normal image item, the conversion operation is defined as a mechanism configured to first convert a region item and then associate the region item with an image item. In this embodiment, a method of achieving this by defining a derived region item type capable of expressing a combined region as a derived region item will be described.

[0219] The difference between this embodiment and the first embodiment is that the generated HEIF file has Figure 22 The structure shown. Figure 1 Block diagram, Figure 3A and Figure 3B The RegionItem data structure shown and Figure 5 The properties of the udes shown are common to the first embodiment. Except that derived region terms are used instead of entity groups, Figure 17 Flowchart, Figure 18 Flowchart and Figure 19 、 Figure 20 and Figure 21 The UI shown is common. Figure 22 To illustrate a data structure of a HEIF file configured to identify a combined region corresponding to a combined type using a derived region item.

[0220] like Figure 22 As shown, the HEIF file 2200 generally includes the following three boxes (storage areas). The first box 2201 (FileTypeBox ("ftyp")) is the same as the box 201 described in the first embodiment. The HEIF file according to this embodiment does not include the box 202 (ExtendedTypeBox ("etyp")) required in the first embodiment, but may include the box 202.

[0221] In the second box 2202 (MetaBox ("meta")), as will be described in detail below, multiple types of information associated with the image are stored in different boxes.

[0222] The third box 2203 (MediaDataBox("mdat")) is common to the box 204 described in the first embodiment.

[0223] Note that if the HEIF file stores moving images, audio, or an image sequence, a MovieBox ("moov") (not shown) may be stored separately, which is common to the first embodiment.

[0224] A block 2211 (HandlerReferenceBox("hdlr")) stores a declaration of a handler type used to analyze the structure of the block 2202. The configuration of the block 2211 is common to that of the block 211 described in the first embodiment.

[0225] A box 2212 (PrimaryItemBox("pitm")) specifies an identifier (item ID) of an image that is a representative item of image items stored in the HEIF file 2200. The configuration of the box 2212 is common to that of the box 212 described in the first embodiment.

[0226] Box 2213 (ItemLocationBox("iloc")) stores information indicating the storage location of information items including image items in the HEIF file 2200. Box 2213 generally describes the storage location of the image item as a byte offset relative to the top of the HEIF file 2200 and a data length relative to the top. That is, through the information in box 2213, the locations of the encoded data 2231 to 2232, the region information 2233 to 2234, and the Exif data block 2235 stored in box 2203 can be specified. For derived items, the information in box 2213 indicates that there is no data in box 2203. In this case, it is indicated that there is no data structure or that the data of the derived item is stored in box 2217 in box 2202.

[0227] Box 2214 (ItemInfoBox("iinf")) defines basic information (item information) such as an item ID and an item type indicating the type of the item for all items included in the HEIF file 2200. In the item information, not only image items such as coded image items and derived image items are specified, but also region items indicating regions in an image and Exif information items indicating Exif data blocks are specified. In addition, information related to derived region items indicating combined regions is defined. As derived region items indicating combined regions, combination type information is provided as the item type of the items designated as each derived region item. For example, the combination type RegionCombinedDerivation (region combined derivation) is a derived region item indicated by the item type "cbrg". When "cbrg" is specified as the item type, it is specified that the item is a region item to which the RegionCombinedDerivation operation is adapted. In addition, information related to region items that are input to form combined regions is provided by box 2215 shown next. This enables combinations corresponding to area items and combination types of area items, or combinations indicating associations between area items that are input as derived area items. Note that if the number of area items input as a derived area item is one, the derived area item can use the item type "iden". In this case, the area item can be derived using a separately defined Transformative item property. An identity derived area item is a derived area item with an item_type value of "iden", and the derived area item does not have an item body. Details of the definition of derived area items for each combination type of the "cbrg" derived area item or other derived area items will be described later.

[0228] Box 2215 (ItemReferenceBox("iref")) stores information describing the association between items stored in the HEIF file 2200. In a mode where the image item is a captured image, box 2215 is used to describe the association between the image item and its item of camera information (Exif data, etc.). In a mode where multiple image items have a relationship of derived images, box 215 is used to describe the association between the image items. In a mode where a region in an image item is indicated, box 215 is used to describe the association between the image item and the region item. In a mode where a region is specified by combining region items, box 215 is used to describe the association between the region item and the derived region item.

[0229] According to the association of the items, the item reference type is specified, and the type of the item reference can be identified. In addition, the reference relationship between the items is described by describing the item ID specified in box 2214 in from_item_ID and to_item_ID. The association between the items in the derived area item is indicated by the item reference type "drgn", and the ID of the item specified in box 2214 is specified as the type of the derived area item in from_item_ID. The ID of one or more area items that serve as input to the derived area item is specified in to_item_ID. When the operation corresponding to the type of the derived area item is applied to the area and reference space of the area item that serves as input to the derived area item, the reference space and area defined by the derived area item are obtained. The details of the operation performed to obtain the reference space and area are identified by the item type. Note that the reference_count of the "drgn" item reference of the "iden" derived area item is equal to 1.

[0230] Box 2216 (ItemPropertiesBox("iprp")) stores various attribute information (item properties) related to information items included in the HEIF file 2200. More specifically, box 2216 includes a box 2221 (ItemPropertyContainerBox("ipco")) describing the attribute information and a box 2222 (ItemPropertyAssociation("ipma")) indicating the association between the attribute information and each item.

[0231] Block 2221 may store, for example, attribute information such as entry data indicating an HEVC parameter set required for decoding of an HEVC image item and entry data indicating the width and height of the image item in pixels, etc. Attribute information capable of specifying arbitrary text information as item properties may also be specified. Figure 5 The UserDescriptionProperty ("udes") shown is an example of attribute information that can store text information. When the "udes" property is associated with a region item or a derived region item, region annotation information can be associated as a text description with the specific region indicated by the region item or the derived region item associated with the image item.

[0232] A TransformativeProperty that is intended to transform and display an image when it is output can also be specified as attribute information that can be specified as an item property. This can store, for example, data indicating rotation information for rotating and displaying an image, or data indicating cutout information for cropping (cutting out) and displaying an image. TransformativeProperty can be applied to a region item.

[0233] Block 2222 stores, for each item, entry data indicating the association with the attribute information stored in block 2221 using the ID of the information item (item ID). Note that if there is no attribute information to be associated with the item (as with an Exif data block information item), entry data indicating the association is not stored.

[0234] Box 2217 (ItemDataBox("idat")) stores data related to the items included in the HEIF file 2200. In box 2217, for example, a data structure for describing a derived image item is stored. Here, for example, if the item type indicated by box 2214 is an item of "grid", a data structure of a grid derived image item that reconstructs an input image in a predetermined grid order and defines the input image is specified. For the input image of the derived image item, box 2215 is used to specify an item reference of the dimg reference type. Similarly, even for a derived region item that specifies an item data structure, a data structure for describing the derived region item is stored in box 2217. Note that if the derived item is, for example, an identity derived image item "iden" that does not have a data structure, no data structure is stored in box 2217. Similarly, even for a derived region item, if the derived region item has a type that does not specify a data structure, no data structure is stored in box 2217. On the other hand, for future expansion, even for derived area items that do not have a data structure such as specific parameters, a data structure indicating parameters for indicating versions and flags can be stored. Therefore, there is a disadvantage that parameters can be easily added when the definition is updated.

[0235] Next, the following definition will be explained, which uses the area information 2233 to 2234 formed according to the definition of the area information described in the first embodiment to set an area in any image item stored in the HEIF file 2200, and associates the area annotation with the area for storage.

[0236] Region annotations are formed by metadata or other image items associated with one or more regions in an image item. Furthermore, when a region item, taken as input, is subjected to a derivative transformation using a derived region item corresponding to a combination type, the region item can be treated as a derived region item formed by combining regions according to the derived region item type. The derived region can be uniquely identified by the derived region item's item ID.

[0237] In the image file storage device 100 according to the present embodiment, information associated with the region information 2233 to 2234 and the derived region item is added to the following boxes in box 2202, thereby allowing region annotations to be associated with regions or combined regions in a specific image item.

[0238] First, in the HEIF file 2200 according to this embodiment, since the encoded data 2231 to 2232 and the region information 2233 to 2234 are stored in box 2203, an item type is set for each data in box 2214 to distinguish them. Regarding the item type, for example, in the case of the encoded data 2231, "hvc1" is set, and in the case of the region information 2233, "rgan" is set. As with the encoded data 2231 to 2232, the storage location of the region information 2233 to 2234 stored in box 2203 in the HEIF file 2200 is stored in box 2213. In addition, since the region information 2233 to 2234 is information for relatively specifying a region in a reference space as described above, it is necessary to specify the image item for setting the region. Information about which image item a region is set for is stored in box 2215. That is, since box 2215 describes the association between the items set in box 2214, for example, information identifying an image item associated with coded data 2231 is stored in this box. For this coded data 2231, a region is set for a region item associated with region information 2233 using region information 2233. Furthermore, when a region item is associated with a derived image item formed as a derived image, a region within the derived image can be set. When a region item is treated as a region by performing a derived region item conversion operation, the derived region item type for the derived region item conversion operation is determined, and the item_type to be set in box 2214 is determined. To treat the region specified for the region item as a logically added region, cbrg is specified as the item_type. Furthermore, box 2215 describes the association between items regarding the ID of the region item serving as input to the derived region item. More specifically, the item ID of the item designated as a derived region item, indicated by the item reference type "drgn," is assigned to from_item_ID. In addition, the item ID of the input area item is assigned to to_item_ID. If multiple area items are input, these IDs can be specified by listing the IDs in to_item_ID. This specifies the information used to form the derived area item. Thus, the area logically added can be designated as a derived area item. In addition, in order to associate the image item and the derived area item, information indicating for which image item the derived area item sets the area is stored in box 2215.

[0239] On the other hand, for a region set for a specific image item, a region annotation for the region may be set by associating any or all of the following data with the region item or the derived region item.

[0240] First, there is a regional annotation using the item property. The udes property stored in box 2221 in box 2216 is associated with the regional item or derived regional item through box 2222. When the udes property is associated with the regional item or derived regional item, the regional annotation information can be associated as a text description.

[0241] Secondly, there is a region annotation using metadata items. Metadata items such as Exif that store metadata in mdat are associated with region items or derived region items through the reference type cdsc in box 2215.

[0242] Finally, there are region annotations that use other image items or entity groups. Region items or derived region items are associated with other image items or entity groups using the reference type eroi. This can provide different image representations for specific regions, for example. By associating, region annotations can be set for specific regions in an image.

[0243] Next, the definition of a method for specifying a combination region of a region item that can be stored in the HEIF file 200 will be described. As a derived region item that can be stored in the HEIF file 200, a region derivation of RegionCombineDerivation can be defined. The item body of RegionCombineDerivation is empty, and the structural data is not defined. On the other hand, as described above, for future expansion, even for a derived region item that does not have a data structure such as specific parameters, a data structure indicating parameters for indicating a version and a flag can be stored. In this case, the storage location of the data structure can be indicated by using information (byte offset or data length, etc.) indicating the storage location in the file of the HEIF file 2200 in the description associated with the derived region item in box 2213 (ItemLocationBox ("iloc")). As the storage location, typically, the data structure held by the derived region item is described in box 2217 (ItemDataBox ("idat")). As the data structure of the derived region item, the data structure such as Figure 25 The data structure shown here only has version parameters and flag parameters. This has the advantage of making it easy to add parameters or switch versions when updating the definitions described below. On the other hand, if only the initial version is used, empty data is stored, and 16 bits of data are wasted.

[0244] The type of such a derived region item can be defined as RegionCombineDerivation as follows. Note that it can be easily understood that the name of the derived region item and the 4CC used to identify the item type of the derived region item can be another name and a different 4CC. For example, UnionRegionDerivation can be used as another name indicating a combination and can be identified by "unrg" or the like.

[0245] An item whose item_type value is "cbrg" defines a derived area item as follows, which is configured to create a combined area by combining one or more (or two or more) input areas via OR. These input areas are displayed as a list in a SingleItemTypeReferenceBox (single item type reference box) of type "drgn" of a derived area item in an ItemReferenceBox. In the input area, an area item or a derived area item is specified. In the SingleItemTypeReferenceBox of type "drgn", the value of from_item_ID identifies the derived area item of type "cbrg", and the value of to_item_ID identifies the input area item. The combined (or union) derived area item is associated with the following image item, in which the area is defined using a reference from the combined (or union) derived area item to the type "cdsc" item of the image item.

[0246] The region generated from the derived region item is the union of all regions of the input region items after being applied to the referenced image item. More flexible region designation can be achieved by specifying derived region items as input regions. When forming a region by ORing these input regions, all regions indicated by any of the regions indicated by these input regions are recognized as regions indicated by the RegionCombineDerivation derived region item.

[0247] Multiple RegionCombineDerivation derived region items with different item_id values ​​can be specified in the same file.

[0248] The number of regions indicated by a region item or a derived region item specified in the input area is recommended to be one (for example, region_count in the region item is 1). In this case, the number of designated input regions (ref_count in box 2215) is recommended to be two or more than two. On the other hand, if the number of regions indicated by one input area is two or more than two (for example, region_count in the region item is 2 or more), the regions may be combined by ORing all regions together with other regions. In this case, the number of designated input areas may be one. As another method, if the number of region items specified in the input area of ​​the derived region item or the number of regions indicated by the derived region item is equal (region_count in all input areas is equal), the regions may be combined in the order in which the regions are input. Various region combination methods corresponding to the number of regions may be individually controlled by the value of flags included in the definition of the item information entry in box 2214. Alternatively, Figure 25 The flags stored in the data structure shown. The region indicated by the RegionCombineDerivation derived region item and the region indicated by the input region item can be individually associated with the image item, and the region in the image can be identified. In addition, different region annotation information can be associated using the method defined in the region annotation.

[0249] As a derived region item that can be stored in the HEIF file 200, a region derivation of RegionMultiplyDerivation can be defined. The item body of RegionMultiplyDerivation is empty and the structural data is not defined. The type of such a derived region item RegionMultiplyDerivation can be defined as follows. Note that it can be easily understood that the name of the derived region item and the 4CC used to identify the item type of the derived region item can be another name and a different 4CC. For example, RegionIntersectionDerivation can be used as another name to indicate a combination and can be identified by "isrg" or the like.

[0250] An item whose item_type value is "mprg" defines a derived region item that is configured to create a combined region by combining one or more input regions via AND. The input regions are displayed as a list in a SingleItemTypeReferenceBox of type "drgn" of a derived region item in an ItemReferenceBox. In the SingleItemTypeReferenceBox of type "drgn", the value of from_item_ID identifies the derived region item of type "mprg", and the value of to_item_ID identifies the input region item. The intersection derived region item is associated with an image item in which a region is defined using a reference from an intersection derived region item to an item of type "cdsc" of the image item.

[0251] The region generated from the derived region item is the intersection of all regions of the input region items after being applied to the image item to be referenced. As the input region, either a region item or a derived region item is specified. More flexible region specification can be achieved by specifying a derived region item as an input region. When forming a region by ANDing these input regions, the common region included in all regions indicated by the input regions is identified as the region indicated by the RegionMultiplyDerivation derived region item.

[0252] Multiple RegionMultiplyDerivation derived region items with different item_id values ​​can be specified in the same file.

[0253] The number of regions indicated by a region item or a derived region item specified in the input region is recommended to be one (for example, region_count in the region item is 1). In this case, the number of specified input regions (ref_count in box 2215) is recommended to be two or more. On the other hand, if the number of regions indicated by an input region is two or more (for example, region_count in the region item is 2 or more), the regions can be combined by ANDing all regions together with other regions. In this case, the number of specified input regions can be one. As another method, if the number of regions indicated by the region items specified in the input region of the derived region item or by the derived region items is equal (region_count in all input regions is equal), the regions can be combined in the order in which the regions are input. Various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of the item information entry in box 2214. The regions indicated by the RegionMultiplyDerivation derived region item and the regions indicated by the input region item can be individually associated with the image item, and the regions in the image can be identified. In addition, different region annotation information can be associated through methods defined in region annotations.

[0254] As a derived region item that can be stored in the HEIF file 200, a region derivation of RegionInverseDerivation can be defined. The item body of RegionInverseDerivation is empty and the structural data is not defined. The type of such a derived region item RegionInverseDerivation can be defined as follows. Note that it can be easily understood that the name of the derived region item and the 4CC used to identify the item type of the derived region item can be another name and a different 4CC. For example, RegionNegationDerivation can be used as another name to indicate the combination and can be identified by "ngrg" or the like.

[0255] An item whose item_type value is "ivrg" defines a derived region item that is configured to create a combined region by combining one or more input regions via NOT or NOR. These input regions are displayed as a list in a SingleItemTypeReferenceBox of type "drgn" of a derived region item in an ItemReferenceBox. In the SingleItemTypeReferenceBox of type "drgn", the value of from_item_ID identifies the derived region item of type "ivrg", and the value of to_item_ID identifies the input region item. The inverse (or inverse or negation) derived region item is associated with an image item in which a region is defined using a reference from the inverse (or inverse or negation) derived region item to an item of type "cdsc" of the image item.

[0256] The region generated from the derived region item is the inverse of all regions of the input region items after being applied to the reference image item. As the input region, either a region item or a derived region item is specified. By specifying a derived region item as the input region, you can specify the region more flexibly.

[0257] When a region is formed by NOT or NOR of these input regions, a region not included in any region indicated by the input regions is identified as a region indicated by the RegionInverseDerivation derived region item.

[0258] Multiple RegionInverseDerivation derived region items with different item_id values ​​can be specified in the same file.

[0259] The number of regions indicated by a region item or a derived region item specified in the input region is recommended to be one (for example, region_count in the region item is 1). In this case, if the number of specified input regions (ref_count in box 2215) is one, the region obtained by NOT of the region is designated as the combined region. That is, the region other than the region indicated by the input region is the target region. On the other hand, if the number of input regions is two or more, the region obtained by NOR of the region indicated by the input region is designated as the combined region. That is, the region indicated by all input regions and not specified in any region is the target region.

[0260] On the other hand, if the number of regions indicated by one input region is two or more than two (for example, region_count in the region item is 2 or more), the regions can be combined by NOR of all regions with the regions indicated by other input regions. As another method, if the number of regions indicated by the region items specified in the input region of the derived region item or by the derived region items is equal (region_count in all input regions is equal), the regions can be combined in the order of input of the regions. Various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of the item information entry in box 2214. The region indicated by the RegionInverseDerivation derived region item and the region indicated by the input region item can be individually associated with the image item, and the region in the image can be identified. In addition, different region annotation information can be associated by the method defined in the region annotation.

[0261] Note that if the number of region items used as input regions is one, then it can be obtained by defining Figure 14 In this case, you can use the iden region derivation to specify NOT or NOR. Figure 14 The shown InverseRegionProperty ("ivrg") is an example of TransformativeProperty attribute information capable of specifying NOT or negation logic of a region. Figure 14 The InverseRegionProperty shown includes a 4-character code "ivrg" indicated by definition 1401, and the InverseRegionProperty is identified using the 4-character code. The ivrg property is a property formed of only the 4-character code, version information, and flags, and does not include other parameters.

[0262] Figure 23 The data structure of RegionLogicalDerivation, which is a derived region item that can be stored in the HEIF file 200, and can define region derivation by logical operation. The data structure of RegionLogicalDerivation is the data structure of the derived region item stored in box 2217, and as shown in FIG. Figure 23As shown, it includes the version indicated by definition 2301, the flags indicated by definition 2302, and the logical_operation indicated by definition 2303. The type of such a derived region item can be defined as RegionLogicalDerivation as follows. Note that it can be easily understood that the name of the derived region item and the 4CC used to identify the item type of the derived region item can be another name and different 4CC.

[0263] An item whose item_type value is "lorg" defines a derived area item that is configured to create a combined area by combining one or more input areas through a logical operation. These input areas are displayed as a list in a SingleItemTypeReferenceBox of type "drgn" of a derived area item in an ItemReferenceBox. In the SingleItemTypeReferenceBox of type "drgn", the value of from_item_ID identifies the derived area item of type "lorg", and the value of to_item_ID identifies the input area item. The logical (or logical or logical operation) derived area item is associated with an image item in which an area is defined using a reference from the logical (or logical or logical operation) derived area item to an item of type "cdsc" of the image item.

[0264] The region generated from the derived region item is a region obtained by adapting the specified logical operation to each input region item after being applied to the image item to be referenced. As the input region, a region item or a derived region item is specified. More flexible region specification can be performed by specifying a derived region item as an input region. Regions are formed by logical operations on these input regions. The operator of the logical operation is provided by the logical_operation parameter of definition 2303. That is, the region obtained as a result of performing a logical operation on the region indicated by the input region using the operation method indicated by the logical_operation parameter is identified as the region indicated by the RegionLogicalDerivation derived region item.

[0265] Multiple RegionCombineDerivation derived region items with different item_id values ​​can be specified in the same file.

[0266] logical_operation indicates the type of logical operation and is an unsigned integer indicating the logical operator applied to the input range as part of the derived range conversion. A value of 0 indicates the logical combination of the ranges by an OR operation. A value of 1 indicates the logical combination of the ranges by an AND operation. A value of 2 indicates the logical combination of the ranges by NOT or NOR. A value of 3 indicates the logical combination of the ranges by XOR. A value of 4 indicates the logical combination of the ranges by NAND. A value of 5 indicates the logical combination of the ranges by XNOR. Other values ​​are reserved.

[0267] The number of regions indicated by a region item assigned to an input region or a derived region item is handled in the same manner as described above, and a method can be defined. Furthermore, various region combination methods corresponding to the number of regions can be individually controlled by the value of definition 2302 (flags) included in the definition of RegionLogicalDerivation. This behavior can be determined by a combination of the value of logical_operation and the value of flags.

[0268] The region indicated by RegionLogicalDerivation and the region indicated by the input region item can be individually associated with the image item, and the region in the image can be identified. In addition, different region annotation information can be associated by the method defined in the region annotation.

[0269] Note that in this embodiment, a method of providing an operator for a logical operation by using the parameter logical_operation included in the data structure of RegionLogicalDerivation is described, but this can be controlled using flags. Note that if the number of region items as input regions is one, in order to specify the combined region by logical operation, Figure 15 The LogicalOperateRegionProperty shown associates it with a region item. The lorg property is a TransformativeProperty and applies the logical operation specified by logical_operation to the associated region item. Note that the region item can be a derived region item. In this case, the iden region derivation can be used to specify the logical operation.

[0270] Similarly, to specify the combination of region information through other logical operations, the item type can be defined as the derivation operation type of each derived region item. For example, XOR, NAND, or XOR can be specified. In addition, if the number of region items as input regions is one (ref_count in box 2215), and one region item indicates multiple regions (for example, region_count in the region item is 2 or more), the item property corresponding to the logical operation can be defined to specify the combined region. In this case, the definition can also be performed by performing iden region derivation.

[0271] The above describes the derived area item type that logically combines areas specified as derived area items. Next, the area derivation operation type for showing the association of input areas included in the derived area item will be described.

[0272] As a derived region item that can be stored in the HEIF file 200, a region derivation of RegionInclusionRelationDerivation can be defined. The item body of RegionInclusionRelationDerivation is empty and no structural data is defined. The type of such a derived region item RegionInclusionRelationDerivation can be defined as follows. Note that it can be easily understood that the name of the derived region item and the 4CC used to identify the item type of the derived region item can be another name and a different 4CC. For example, CompoundRegionDerivation can be used as another name to indicate a combination and can be identified by "corg" or the like.

[0273] An item whose item_type value is "irrg" defines a derived area item that specifies an association between areas of an inclusion relationship indicating one or more input areas. These input areas are displayed as a list in a SingleItemTypeReferenceBox of type "drgn" of a derived area item in an ItemReferenceBox. In the SingleItemTypeReferenceBox of type "drgn", the value of from_item_ID identifies the derived area item of type "irrg", and the value of to_item_ID identifies the input area item. The inclusion relationship (or composite or mixed) derived area item is associated with an image item in which an area is defined using a reference to an item of type "cdsc" from the inclusion relationship (or composite or mixed) derived area item to the image item.

[0274] The region generated from the derived region item indicates the relationship between the following regions: the first specified input region item in the input region item after the image item to be referenced includes the second or subsequently specified input region item. As the input region, the region item or the derived region item is specified. More flexible region specification can be performed by specifying the derived region item as the input region. The region is identified as the following region: the second or subsequently specified input region is included in the region indicated by the first specified input region. Note that in this embodiment, as the definition of RegionInclusionRelationDerivation, a derived region indicating an inclusion relationship in which the second or subsequent input region is included in the first input region is defined. However, the data structure of the derived region item can be used to make flexible specification possible. More specifically, parameters can be defined so that it is possible to identify which input region includes which input region or whether multiple input regions include other input regions. For example, specification is performed so that the parent / child relationship indicating the inclusion relationship can be identified using ref_count through parameters for each input region.

[0275] Multiple RegionInclusionRelationDerivation derived region items with different item_id values ​​can be specified in the same file.

[0276] The number of regions indicated by a region item or a derived region item designated as an input region is recommended to be one (for example, region_count in the region item is 1). In this case, the number of designated input regions (ref_count in box 2215) is recommended to be two or more. On the other hand, if the number of regions indicated by an input region is two or more (for example, region_count in the region item is 2 or more), and the input region is the first input region, it can be defined so that the first region specified as region information includes all subsequent regions and the regions indicated by the second and subsequent input regions. In this case, the number of designated input regions can be one. Alternatively, it can be defined so that the region indicated by the first input region is treated as one region and includes the second and subsequent input regions. In this case, the number of designated input regions (ref_count in box 2215) is recommended to be two or more. As another method, if the number of regions indicated by the region item specified in the input region of the derived region item or by the derived region item is equal (region_count in all input regions is equal), association can be indicated so that each region is included in the first region in the order in which the regions were input. Various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of the item information entry in box 2214. The region indicated by the RegionInclusionRelationDerivation derived region item and the region indicated by the input region item can be individually associated with the image item. In addition, different region annotation information can be associated by the method defined in the region annotation. Note that in the present embodiment, a derived region item for indicating the relationship between regions is defined, and the relationship between region items as the input region is shown. However, instead of being defined as a derived region item, it is possible to simply use an item reference between region items to describe. In this case, for example, it is described as an item reference from the first specified region item in the derived region item to other region items.

[0277] As a derived region item that can be stored in the HEIF file 200, a region derivation of RegionTouchRelationDerivation can be defined. The item body of RegionTouchRelationDerivation is empty and no structural data is defined. The type of such a derived region item RegionTouchRelationDerivation can be defined as follows. Note that it can be easily understood that the name of the derived region item and the 4CC used to identify the item type of the derived region item can be another name and a different 4CC.

[0278] The item whose item_type value is "tcrg" defines the following derived area item, which specifies the association between areas indicating that one or more input areas maintain a contact relationship. The number of input areas is two, and these input areas are displayed as a list in a SingleItemTypeReferenceBox of type "drgn" of the derived area item in the ItemReferenceBox. In the SingleItemTypeReferenceBox of type "drgn", the value of from_item_ID identifies the derived area item of type "tcrg", and the value of to_item_ID identifies the input area item. The contact relationship (or touch or common boundary) derived area item is associated with the following image item, in which the area is defined using a reference from the contact relationship (or touch or common boundary) derived area item to the type "cdsc" item of the image item.

[0279] The area generated from the derived area item indicates the relationship between the following areas: the areas specified in each input area item after being applied to the image item to be referenced maintain a contact relationship. As the input area, the area item or the derived area item is specified. More flexible area specification can be performed by specifying the derived area item as the input area. The two areas in the image associated by the derived area item identify that the subject objects maintain a contact relationship. This does not simply indicate that the areas are in contact or overlap with each other, but indicates that the subject objects in the image maintain a contact relationship.

[0280] Multiple RegionTouchRelationDerivation derived region items with different item_id values ​​can be specified in the same file.

[0281] The number of regions indicated by a region item or a derived region item designated as an input region is preferably one (e.g., region_count in the region item is 1). On the other hand, if the number of regions indicated by an input region is two (e.g., region_count in the region item is 2), the number of designated input regions (ref_count in block 2215) may be one. In this case, the two regions included in the input region are in contact with each other. Furthermore, if the number of regions indicated by an input region is two or more (e.g., region_count in the region item is 2 or more), or if the number of input regions is two or more, it may be defined that all regions of the first input region are in contact with all regions indicated by the second input region. Furthermore, if the number of regions indicated by the region items or derived region items designated in the input regions is equal (region_count in all input regions is equal), it may be defined that the regions designated in the first input region are in contact with the regions designated by the second or subsequent input regions in the order in which they are designated. Various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of the item information entry in block 2214. The region indicated by the RegionTouchRelationDerivation derived region item and the region indicated by the input region item can be individually associated with the image item. In addition, different region annotation information can be associated by the method defined in the region annotation. Note that in the present embodiment, the derived region item type indicating a contact relationship is described. However, a type indicating non-contact can be used. In addition, both contact and non-contact relationships can be defined by defining flags or additional parameters in the data structure of the RegionTouchRelationDerivation derived region item. Note that in the present embodiment, a derived region item is defined for indicating the relationship between regions, and the relationship between region items as input regions is shown. However, instead of being defined as a derived region item, it is possible to simply use an item reference between region items to describe it. In this case, for example, it can be described as an item reference from an item designated as an input region item in a derived region item to another input region item, or the item references can be described to each other.

[0282] As a derived region item that can be stored in the HEIF file 200, a region derivation of RegionDepthOrderRelationDerivation can be defined. The data structure of RegionDepthOrderRelationDerivation is not shown, but includes Figure 12The same parameters are used for the entity groups shown. The type of such a derived region item can be defined as RegionTouchRelationDerivation as follows. Note that it can be easily understood that the name of the derived region item and the 4CC used to identify the item type of the derived region item can be another name and a different 4CC.

[0283] An item whose item_type value is "dorg" defines a derived region item that is used to specify an association between regions by a front / back relationship of the foreground / background of one or more input regions. The input regions of RegionDepthOrderRelationDerivation are displayed as a list in a SingleItemTypeReferenceBox of type "drgn" of derived region items in ItemReferenceBox in the order of the direction indicated by depth_direction. In the SingleItemTypeReferenceBox of type "drgn", the value of from_item_ID identifies the derived region item of type "dorg", and the value of to_item_ID identifies the input region item. The front / back relationship (or depth or foreground / background) derived region item is associated with an image item in which a region is defined using a reference from the front / back relationship (or depth or foreground / background) derived region item to an item of type "cdsc" of the image item.

[0284] The regions generated from the derived region items indicate the relationship between regions where the regions specified in the input region items after being applied to the referenced image item maintain a front / back relationship. As the input region, either a region item or a derived region item is specified. By specifying a derived region item as the input region, more flexible region specification can be achieved.

[0285] depth_direction indicates the depth direction (foreground / background) and is an unsigned integer indicating the direction in the depth direction of the input region from which the region is derived. A value of 0 indicates that the input region is listed from the foreground to the background. A value of 1 indicates that the input region is listed from the background to the foreground. Other values ​​are reserved.

[0286] Multiple RegionDepthOrderRelationDerivation derived region items with different item_id values ​​can be specified in the same file.

[0287] The number of regions indicated by a region item or a derived region item specified in the input area is recommended to be one (for example, the region_count in the region item is 1). In addition, the number of region items or derived region items as input areas is recommended to be two or more. On the other hand, if the number of regions indicated by one input area is two or more (for example, the region_count in the region item is 2 or more), the number of input areas specified (ref_count in the iref box) may be one. In this case, this indicates that the multiple regions included in one input area are specified sequentially in the direction indicated by depth_direction in the storage order. As another method, the multiple regions included in one input area may be defined as regions at the same position (that is, at the same depth). If multiple regions are included in one input area, the number of input areas may be one. Alternatively, if the number of regions indicated by the input areas is equal, it may be defined so that the regions are arranged in the direction indicated by depth_direction in the order in which the regions are specified in the region count order. Various region combination methods corresponding to the number of regions can be individually controlled by defining flags or additional parameters in the data structure of the RegionDepthOrderRelationDerivation derived region item. In addition, additional parameters can be defined and specified to indicate whether the respective regions have the same depth or the next depth. Note that in the present embodiment, derived region items are defined to indicate the relationship between regions, and the relationship between region items as input regions is shown. However, instead of being defined as derived region items, it is possible to simply use item references between region items for description. In this case, for example, it is described as an item reference from the first specified region item in the derived region item to other region items. In this case, the arrangement order in depth can be identified in the order in which the item references are specified.

[0288] As a derived region item that can be stored in the HEIF file 200, a region derivation of RegionForbidRelationDerivation can be defined. The item body of RegionForbidRelationDerivation is empty and the structural data is not defined. The type of such a derived region item RegionForbidRelationDerivation can be defined as follows. Note that it can be easily understood that the name of the derived region item and the 4CC used to identify the item type of the derived region item can be another name and a different 4CC. For example, OclusionDerivation can be used as another name to indicate a combination and can be identified by "corg" or the like.

[0289] The item whose item_type value is "fbrg" defines the following derived area item, which specifies the association between areas used to indicate that the second input area is partially or completely hidden by the first input area. The number of input areas is two, and these input areas are displayed as a list in the SingleItemTypeReferenceBox of type "drgn" of the derived area item in the ItemReferenceBox. In the SingleItemTypeReferenceBox of type "drgn", the value of from_item_ID identifies the derived area item of type "fbrg", and the value of to_item_ID identifies the input area item. The hidden relationship (or occlusion or interference) derived area item is associated with the following image item, in which the area is defined using a reference from the hidden relationship (or occlusion or interference) derived area item to the type "cdsc" item of the image item.

[0290] The region generated from the derived region item indicates the relationship between the following regions: the regions specified in the respective input region items after being applied to the image item to be referenced maintain a hidden relationship. As the input region, the region item or the derived region item is specified. More flexible region specification can be performed by specifying the derived region item as the input region. It is identified that one region hides the object indicated by the other region by two regions in the image associated by the derived region item. For example, if all objects indicated by the other region are hidden by the object indicated by one region, the hidden object does not appear as a subject in the image. That is, this indicates that a hidden region is set based on information that can be estimated from another image, etc., and the hidden region is hidden by the object indicated by the other region.

[0291] Multiple RegionForbidRelationDerivation derived region items with different item_id values ​​can be specified in the same file.

[0292] The number of regions indicated by a region item or a derived region item designated as an input region is recommended to be one (for example, region_count in the region item is 1). On the other hand, if the number of regions indicated by one input region is two (for example, region_count in the region item is 2), the number of designated input regions (ref_count in box 2215) may be one. In this case, this indicates that the object indicated by the first region of the two regions included in one input region hides the object indicated by the second region. In addition, if the number of regions indicated by one input region is two or more (for example, region_count in the region item is 2 or more), or the number of input regions is two or more, this may indicate that the region indicated by the first input region hides the object in the region designated by the second or subsequent input region.

[0293] Various region combination methods corresponding to the number of regions can be individually controlled by the values ​​of flags included in the definition of the item information entry in box 2214. The region indicated by the RegionForbidRelationDerivation derived region item and the region indicated by the input region item can be individually associated with the image item. In addition, different region annotation information can be associated by the method defined in the region annotation. Note that in the present embodiment, a derived region item for indicating the relationship between regions is defined, and the relationship between region items as input regions is shown. However, instead of being defined as a derived region item, an item reference between region items can be simply used for description. In this case, for example, it is described as an item reference from the region item first specified in the derived region item to other region items.

[0294] In the above-mentioned definitions of derived area items corresponding to each combination type, a method of using flags in each definition to define the number of areas held by the area item as the input area is described. As another combination area description method, refer to Figure 26 The following method is described for realizing this by defining a data structure of different derived area items even when multiple areas are indicated by one area item or derived area item specified in the input area, regardless of the control using the value of flags. Figure 26 In the example shown, a combination area using OR is described as an example. Even other combination types can be defined using the same mechanism. Figure 26 The definition of CombinedRegionItem shown illustrates a data structure that may be stored in an idat box as the data structure of a derived region item.

[0295] Figure 26The data structure shown is an example of an extension of a combined region item. The extended combined derived region item has a configuration capable of describing multiple combined regions in one combined derived region item. In a region item, when multiple regions are described in one region item, these regions can be described by specifying a value of 2 or more in region_count. Like this method, this definition enables multiple regions to be described even in a combined region. For each of the number (region_count) of regions defined by the extended combined derived region item, the number of input regions (input_count) that form the combined region is given. Each input region item is specified by an index (region_item_index) of a region item that defines the order of input to the combined region and an index (region_index) in the list of regions defined by the input region item.

[0296] Here, version is defined as 0 or 1. In future extensions, new version values ​​may be specified for version. Here, flags is set to 0, and all flag values ​​are retained. Here, region_count is the number of regions defined by the combined region item. Here, input_count is the number of regions that are input regions for each combined region defined in the combined region item. Here, region_item_index is the index of the input region item in the list of input region items specified by the SingleItemTypeReferenceBox of the type "drgn" of the derived region item, and indicates one of the input regions forming a region in the combined region defined by the derived region item. Here, region_index is the index of the region in the input region item notified by the region_item_index field as the input region forming one of the combined regions.

[0297] By defining the data structure of derived region items in this manner, derived region items can be easily created regardless of the number of regions that form the region item as the input region. Furthermore, if the same annotation needs to be added to multiple combined regions, a common annotation can be added even if no annotation is added for each derived region item, as with existing region items.

[0298] The data structure can also be adapted to other combination types. In a derived region item with a parameter like RegionLogicalDerivation, the same definition can be made by adding the parameter to the data structure.

[0299] Here, we will refer to Figures 24A to 24CAn example of an image file output by the image file storage device 100 according to this embodiment is described below. Note that in this embodiment, an image file is configured such that a region obtained by combining region items via two RegionCombineDerivation derivation region items is stored in the file as a file data structure.

[0300] exist Figures 24A to 24C In the example shown, as shown by the description 2403 corresponding to the block 2203, descriptions 2430 and 2431 corresponding to HEVC encoded data (HEVC image data) 2231 to 2232 are stored. In addition, an image file is exemplarily shown in which descriptions 2432 and 2433 corresponding to region information (region item data) 2233 to 2234 and a description 2434 corresponding to the Exif data block 2235 are stored. As shown by the descriptions 2432 and 2433, the region information 2233 to 2234 conforms to the Figure 3A and Figure 3B The definitions shown in the figure each specify a rectangular area in a reference space having an image size of 4032 pixels by 3024 pixels. Furthermore, for the area specified by the description, the coordinates (x0, y0) of the reference point in the reference space of the area and the image size (w0×h0) and the coordinates (x7, y7) of the reference point and the image size (w7×h7) are specified.

[0301] Description 2401 corresponds to block 2201. As the type value major-brand of the brand definition conforming to the HEIF file, "mif1" is stored, and as the type value compatible-brands of the compatible brand definition, "heic" is stored.

[0302] Description 2402 corresponding to box 2202 shows various information describing metadata of the non-timing data stored in the output file example. Description 2410 corresponds to box 2211 and specifies that the handler type of the MetaDataBox (meta) is "pict." Description 2411 corresponds to box 2212 and stores 5 as item_ID, specifying the ID of the image to be displayed as the first priority image.

[0303] Description 2412 corresponds to box 2214 and indicates item information (item ID (item_ID) and item type (item_type)) for each item. Each item can be identified by item_ID, and it shows what kind of item is identified by item_ID. Figures 24A to 24CIn the example shown, since 16 items are stored, entry_count is 16. Sixteen types of information are arranged in description 2412, and the item ID and item type are specified for each of these 16 types of information. In the image files shown in these figures, the first to fourth information corresponding to descriptions 2440 and 2441 are HEVC coded image items of type hvc1. Furthermore, the fifth information indicated by description 2442 specifies a grid-derived image item of item type grid. The sixth information indicated by description 2443 is an HEVC coded image item of item type hvc1 used as a thumbnail image. The seventh to thirteenth information corresponding to descriptions 2444 and 2445 are region items of item type rgan indicating regions. The fourteenth and fifteenth information corresponding to descriptions 2446 and 2447 are each derived region items of item type cbrg indicating region derivation using RegionCombineDerivation. The cbrg region derived items specified by description 2446 and description 2447 can use item ID 57 and item ID 58, respectively, to identify the combined region.

[0304] In the sixteenth information corresponding to the description 2448, a metadata item indicating the item type Exif of the Exif data block is specified.

[0305] Description 2413 corresponds to box 2213 and specifies the storage location and data size information of each item in the HEIF file. For example, the coded image item with item_ID 1 indicates that the data is located at offset O1 in the file, has a size of L1 bytes, and specifies the location of the data in box 2203.

[0306] Description 2414 corresponds to block 2215 and indicates a reference relationship (association) between items. The item reference indicated by description 2450 specifies dimg, which indicates that the reference type is a derived image. Furthermore, it indicates that the HEVC coded image items of item_IDs 1 to 4 specified by to_item_ID are referenced by the grid derived image item of item_ID 5 specified by from_item_ID. Thus, the coded image items of item_IDs 1 to 4 are designated as input images to the derived image items.

[0307] The item reference indicated by description 2451 specifies a thumbnail relationship as the reference type. This indicates that the grid-derived image item with item_ID 5 specified by to_item_ID is referenced by the HEVC coded image item with item_ID 6 specified by from_item_ID. This indicates that the HEVC coded image item with item_ID 6 is a thumbnail image of the grid-derived image item with item_ID 5.

[0308] The item references indicated by descriptions 2452, 2453, 2456, 2457, and 2458 specify a cdsc indicating a content description relationship as the reference type. Description 2452 indicates that the grid-derived image item with item_ID 5 specified by to_item_ID is referenced by the region information item with item_ID 50 specified by from_item_ID. Therefore, the region information item with item_ID 50 indicates a partial region within the grid-derived image item with item_ID 5. Similarly, description 2453 indicates that the grid-derived image item with item_ID 5 specified by to_item_ID is referenced by the region information item with item_ID 56 specified by from_item_ID. Therefore, the region information item with item_ID 56 indicates a partial region within the grid-derived image item with item_ID 5. Description 2456 indicates that the grid-derived image item of item_ID 5 specified by to_item_ID is referenced by the cbrg-derived region item of item_ID 57 specified by from_item_ID. Thus, the combined region indicated by the cbrg-derived region item of item_ID 57 indicates a partial region in the grid-derived image item of item_ID 5. Similarly, description 2457 indicates that the grid-derived image item of item_ID 5 specified by to_item_ID is referenced by the cbrg-derived region item of item_ID 58 specified by from_item_ID. Thus, the combined region indicated by the cbrg-derived region item of item_ID 58 indicates a partial region in the grid-derived image item of item_ID 5. Description 2458 indicates that the grid-derived image item of item_ID 5 specified by to_item_ID is referenced by the Exif data block item of item_ID 1000 specified by from_item_ID. This indicates that the Exif data block item of item_ID 1000 thus describes information at the time of imaging of the grid-derived image item of item_ID 5 .

[0309] The item references indicated by descriptions 2454 and 2455 specify a drgn indicating that the reference type is a derived region. Description 2454 indicates that the region information items for item_IDs 50, 51, 52, 54, and 55 specified by to_item_ID are referenced by the cbrg derived region item for item_ID 57 specified by from_item_ID. Consequently, the region information items for item_IDs 50, 51, 52, 54, and 55 are designated as input regions to the derived region item. Consequently, the regions indicated by the region information items for item_IDs 50, 51, 52, 54, and 55 are combined into a combined type of cbrg, and all regions included in any of these five regions are designated as regions indicated by the cbrg derived region item. Description 2455 indicates that the region information items for item_IDs 53 and 56 specified by to_item_ID are referenced by the cbrg derived region item for item_ID 58 specified by from_item_ID. Thus, the region information items of item_IDs 53 and 56 are designated as input regions to the derived region item. Therefore, the regions indicated by the region information items of item_IDs 53 and 56 are combined into a combined type of cbrg, and all regions included in any of the two regions are designated as regions indicated by the cbrg derived region item.

[0310] Description 2415 corresponds to box 2217 and specifies the data structure of the grid-derived image item of item ID 5. In the HEIF file according to the present embodiment, 2 is specified in rows_minus_one of the grid-derived image, and 2 is specified in columns_minus_one. This indicates that 2×2 input images are formed as grid-derived images. Note that as the number of input images referenced by the item indicated by description 2450, the item count "4" specified in ref_count and the number of grids (that is, 2×2 (4)) are specified (both are values ​​consistent with each other). 4032 is specified in the number or output_width, and 3024 is specified in output_height. The image is formed as an image with a width of 4032 pixels and a height of 3024 pixels. Note that since the cbrg-derived region item described in the present embodiment does not have a data structure, the data structure is not stored in box 2217. However, in the case of a derived region item that stores a data structure, such as the RegionLogicalDerivation derived region item, the data structure is stored in box 2217 of description 2415.

[0311] Description 2416 corresponds to box 2216 and includes description 2420 corresponding to box 2221 and description 2421 corresponding to box 2222. Description 2420 lists the attribute information available in the item or entity group as entry data. As shown in the figure, description 2420 includes not only the first and second entries indicating encoding parameters and the third, fourth, and fifth entries each indicating the display pixel size of the item, but also the sixth to fourteenth entries indicating annotation descriptive information. As annotation descriptive information, lang is American English (en-US) in all entries, "human" is set as name in the sixth to twelfth entries, and "family" is set in the thirteenth and fourteenth entries. As description, "Hanako," "Taro," "Shohei," "Mike," "Naomi," "Ginnji," and "Susan" are sequentially set to represent the names of the people indicated by the region in the sixth to twelfth entries. Furthermore, as description, “Taro family” and “Mike family” indicating families are set in the thirteenth and fourteenth entries. In tags, “AutoRecognition” is set to add a tag indicating that the area is an automatically recognized area.

[0312] In the entry data of the description 2421 corresponding to the box 2222, the attribute information listed in the description 2420 is associated with the item or entity group stored in the HEIF file. Figures 24A to 24C In the example shown, a common "ispe" (property_index is 4) is associated with the image items of item_IDs 1 to 4, and this indicates that the images have the same image size of 2016 pixels × 1512 pixels. Similarly, a common "hvcC" (property_index is 1) is associated with the image items of item_IDs 1 to 4, and this indicates that the same encoding parameters are used. On the other hand, different item properties are associated with the items or entity groups of item_IDs 5 to 15. "ispe" (property_index is 3) is associated with the derived image item of item_ID 5, and this indicates that the image size is 4032 pixels × 3024 pixels.

[0313] "ispe" (property_index is 5) is associated with the image item of item_ID 6, indicating that it is an image with an image size of 768 pixels × 576 pixels. Similarly, "hvcC" (property_index is 2) is associated with the image item of item_ID 6, indicating encoding parameters. "udes" (property_index is 6) is associated with the region information item of item_ID 50, indicating region annotation information for the region. "udes" (property_index is 12) is associated with the region information item of item_ID 56, indicating region annotation information for the region. Thus, adding region annotation information to the region is achieved by associating the "udes" property. In addition, "udes" (property_index is 13) is associated with the derived region item of item_ID 57, indicating region annotation information for the combined region. "udes" (property_index is 14) is associated with the derived region item of item_ID 58, indicating region annotation information for the combined region. Thus, adding region annotation information to the combined region is achieved by associating the "udes" property with the entity group. Note that since no item property is associated with the Exif data block item of item_ID 1000, the corresponding entry information is not stored.

[0314] In the above embodiment, with respect to the combined area information corresponding to the combination type stored in the image file, the input area is used as a derived area using derived transformation type information or parameters, and the combination type is specified using area derivation. When the derived area item is associated with the image, the subject in the image can be identified by combining multiple areas. In addition, area annotation information can be associated with the combined area. Here, the type of derived transformation is merely an example. Other combination types can also be defined, and any derived transformation configured to handle areas in combination can be used. When performing a derived transformation of an area, other derived areas can also be specified for the input area. This enables more flexible area specification. In addition, when handling each area as a derived area used as an input area, input areas to different derived areas can be used for each area. Accordingly, a subject object can belong to multiple derived areas and can be identified, and area annotations can be added to each area. Note that the 4-character code for identifying the derived area item type of the combined area shown in the embodiment is merely an example, and any derived area item type with an arbitrary 4-character code reserved for each derived area item type can be used.

[0315] [Third embodiment]

[0316] In the above embodiment, a method using entity groups and a method using derived regions were described as methods for identifying region information as combined regions based on type. However, the present invention is not limited to these methods. In this embodiment, a method for improving the data structure of region information defined as items will be described as a method for identifying combined regions.

[0317] More specifically, define Figure 3A and Figure 3B Flags 302 in the definition of the region information shown or a separate additional parameter, and defines a value that can identify the combination type described in the above embodiment. For example, when using flags, if flags&2 is valid, a combination region of the cbrg type is specified. In this case, multiple region items cannot be combined. However, multiple regions can be defined in one region item (region_count is 2 or more), and the region obtained by combining these regions can be specified. For combination types that require parameters separately, the item properties can be used to associate the parameters.

[0318] As described above, according to this embodiment, when one or more pieces of region information for identifying an object captured in an image and associated with a portion of the image are stored in association with the image file, combination type information for combining the region information and combined region information for identifying the combined region corresponding to the combination type information are stored, thereby efficiently enabling flexible region designation. In this case, region annotation information cannot be associated with each region, but a region can be flexibly designated as a region item.

[0319] [Fourth embodiment]

[0320] The above embodiment describes a method for identifying region information in a two-dimensional image as a combined region based on type and associating the combined region with the two-dimensional image. However, the present invention is not limited to this. Specifically, this can be achieved by defining region information having a three-dimensional shape for a three-dimensional graphic image and combining the methods described in the above embodiment.

[0321] According to this, three-dimensional region information can be individually identified not only for a two-dimensional image but also for a three-dimensional image formed of point group data or mesh data, and regions can be combined and identified according to type.

[0322] As described above, according to this embodiment, in a case where one or more three-dimensional region information for identifying a three-dimensional object represented by a three-dimensional image and associated with a part of the three-dimensional image is stored in an image file in association with the three-dimensional image, combination type information for combining the three-dimensional region information and combination region information for identifying the combination region corresponding to the combination type information are stored, thereby efficiently allowing flexible region designation.

[0323] [Fifth embodiment]

[0324] In the above embodiment, still images are mainly used as images, and region information is associated with the still images and identified. A method of identifying regions as combined regions according to type and associating them with still images is described. However, the present invention is not limited thereto.

[0325] That is, region information is defined for a moving image or image sequence, and a metadata structure for storing moving images can be used to specify a combined region corresponding to the type. In a moving image or image sequence, the region corresponding to the subject object detected by the recognition processing unit 114 from the image may move within the image over time. For this reason, regions may overlap in the image at a certain point in time, and regions may appear or disappear (due to being obscured by other regions or objects outside the image or frame, etc.).

[0326] As a storage method, the moov box, which is a metadata structure defined by ISOBMFF, is used to describe metadata related to a motion picture or an image sequence. This indicates a time-specified sequence of media data belonging to the presentation of the motion picture in the trak box specified in the moov box. For example, a sequence of video frames, a sequence of sub-parts of video frames, a sequence of audio samples, or a sequence of time-specified metadata samples is described. In each track, each time unit of data is called a sample. This can be a frame of video, audio, or time-specified metadata, a sub-part of a frame, or an image in an image sequence. A sample is defined as all media data associated with the same presentation time in a track.

[0327] Region information is stored in the mdat box for storing encoded data of a moving image. Metadata describing a timed sequence of region information is configured as a track called a metadata track and storing timed metadata. When the metadata track is associated with the track of the moving image using the tref box, the region in the moving image can be indicated. Regarding multiple combined regions, multiple region metadata tracks are described, these tracks are placed in a track group, and the combined region corresponding to the track group type can be specified. In addition, the track group description box can be used to associate region annotation information. If regions related to multiple different objects are specified in one region metadata track, these regions can be identified by an ID, etc. in the region information stored in the mdat box and placed in a group using a sample group structure. The sample group can similarly specify group type information, and the sample group description box can be used to add region annotation information to the combined region corresponding to the group type.

[0328] In addition, samples for describing combined regions can be defined and stored as the same or separate metadata tracks. Figure 28 、 Figure 29 、 Figure 30 31, a method for defining region information for a moving image or an image sequence, specifying a combined region corresponding to the type using a metadata structure for storing the moving image, and storing the combined region in a file is described.

[0329] Figure 28 An example of a presentation including region information stored in a media file is shown. Here, according to several embodiments, a mapping between regions and annotations associated with the regions is included in samples of a region track. Media data formed by a time-specified sequence of image or motion picture frames is stored in track 2810, referred to as a source track. In this source track, time-specified units of the time-specified sequence are encapsulated into samples (2811, 2812, and 2813). For example, the source track can be a video track (e.g., a track including a handler box of the handler type "vide") or an image sequence track (e.g., a track including a handler box of the handler type "pict"). Region information can be specified in a portion of samples 2811, 2812, and 2813, and annotations associated with the region can be specified. Regions are detected by an image analysis algorithm or selected by the user. The geometry (i.e., position and / or shape) of each region in samples 2811, 2812, or 2813 of source track 2810 is described in region track 2820, which can be considered a metadata track. This is for example a track comprising a handler box of the handler type "meta".

[0330] The metadata track 2820 is formed by, for example, sample entries with coded names or a 4-character sample entry type "rgan" specific to a "region annotation" track or a "region" track. The metadata track 2820 is referred to as a "region track." In a region track, a sample defines one or more regions of an image included in the sample of the associated image sequence or video track (also referred to as a source track).

[0331] In the region track 2820 , as shown at 28252 , a region is defined using a track reference of type “cdsc” from the region track to the source track, and the region is associated with the source track 2810 including track_ID 1 of the sample.

[0332] Figure 28The illustrated region track 2820 is formed by sample 2821 describing two regions with region identifiers equal to 1 and 2. These regions shift between consecutive samples. For example, the positions and (in some cases) shapes of regions 1 and 2 change in sample 2822. Subsequently, the regions with region identifiers 1 and 2 disappear, and a new region with region identifier 3 appears, as in sample 2823. The generation unit 113 generates region identifiers and positional and shape information for the corresponding regions based on information about the subject detected from the image by the recognition processing unit 114. Note that when generating region information, the generation is based not only on the subject detection results of the recognition processing unit 114, but also on information about the region specified by the user through operation of the operation input unit 107. The generated region identifiers and region information are stored as metadata in a file by the metadata processing unit 112. The region track contains annotations called region comments, which are sometimes associated with one or more samples in the region track (such as samples 2821, 2822, and 2823). Region annotations can be items such as those shown in 2840, such as image items or metadata items, another track 2830, samples 2831, 2832, and 2833 in the other track 2830, or simple labels or text as shown in the example of 2824. The simple annotation indicated by 2824 is provided as a SampleGroupDescriptionEntry in the SampleGroupDescriptionBox or as an item property of the region track 2820 itself (not shown). In 2824, the SampleGroupDescriptionBox can have any value for grouping_type indicating the type of property described by the SampleGroupDescriptionEntry. In 2824, the annotation is formed by a user description in text format as a grouping_type value represented by the four-character code "udes". If the region annotation is formed by another track such as the annotation track 2830, the track is a metadata track, a video track, an audio track, a 3D video track, or an image sequence track, depending on the annotation type. In the region track 2820, the track reference type "anot" is used to associate with the region annotation track 2830, as shown in 28251. Using the same mechanism (28251), the region track 2820 can reference annotation items, as indicated by 2840. In this case, as in the example shown in the first embodiment, the "unif" brand needs to be set in the media file. Each region in sample 2821, 2822, or 2823 of the region track 2820 is associated with a region identifier. By using the same region identifier between consecutive samples, the same moving region can be identified between consecutive samples.The range of region identifiers (uniquely identifiable ranges) can persist across multiple samples in a region track, or the range of region identifiers can cover the entire media file. When the range is limited to a region track, files can be easily edited. As a detailed example, when creating a new region track, there is no need to analyze the region identifiers of other region tracks to detect the possibility of conflicts, and any region identifier can be used in the recreated region track. On the other hand, if the range of region identifiers covers the entire media file, it is easy to describe the combination of regions or the relationship between regions. This will be explained in detail later regarding the combination of regions or the relationship between regions. If the range of region identifiers is specified in the media file (the entire media file or multiple samples in the region track), it is guaranteed that the client or reader can identify the same region by the same region identifier in different samples. If the range of region identifiers is not shown in the media file, the content creator may use the same region identifier between samples or not use the same region identifier between samples, and there is no guarantee that the client or reader can use the same region identifier to identify the same moving region between different samples.

[0333] Information specifying the range of region identifiers in a region track may be stored in a sample entry for a region track using a flag indicating whether the range of region identifiers persists for all samples associated with the sample entry. For example, a RegionSampleEntry or one of the sub-boxes such as a RegionTrackConfigBox is used to describe the samples of the region track. Alternatively, in some cases, in the sample entry, the range of the region identifier to the region track is specified as a specific flag value, for example, by the track header box. A level box structure at a higher level than the track is used to identify the storage of identification information in cases where the range of the region identifier covers the entire media file (for example, as an "ftyp" or "styp" brand, maintaining the same range, or describing the same source track, or being described in a track group of a region track in an entity group). The HEIF specification defines the payload of samples in a region track. The data (or payload) of a sample of a region track describes the shape, position, size, etc. of the region.

[0334] These region information define the shape, position and size of the region in the reference space mapped in the samples 2811, 2812 and 2813 of the source track 2810 associated with the region track 2820. For example, the shape of the region can be a point, a polyline, a rectangle, a polygon or a binary mask, etc., and this is indicated in the payload of the sample by the geometry_type parameter. In the HEIF specification, means for associating annotations with regions in samples of the region track are also defined. For example, a sample-to-region-id-mapping ("regm") sample group (not shown) can be used.

[0335] Figure 29 An example of a region track 2920 is shown. For example, samples 2921 and 2922 include regions 1 and 2, and the union of these two regions as third regions 2927 and 2928. This is an example of a region combination. The combination methods shown in the first and second embodiments (such as OR, AND, and NOT) can be similarly adapted. In an embodiment, the region track 2920 is extended to support regions generated from operations on regions. For clarity and conciseness, regions generated as a result of an operation or conversion to one or more regions will be referred to as "combined regions" or "derived regions." Furthermore, for clarity and conciseness, regions used in a conversion or operation will be referred to as "input" regions, "simple" regions, or "base" regions. In this embodiment, the region track is defined as a metadata track of the "rgan" sample entry type, which includes samples of one or more regions in an image that defines an associated image sequence or video track. These one or more regions can be simple regions (or base regions) or combined regions (or derived regions). It is also possible to further combine simple regions (or basic regions) and composite regions (or derived regions), or to use composite regions (or derived regions) in combination. Each region in the sample of the region track is associated with a region identifier independently of its type. Thus, annotations can be associated with regions by region identifiers using existing mechanisms predetermined by HEIF, etc. Composite or derived regions are distinguished from simple or basic regions in the samples of the region track by a specific value of geometry_type.

[0336] As a method of describing a combined region of one or more than one regions in an image sequence or video track, several methods will be further described. As a first specification method, the shape type (geometry_type) directly indicates the type of combination or derivation.

[0337] For these types, the combination types shown in the first and second embodiments (such as OR, AND, and NOT) can be specified without restriction. Then, specific integer values ​​of geometry_type are reserved to represent one or another type combination, enabling interoperability between a media file writer (or storage module or packaging module) and a reader (or analysis module or media player). The payload (or data) of a sample from a region track can be expanded as follows.

[0338] aligned(8)class RegionSample{

[0339] unsigned int field_size=((RegionTrackConfigBox.flags&field_length_size)==field_length_size)? 32:16;

[0340] / / this is a temporary, non-parsable variable

[0341] unsigned int(32)region_count;

[0342] for(r=0;r <region_count;r++){

[0343] unsigned int(32)region_identifier;

[0344] unsigned int(8)geometry_type;

[0345] if(geometry_type==0){

[0346] / / point

[0347] signed int(field_size)x;

[0348] signed int(field_size)y;

[0349] }

[0350] else if(geometry_type==1){

[0351] / / rectangle

[0352] signed int(field_size)x;

[0353] signed int(field_size)y;

[0354] unsigned int(field_size)width;

[0355] unsigned int(field_size)height;

[0356] }

[0357] / *Give other basic area shape categories Figure 3A sum Figure 3B Middle phase same and general omitted * /

[0358] else if(geometry_type==6){

[0359] / / union of regions

[0360] unsigned int(8)num_regions;

[0361] (for int n=0;n <num_regions,n++){

[0362] unsigned int(32)region_id; / / regions that are part of the union

[0363] }

[0364] }

[0365] else if(geometry_type==7){

[0366] / / intersection of regionsunsigned int(8)num_regions;

[0367] (for int n=0;n <num_regions,n++){

[0368] unsigned int(32)region_id; / / regions that are part of the Intersection

[0369] }

[0370] else if(geometry_type==8){

[0371] / / negation of regions

[0372] unsigned int(8)num_regions;

[0373] (for int n=0;n <num_regions,n++){

[0374] unsigned int(32)region_id; / / regions that are part of the negation

[0375] }

[0376] / *After that, the same specification can be made even for other combination types* /

[0377] }

[0378] Note that the value of geometry_type of each combined region in the above RegionSample (region sample) only needs to be a specifiable value.

[0379] Here, num_regions indicates the number of input regions for derivation or combination. Here, region_id indicates the region_identifier for the input region for derivation or combination. In the case of a combined region or a derived region, data related to shape information (shape, position, size, etc.) does not exist (can be omitted). As needed, the shape information of the combined region or the derived region can be estimated from the input region. For example, if all input regions have a geometry type specified as a rectangular shape and are arranged in a grid to form a rectangular bounding box, or if the input regions are formed as peripheral regions regularly arranged around a central region, the shape can be easily estimated.

[0380] When, for example, object detection or recognition is performed in a single pass, the input regions and combined regions are sometimes interleaved in the sample payload. That is, the order in which the shape types of the basic regions specified in a sample and the shape types of the combined regions are specified can be arbitrarily specified. In addition, when, for example, object detection or recognition is performed in multiple passes, the combined regions are sometimes defined at the end of the sample payload. As an option, a most significant bit designated as 1 can be added before the integer value of the geometry_type corresponding to the combination of regions, or the most significant bit can be defined in a reserved range corresponding to the largest value among the 8-bit integer values ​​that can be specified by geometry_type. For example, the reserved range of values ​​for specifying a combination of regions can be from 11110001 to 11111111 (in binary notation) (that is, from 241 to 255 (in decimal notation)). Therefore, 15 types can be used as combinations, and 241 types can be used as other geometry types used as basic regions to describe region shapes. The range can be adapted so that more values ​​can be assigned to the combinations as needed.

[0381] As a second designation method, geometry_type indicates whether the region is a combined region or a derived region using a specific integer value (e.g., 0xFF), and combination_type is used to specify the combination type information of the combined region or derived region. As described above, the combination types shown in the first and second embodiments (such as OR, AND, and NOT) can be specified without limitation. In this embodiment, the sample payload is updated as follows.

[0382] aligned(8)class RegionSample{

[0383] unsigned int field_size=((RegionTrackConfigBox.flags&field_length_size)==field_length_size)? 32:16;

[0384] / / this is a temporary, non-parsable variable

[0385] unsigned int(32)region_count;

[0386] for(r=0;r <region_count;r++){

[0387] unsigned int(32) region_identifier;

[0388] unsigned int(8) geometry_type;

[0389] if(geometry_type == 0){

[0390] / / point

[0391] signed int(field_size) x;

[0392] signed int(field_size) y;

[0393] }

[0394] else if(geometry_type == 1){

[0395] / / rectangle

[0396] signed int(field_size) x;

[0397] signed int(field_size) y;

[0398] unsigned int(field_size) width;

[0399] unsigned int(field_size) height;

[0400] }

[0401] / * Other basic region shape types are the same as in Figure 3A and Figure 3B and will be omitted * /

[0402] else if(geometry_type == 0xFF){ / * reserved value for generic combination * /

[0403] unsigned int(8) combination_type;

[0404] unsigned int(8) num_regions;

[0405] for(int n = 0; n < num_regions, n++){

[0406] unsigned int(32)region_id; / / regions that are part of the combination

[0407] }

[0408] Here, combination_type indicates the type of derivation of the region or combination of the regions (for example: OR, AND, NOT, include relationship and contact relationship, etc.). Here, num_regions indicates the number of input regions for derivation or combination. Here, region_id indicates the region_identifier of the input region for derivation or combination. As an option, in the first specification method or the second specification method, additional information such as parameters or flags may be included in the sample entry of the region track "rgan". As additional information, for example, "combination_present" is defined as a flag value specified in the RegionTrackConfigBox that can be predetermined in the HEIF specification. If the value is set to a specific value, this indicates that a region combination or derivation can be included in the region track. If the value is not set to a specific value, this indicates that only basic regions or simple regions are included in the region track.

[0409] aligned(8)class RegionTrackConfigBox()extends FullBox('rgaC',version=0,flags){

[0410] unsigned int((field_length_size+1)*16)reference_width;

[0411] unsigned int((field_length_size+1)*16)reference_height;

[0412] unsigned int(1)combination_present;

[0413] unsigned int(7)reserved;

[0414] }

[0415] Here, if combination_present is set, this indicates that a combination of regions may be included in the sample referenced by the sample entry including the box. If combination_present is not set, the parser or reader can recognize that a combination of regions is not included in the sample referencing the sample entry including the box. When this information is maintained at the sample entry level, the parser or reader can determine that a certain region combination exists from the top of the file. In the case of a conventional player that does not support region combinations, the region track is not processed, or is processed while ignoring a portion of the sample with the geometry_type corresponding to the region combination, so that recognition or interpretation may not be possible. In this case, when a region is specified from a simple region or a basic region and stored (without interleaving simple regions and combined regions), and the combined region is stored at the end of the sample, processing can be interrupted when the combined region appears, and only the simple region or basic region can be processed. Alternatively, all simple regions or basic regions can be processed.

[0416] As a third designation method, designation of a flag value when region combination or derived combination_present is included in a region track is used, and combination_type is indicated at the same level as region_identifier in the payload of samples of the region track as in the second designation method.

[0417]

[0418]

[0419] Here, combination_type indicates the derivation type of a region or the combination of regions (e.g., OR, AND, and NOT). For example, a specific value such as 0 indicates that the combination is "no combination" or "does not exist." Therefore, if a sample entry indicates that there is no combination or derived region, the specification of combination_type can be avoided. Next, geometry_type uses a universal value (e.g., 0xFF). The information describing a combination or derived region in the payload of a sample in a region track is formed simply by a list of region_identifiers of the input regions associated with the combination or derivation.

[0420] The above-mentioned several designation methods and optional methods according to the present embodiment are effective when one region track is included in the media file. On the other hand, there are several disadvantages, especially when the decision of region or object recognition is performed multiple times. For example, it is necessary to update the sample payload and (sometimes) the corresponding sample size, as well as the block offset in the sample description of the region track. In addition, combinations of regions across different region tracks cannot be processed. On the other hand, in the method according to the present embodiment, the number of region tracks 2920 can be suppressed to be small, and there is a possibility that the number of region tracks in normal use cases can be one. These indications describe the trade-off between the cost and the number of specifications in the target usage method. When a combination or derivative of the regions described in the sample can be specified and the region identifier is maintained, a unified method for associating annotations with regions can be provided independently of the type such as simple, basic, or combination or derivative (see Figure 28 ). Additionally, the timing aspects of the combinations can be manipulated. If a combination declared in the sample payload is not present, this means that the region combination does not exist in the sample.

[0421] Regarding the method for extending the definition of RegionSample as described above, the same extended definition is possible even in the method using the definition of RegionItem shown in the third embodiment.

[0422] Figure 30 Another example in this embodiment is shown. In this embodiment, some of the above disadvantages (particularly the necessity to update the payload of the samples of the regional track) are improved.

[0423] In this embodiment, a combined region or derived region is defined in a specific track 3050. The track is newly defined as CombinedRegionTrack 3050 and includes a specific sample entry 3051. As a method of providing a region combination type 3052 of a region, the track sometimes uses a specific box in a sample entry.

[0424] CombinedRegionTrack is a metadata track that has an arbitrary 4-character code and is reserved for describing the sample entry "cbrg" or "combined region".

[0425] This is a region track whose samples include the region defined by one or more region tracks 3020 and / or 3030 as input regions and define one or more regions as combined regions or derived regions. The combined region track 3050 is associated with the region track that defines the input region to the combined region or derived region using a track reference of type "cbrg" (3053) from the combined region track 3050 to one or more region tracks 3020 and / or 3030. The combined region track can be associated with the source track 3010 via a track reference of type "cdsc" 3054. If 3054 is not present, the source track can be obtained via the "cbrg" track reference type 3053, and for at least one region track referenced by the "cbrg" track reference type 3053, the track reference type "cdsc" 3021 or 3031 can be traced. If 3054 is present, the source track can be directly identified from the combined region track 3050. As with the region tracks 3020 or 3030, the combined region track 3050 can associate region annotations with each combined region or derived region defined in the samples of the associated source track. For region tracks, region annotations can be items (e.g., image items or metadata items), samples in another track (see Figure 28Each region in a sample of a combined region track is associated with a region identifier. The range that can be uniquely identified by the region identifier of a combined region is indicated by the persistent_region_ids flag in a sample entry of the combined region track or its subframes. By using region identifiers, annotations can be associated with the combined region in a sample via an association mapping defined by the sample-to-region-id-mapping sample group. In this case, the value of the region identifier in the sample and the value of the region identifier in the sample group are in the same value space. Because the combined region in the combined region track can be calculated based on the input region, descriptions of shape, position, size, etc. may not be present. If the shape, position, and size of the entire combined region are maintained during file generation, these can be included in the payload of the samples of the combined region track. Samples of the combined region track need to be synchronized or aligned in time with the corresponding samples of the region track referenced via the "cbrg" track reference 3053. Note that the combined region track can contain a mixture of combined regions and basic regions or simple regions in a sample. This is the case, for example, when repeated object detection or recognition is performed and additional simple regions or basic regions are defined by the iterations. It may be easier to specify these new regions in the combined region track than to edit the existing region track. The geometry_type described in the above embodiment and its variants can be used to distinguish between combined regions and basic regions. For the sync sample 3016 of the source track or for each sync sample 3026 or 3036 of the region track, a sync sample 3056 that is aligned in time with the sync sample is preferably present in the combined region track 3050 ( Figure 30 When defining the combination type 3052 of a combination region track, there are several options depending on whether it includes a combination of a single type or a combination or derivation with different types of regions mixed together.

[0426] In a first approach, a single combined type of combined region or derived region is included in the combined region track 3050. In this case, the combined type is specified in the sample entry or in one of its subboxes (that is, in the "stsd" box or in one of its subboxes). This avoids duplication of the combined type (3055 or 3056, etc.) in each region in each sample of the combined region track. This can be done in a new version of the RegionTrackConfigBox, a new configuration box specific to the combined region track, or a new box included in the sample entry of type "cbrg" (e.g., in addition to the RegionTrackConfigBox defined as version 0 in the HEIF specification).

[0427] In order to extend RegionTrackConfigBox and indicate the combination type 3052 of the combined region track 3050, the following description is possible.

[0428] aligned(8)class RegionTrackConfigBox()extends FullBox('rgaC',version,flags){

[0429] unsigned int((field_length_size+1)*16)reference_width;

[0430] unsigned int((field_length_size+1)*16)reference_height;

[0431] if(version>0)unsigned int(8)combination_type;

[0432] }

[0433] aligned(8)class CombinedRegionSampleEntry extends MetadataSampleEntry('cbrg'){

[0434] RegionTrackConfigBox config; / / mandatory with version=1

[0435] }

[0436] If combination_type 3052 is included in a new configuration box specific to combination region track 3050, the box may be defined as follows.

[0437] aligned(8)class CombinedRegionTrackConfigBox()extends FullBox('cbrC',version,flags){

[0438] unsigned int(8)combination_type;

[0439] / / optional parameters,can be controlled by flags or version fields of the box

[0440] unsigned int((field_length_size+1)*16)reference_width;

[0441] unsigned int((field_length_size+1)*16)reference_height;

[0442] }

[0443] aligned(8)class CombinedRegionSampleEntry extends MetadataSampleEntry('cbrg'){

[0444] CombinedRegionTrackConfigBox config;

[0445] }

[0446] In CombinedRegionSampleEntry (combined region sample entry), in general usage, a combination of regions using OR is defined as the default combination type for the combined region track, omitting the configuration box. That is, if the default combination type is used, no specification is required. If a combination type other than the default is used as combination_type, a configuration box is required in the sample entry describing the sample of the combined region track.

[0447] The parameters reference_width and reference_height are options in the CombinedRegionTrackConfigBox because they can be the same as the region track to be referenced, especially when the same reference space is shared or there is only one region track. If the same reference space does not exist in the region track to be referenced, these parameters can be specified in the CombinedRegionTrackConfigBox by the encapsulation module so that the reader can correctly place the combined region on the source image.

[0448] If combination_type 3052 is in a new box included in a sample entry of type "cbrg", it may be defined as follows. (The names and 4-character codes used below are examples only, and any reserved names and 4-character codes may be used unless these conflict with other names and 4-character codes already in use).

[0449] aligned(8) class CombinedRegionSampleEntry extends MetadataSampleEntry('cbrg') {

[0450] RegionTrackConfigBox config; / / with version = 0

[0451] RegionCombinationTypeBox combination_type;

[0452] }

[0453] For example, define RegionCombinationTypeBox (region combination type box) as follows.

[0454] aligned(8) class RegionCombinationTypeBox() extends FullBox('rcbt', version, flags) {

[0455] unsigned int(8) combination_type;

[0456] / / optional parameters

[0457] unsigned int(8) num_input_regions;

[0458] unsigned int(8) combination_shape;

[0459] / / possibly other parameters avoiding repetition in the samples

[0460] }

[0461] Regardless of the aforementioned means, optional parameters may be included for the combination type designation 3052, particularly when a fixed number of regions are used in the combination or a single shape is formed. Optional parameters may be provided to indicate how many input regions are used as a fixed number of input regions (e.g., the num_input_regions parameter) or to indicate shape information when a single shape is repeated (e.g., the combination_shape parameter) to prevent duplication in samples 3055 or 3056 of the combined region track 3050. The values ​​used for the shape information specified as the parameters here may be the values ​​defined by the geometry_type value of the region track samples (e.g., 1 for rectangle, 2 for ellipse, etc.). If these parameter values ​​are specified as 0, these values ​​are reserved to indicate that a specific combined shape is not used and that the specific combined shape may be optionally present in samples 3055 or 3056 of the combined region track 3050. In the case of num_input_regions, it should be explained that the number of input regions can vary for each combined region, and the number of input regions is indicated in samples 3055 or 3056 of the combined region track 3050. Next, in a first method, samples of a combined region track may be represented as follows according to parameters present in a sample entry or its subboxes.

[0462] aligned(8)class RegionCombinationSample{

[0463] unsigned int(8)combined_region_count;

[0464] for(r=0;r <combined_region_count;r++){

[0465] unsigned int(8)combined_region_identifier;

[0466] / *if not provided at sample entry level(or sub-box)* /

[0467] unsigned int(8)num_input_regions;

[0468] for(n=0;n <num_input_regions,n++){

[0469] unsigned int(32)input_region_identifier; / / region that are part of the union

[0470] }

[0471] / / optionally shape, position, size information may be described for the combined region, if not provided at sample entry level (or sub-box)

[0472] }

[0473] }

[0474] num_input_regions indicates the number of input regions used in the combined region. This can be obtained from the sample entry or one of its subboxes if applicable, otherwise it is declared in the sample.

[0475] In the second method, a combined region track such as 3050 includes several types of combined regions or derived regions. In this case, a combined type 3052 can be pre-declared in a sample entry or a sub-box of the sample entry (such as a configuration box or a dedicated box), and the combined type 3052 can be referenced by the sample. If the combined type 3052 is pre-declared in the sample entry and referenced by the sample, the sample entry can be defined as follows.

[0476] aligned(8)class CombinedRegionSampleEntry extends MetadataSampleEntry('cbrg'){

[0477] unsigned int(8)num_combination_types;

[0478] unsigned int(8)combination_type[num_combination_types]

[0479] / / possibly additional parameters describing combinations that are common to several samples

[0480] }

[0481] Here, num_combination_types indicates the number of combination types used in the samples of the reference sample entry.

[0482] Here, combination_type indicates a list of combination types used in the sample of the reference sample entry. The combination type is mapped to a reserved value such as an 8-bit integer value. For example, the value is defined as 1 for the union of regions, 2 for the intersection, and so on, and the value 0 is reserved. These combination type values ​​are used in sample 3055 or 3056 as follows.

[0483] aligned(8)class CombinedRegionSample{

[0484] unsigned int(8)combined_region_count;

[0485] for(r=0;r <combined_region_count;r++){

[0486] unsigned int(8)combined_region_identifier;

[0487] unsigned int(8)combination_idx;

[0488] unsigned int(8)num_input_regions;

[0489] for(i=0;i <num_input_regions;i++){

[0490] unsigned int(8)input_region_identifier;

[0491] }

[0492] }

[0493] }

[0494] Here, combined_region_count indicates the number of combined regions in the sample. This number needs to be at least 1 (if the number is 0, no declaration in the sample is required). Here, combined_region_identifier indicates the region identifier of the combined region. The region identifier can be used as an arbitrary region identifier to express, for example, the association, recombination or relationship of an annotation with other regions. Here, combination_idx indicates in a list the index of the combination type declared in the sample entry to which the sample is to refer. The value 0 is reserved. As a value, a predefined value (for example, an integer value for interoperability (1 for union or 2 for intersection, etc.) needs to be used).

[0495] num_input_regions indicates the number of regions used in the combined region. Here, input_region_identifier indicates the region identifier of the region used in the combined region.

[0496] If the combination type 3052 is previously declared in a sub-box of a sample entry (such as a configuration box), and the combination type 3052 is referenced by the sample, the sample entry and the configuration box may be defined as follows.

[0497] aligned(8)class CombinedRegionSampleEntry extends MetadataSampleEntry('cbrg'){

[0498] RegionTrackConfigBox config; / / mandatory with version=1

[0499] }

[0500] aligned(8)class RegionTrackConfigBox()extends FullBox('rgaC',version,flags){

[0501] unsigned int((field_length_size+1)*16)reference_width;

[0502] unsigned int((field_length_size+1)*16)reference_height;

[0503] if(version>0){

[0504] unsigned int(8)num_combination_types;

[0505] unsigned int(8)combination_type[num_combination_types];

[0506] }

[0507] / / possibly additional parameters describing combinations that are common to several samples

[0508] }

[0509] Here, num_combination_types and combination_type parameters have the same meaning as in the above configuration. In this description method, the new version of RegionTrackBox is used, but a specific configuration box can also be used as follows.

[0510] aligned(8)class CombinedRegionSampleEntry extends MetadataSampleEntry('cbrg'){

[0511] CombinedRegionTrackConfigBox config;

[0512] }

[0513] aligned(8)class CombinedRegionTrackConfigBox()extends FullBox('cbrC',version,flags){

[0514] unsigned int(8)num_combination_types;

[0515] unsigned int(8)combination_type[num_combination_types];

[0516] / / optional parameters,can be controlled by flags or version fields of the boxunsigned int((field_length_size+1)*16)reference_width;

[0517] unsigned int((field_length_size+1)*16)reference_height;

[0518] / / possibly additional parameters describing combinations that are common to several samples

[0519] }

[0520] Here, the num_combination_types and combination_type parameters have the same meaning as in the above configuration. Here, identification related to the reference space is indicated by an option, for example, if it cannot be estimated from the reference regional track. For example, if there is only one regional track, or all regional tracks to be referenced have the same reference space, the parameters associated with the reference space can be omitted.

[0521] As with the above method, reference to the combination type in a sample is made via the combination_idx parameter providing a predefined index in the list. In this method, the reference is made in the configuration box of the sample entry to which the sample refers.

[0522] If the combination type 3052 is previously declared in a sub-box (such as a dedicated box) of a sample entry and the combination type 3052 is referenced by the sample, the sample entry and the dedicated box may be defined as follows.

[0523] aligned(8)class CombinedRegionSampleEntry extends MetadataSampleEntry('cbrg'){

[0524] RegionTrackConfigBox config; / / with version=0

[0525] RegionCombinationTypeBox combination_types;

[0526] }

[0527] For example, RegionCombinationTypeBox can be defined as follows.

[0528] aligned(8) class RegionCombinationTypeBox() extends FullBox('rcbt', version, flags) {

[0529] unsigned int(8) num_combination_types;

[0530] unsigned int(8) combination_type[num_combination_types];

[0531] / / optional parameterss

[0532] unsigned int(8) num_input_regions;

[0533] unsigned int(8) combination_shape;

[0534] / / possibly other parameters avoiding repetition in the samples

[0535] }

[0536] Here, the num_combination_types and combination_type parameters are the same as those described above. As additional data related to the combination type 3052, optional parameters common to several samples may also be included, regardless of the above-described means, in cases where, for example, a fixed number of regions are used in combination or a single shape is formed. Optional parameters may be provided to indicate how many input regions are used as a fixed number of input regions (e.g., the num_input_regions parameter) or to indicate shape information when a single shape is repeated (e.g., the combination_shape parameter) to prevent duplication in samples 3055 or 3056 of the combined region track 3050. As the values ​​used for the shape information used as the parameters specified here, the values ​​defined by the geometry_type values ​​of the samples of the region track (e.g., 1 for a rectangle, 2 for an ellipse, etc.) may be reused. If the values ​​of these parameters are specified as 0, these values ​​are reserved to indicate that a specific combination shape is not used, and that the specific combination shape may optionally be present in samples 3055 or 3056 of the combined region track 3050. In the case of num_input_regions, it needs to be explained that the number of input regions may vary for each combined region, and the number of input regions is shown in sample 3055 or 3056 of the combined region track 3050 .

[0537] As a third method, regardless of the number of combination types in use, the combination type 3052 in the combined region track like 3050 may be described in the payload of the sample 3055 or 3056. For example, one or more regions may be defined in a sample of the combined region track as follows.

[0538] aligned(8)class RegionCombinationSample{

[0539] unsigned int(8)combined_region_count;

[0540] for(r=0;r <combined_region_count;r++){

[0541] unsigned int(8)combined_region_identifier;

[0542] unsigned int(8)combination_type;

[0543] unsigned int(8)num_input_regions;

[0544] for(n=0;n <num_input_regions,n++){

[0545] unsigned int(32)input_region_identifier; / / region that are part of the union

[0546] }

[0547] / / optionally shape, position, size information may be described for the combined region.

[0548] }

[0549] }

[0550] Here, combined_region_count indicates the number of combined regions in the sample. The number needs to be at least 1 (if the number is 0, no declaration in the sample is required). Here, combined_region_identifier indicates the region identifier of the combined region. The region identifier can be used as an arbitrary region identifier such as an annotation's association, recombination, or expression of a relationship with other regions. Here, combination_type indicates the combination type. The value 0 is sometimes reserved to, for example, indicate that the region is a basic region (no combination exists). As a value, a predefined value (for example, an integer value for interoperability (1 for union or 2 for intersection, etc.)) needs to be used. num_input_regions indicates the number of regions used in the combined region. Here, input_region_identifier indicates the region identifier of the input region used in the combined region.

[0551] In this definition, repetition between samples is necessary. On the other hand, different combination types can be handled in the same sample, and since the sample is self-contained, flexible description is possible. However, in this method, it is not possible to determine which combination is being used before processing the sample.

[0552] In some specific usage methods, where the number of combined regions does not change over time (e.g., in the case of an entire presentation or segment of an animation), the combination type is a predetermined type, and the same number of input regions are combined, the sample of the combined region track can be a simple payload as follows.

[0553] aligned(8)class RegionCombinationSample{

[0554] for(n=0;n <num_input_regions,n++){

[0555] unsigned int(32)input_region_identifier; / / region that are part of the union

[0556] }

[0557] }

[0558] Here, num_input_regions indicates the number of regions used in the combined region, and this can be obtained from the sample entry or one of its subboxes. Here, input_region_identifier indicates the region identifier of the region used in the combined region.

[0559] In this particular usage, static information (ie, this does not change over time) may be described at the sample entry level (in the "stsd" box or one of its subboxes).

[0560] Figure 31A A region track 3110 (e.g., an example of an "rgan" track) is shown, and a description of a combined region 3115 is included in the metadata section. The region track includes samples 3111, 3112, or 3113 that provide descriptions of the region. Here, no association with a source track is shown. The description of the combined region 3115 is sometimes shown as a sample group with a dedicated grouping_type. For example, in the case of information related to a combined region, "crif" is shown to identify the sample group type. Sample groups are defined in a SampleToGroupBox ("sbgp" section of 3115) with grouping_type equal to "crif". By grouping, samples that combine regions to form one or more combined regions can be grouped. The associated SampleGroupDescriptionBox ("sgpd" of 3115) with grouping_type equal to "crif" includes one or more specific SampleGroupDescriptionEntry entries that provide descriptions of these combined regions. For example, the "sgpd" entry can be defined as follows.

[0561] class CombinedRegionGroupEntry()extends SampleGroupDescriptionEntry('crif'){

[0562] unsigned int(32)entry_count;

[0563] for(i=1;i<=entry_count;i++){

[0564] unsigned int(8)regionID;

[0565] unsigned int(8)combination_type;

[0566] unsigned int(8)num_input_regions;

[0567] for(n=0;n <num_input_regions,n++){

[0568] unsigned int(32)input_region_identifier; / / region that are part of the union

[0569] }

[0570] }

[0571] Here, entry_count is an integer indicating the number of combined regions in the entry. Here, regionID (region ID) is an integer specifying the region identifier of the combined region. Here, combination_type is an integer specifying the combination type. The value 0 is a reserved value, and as a value, a predefined value (for example, an integer value for interoperability (1 for union or 2 for intersection, etc.) needs to be used). This parameter may be optional if the combination type is a default combination such as a union of regions, or is specified elsewhere such as in the description of the track configuration or sample ("stbl" or "stsd" or its subbox). Here, num_input_regions is an integer specifying the number of input regions used in the combined region. Here, input_region_identifier indicates the region identifier of the input region used in the combined region. This specifies the value of one region identifier in the sample of the region track.

[0572] According to this embodiment, the annotation association mechanism has been expanded, and not only the region identifiers included in the samples of the region track, but also the region IDs included in the sample group description entries can be considered region identifiers. If multiple region tracks are included in a media file, there are region tracks that can include sample groups that describe the combination of regions between samples of the region tracks. In this example, the combined regions declared in the metadata section 3115 of the region track only use the input regions from the region tracks. In other words, the region combination is simply the regions included in the track. Figure 31B An example is shown in which a description of a combined region 3155 is included in the metadata section of a combined region track 3150. The combined region track is indicated by, for example, a sample entry type "cbrg" 3151 in the "stsd" box. Track 3150 references a region track 3160, and its samples include region samples 3161, 3162, or 3163 that define a region, a simple region, or a basic region. Although not illustrated, the association between a region track and a source track can be similarly specified.

[0573] Regarding the description of the combined region, for example, the combination of regions 1 and 2 of samples 3161 and 3162 can be indicated as a sample group having a dedicated grouping_type such as "crif" (which is information related to the combined region of the combined region track). The sample group is defined by SampleToGroupBox ("cbgp" of 3155) whose grouping_type is "crif". By grouping, samples such as 3156 or 3157 can be grouped from combined region tracks having the same combined region (such as regions indicated by the same combination type, the same number of input regions, and the same region identifier, etc.) (even if the sample is an empty sample).

[0574] For example, if two regions with persistent identifiers from time t1 to t2 are declared as the union of a region with region identifier 3, then samples 3156 and 3157 between t1 and t2 form part of the same sample group. The two regions can be regions of the same region track or different region tracks. The sample group of a combined region can be mapped to the description entry of the combined region. If not mapped, there is no combination of regions of the sample group. The combined region description entry can be defined as an entry of the "sgpd" box as follows (in the case where the scope of the region identifier is global for the file).

[0575] class CombinedRegionGroupEntry()extends SampleGroupDescriptionEntry('crif'){

[0576] unsigned int(32)entry_count;

[0577] for(i=1;i<=entry_count;i++){

[0578] unsigned int(8)regionID;

[0579] unsigned int(8)combination_type;

[0580] unsigned int(8)num_input_regions;

[0581] for(n=0;n <num_input_regions,n++){

[0582] unsigned int(32)input_region_identifier; / / region that are part of the union

[0583] }

[0584] }

[0585] Here, entry_count is an integer indicating the number of combined regions in the entry. Here, regionID is an integer specifying the region identifier of the combined region. Here, combination_type is an integer specifying the combination type. The value 0 is a reserved value, and as a value, a predefined value (for example, an integer value for interoperability (1 for union or 2 for intersection, etc.) needs to be used). This parameter may be optional if the combination type is a default combination such as a union of regions, or is specified elsewhere such as in the description of the track configuration or sample ("stbl" or "stsd" or its subbox). Here, num_input_regions is an integer specifying the number of input regions used in the combined region. Here, input_region_identifier indicates the region identifier of the region used in the region track to be referenced. This specifies the value of one region identifier in one sample of the region track.

[0586] In this embodiment, the combined region declared by the metadata portion 3152 of the combined region track uses input regions from one or more region tracks to be referenced. That is, regions can be combined both across tracks and from a single region track. If the scope is global to the media file, the input region is indicated by a region identifier (input_region_identifier parameter). If the scope of the region identifier is at the track level, the scope can be indicated via a pair of input_region_identifier and trackID (track ID) or via a pair of input_region_identifier and track_ref_idx. Here, track_ref_idx is the index of an entry in a track reference box of type "cbrg" 3159 that associates the combined region 3150 with one or more region tracks 3160.

[0587] In a variant, the combination_type parameter may not be included in the "CombinedRegionGroupEntry". In this case the parameter may be provided by the grouping_type_parameter of the SampleToGroupBox with grouping_type equal to "crif". This means that there may be as many "sbgp" boxes and "sgpd" boxes as there are combination types in use. Since the SampleToGroupBox "sbgp" may not be present, it is not appropriate when applying the default sample grouping to describe the 3152 or 3115 region combinations that are applied to all samples of a track or track fragment. Figure 31A or Figure 31B In a corresponding embodiment, the region combination can be declared after the track is constructed (the data portion of the track in the "mdat" box and at least a portion of the description of the sample), and no editing of the sample payload is required.

[0588] When providing a derived or combined region ( Figure 31B The combined area track 3150 shown, Figure 30 The combined area rail 3050 or Figure 29For example, in a track described by the extended region track shown in the example above, a "regm" sample group may be present in the track. For the sample group, a region_identifier corresponding to a region identifier of a combined region or an input region may be considered. Regarding the persistence of the region identifier, if it is set in the flags of the sample entry, it may be applied to both basic regions and combined regions (example: a sample entry type of "rgan" or "cbrg", or a 4-character code for a region track or a combined region track). In order to control the persistence of the region identifier more finely and to distinguish between the persistence of simple regions and the persistence of combined regions, an additional flag value may be defined in the sample entry of the region track or the combined region track as follows (example: a sample entry type of "rgan" or "cbrg", or a 4-character code for a region track or a combined region track).

[0589] 0x000004persistent_combined_region_ids indicates the scope of region identifiers that exist in a sample to reference a combined region of a sample entry. When set, it specifies that the scope of combined region identifiers is persistent across samples. That is, the same region identifier for a combined region in different samples mapped to a sample entry identifies the same combined region. Otherwise, the scope of the region identifier for a combined region is sample-wide. If a separate flag value is used, the value 0x000002persistent_region_ids applies only to simple or basic regions.

[0590] Note, reference Figure 31A or Figure 31B The sample group mechanism of the combined region can also be used to express the relationship between regions. In addition to or instead of specifying the combination type, information about the relationship between regions can be indicated by extending CombinedRegionGroupEntry. For example, the relationship information between regions is indicated as an integer value in a list of predefined values. The relationship information can also be represented by a 4-character code, but the number of bits required for the description is large. For example, in the case of an included relationship, the predefined value is specified as 0. This means that the first region in the list indicated by region_identifier includes the other regions in the list. In addition, 1 indicates a contact relationship of a common boundary between regions. In addition, 2 indicates a depth relationship. 3 can indicate the relationship between the regions described in the first and second embodiments, such as an occlusion relationship (for example, the first region hides the second region) and the like.

[0591] To identify grouped regions and the relationships between regions, different grouping_type values ​​are preferably used. For example, "crif" is the type used for grouped regions, and another 4-character code such as "rrel" is reserved to indicate region relationships. The entries describing the relationships between regions in the SampleGroupDescriptionBox "sgpd" are as follows.

[0592] class RegionRelationshipGroupEntry()extendsSampleGroupDescriptionEntry('rrel'){

[0593] unsigned int(32)entry_count;

[0594] for(i=1;i<=entry_count;i++){

[0595] unsigned int(8)relation_type;

[0596] unsigned int(8)num_input_regions;

[0597] for(n=0;n <num_input_regions,n++){

[0598] unsigned int(32)input_region_identifier; / / region that are part of the union

[0599] }

[0600] }

[0601] Here, entry_count is an integer indicating the number of relationships described in the entry. Note that for relationships between regions, it is not necessary to define region identifiers. Next, the following parameters are formed by specifying the region identifiers of regions with a certain relationship. Figure 31BIn a related embodiment, depending on the scope of the region identifier, if the scope is global for the media file, the scope is indicated by the region identifier (input_region_identifier parameter). If the scope of the region identifier is at the track level, the region can be indicated via a pair of input_region_identifier and trackID, or as a pair of input_region_identifier and track_ref_idx. Here, track_ref_idx is the index of an entry in the track reference box of type "cbrg" indicated by 3159 that associates the combined region track 3150 with one or more region tracks 3160. In addition, such identification information can be specified as the grouping_type_parameter in the SampleToGroupBox used for the description of the combined region "crif", or as the grouping_type_parameter in the SampleToGroupBox used for the description of the region relationship "rrel". In this case, the parameter relation_type is not present in the CombinedRegionGroupEntry or RegionRelationshipGroupEntry (region relationship group entry).

[0602] As described above, not only in still images but also in moving images, region information can be individually identified, and these regions can also be combined and identified according to type. Note that, of course, even for three-dimensional moving images, it can be achieved by combining the above embodiments.

[0603] In addition, the numerical values, processing timing, processing order, processing subject, and data (information) configuration / acquisition method / sending destination / sending source / storage location used in the above embodiments are merely examples for detailed description, and the present invention is not intended to be limited to these examples.

[0604] A part or all of the above embodiments may be used in combination as appropriate. A part or all of the above embodiments may be used selectively.

[0605] (Other embodiments)

[0606] The present invention can be implemented by providing a program for implementing one or more functions of the above-described embodiments to a system or device via a network or storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. The present invention can also be implemented by a circuit (e.g., an ASIC) for implementing one or more functions.

[0607] The present invention is not limited to the above embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, in order to inform the public of the scope of the present invention, the following claims are added.

[0608] This application claims the benefit of Japanese Patent Application No. 2022-104896, filed on June 29, 2022, and Japanese Patent Application No. 2022-176531, filed on November 2, 2022, which are hereby incorporated by reference herein.

Claims

1. An information processing device for generating a media file, the information processing device comprising: An acquisition component, used for acquiring an image; A first generating component, used for generating at least one area information, wherein the at least one area information is used for describing at least one shape information for identifying an object included in the image; A second generating component, used for generating association information for associating the image with the region information; A third generating component, used to generate combination type information for combining the area information; a fourth generating component, configured to generate combined region information for identifying regions combined according to the combination type information; as well as The storage component is used to store the at least one area information, the association information, the combination type information, the combined area information and the image in a media file.

2. An information processing device for processing a media file, the information processing device comprising: an acquisition component, configured to acquire a media file, the media file comprising: an image; at least one region information for describing at least one shape information for identifying an object included in the image; association information for associating the image with the region information; combination type information for combining the region information; and combination region information for identifying a region combined according to the combination type information; and A processing component is used to process the image using the information of the media file.

3. The information processing device according to claim 1 or 2, characterized in that: The image is one of a still image, a moving image, a three-dimensional still image, and a three-dimensional moving image.

4. The information processing device according to any one of claims 1 to 3, characterized in that: The combined area information is information that can be identified independently of the area information.

5. The information processing device according to any one of claims 1 to 4, characterized in that: The combined area information is information for grouping and identifying the at least one area information.

6. The information processing device according to any one of claims 1 to 4, characterized in that: The combined region information is information for deriving the at least one region information into a derived region and identifying the derived region.

7. The information processing device according to any one of claims 1 to 4, characterized in that: The combined region information is information for generating and identifying a combination of the at least one region information using the same data structure as the region information.

8. The information processing device according to any one of claims 1 to 4, characterized in that: The combined region information is information for identifying a track that describes a combination of the at least one region information.

9. The information processing device according to any one of claims 1 to 4, characterized in that: The combined region information is information generated to describe and identify a sample of a combination of the at least one region information.

10. The information processing device according to any one of claims 1 to 9, characterized in that: The combination type information is information for identifying one of the following areas as a combined area: an area obtained by synthesizing, i.e., OR, areas indicated by at least two area information; an area obtained by inverting, i.e., NOT, areas indicated by at least one area information; an area obtained by performing NOR of areas indicated by at least two area information; and an area obtained by performing a logical operation of areas indicated by at least one area information.

11. The information processing device according to any one of claims 1 to 9, characterized in that: The combination type information is one of the following information: information capable of identifying an inclusion relationship between areas indicated by at least two area information; information capable of identifying a relationship that one of the areas indicated by at least two area information includes another area as a constituent element of the other area; information capable of identifying an arrangement order of a foreground / background relationship of the areas indicated by at least two area information; information capable of identifying that an area of ​​an object indicated by one area information in at least two area information is an area hidden by an area of ​​an object indicated by another area information; and information capable of identifying a contact relationship between a region of an object indicated by one of at least two region information and a region of an object indicated by another region information, and The combination type information is information for identifying association between areas.

12. The information processing device according to any one of claims 1 to 11, characterized in that: The region information and the combined region information are information that can be associated with annotation information, respectively.

13. The information processing device according to any one of claims 1 to 12, characterized in that: The region information of the regions combined according to the combination type information is information capable of specifying the individually combined combined region information as the region information.

14. An information processing method for generating a media file, the information processing method being performed by an information processing device, the information processing method being characterized by comprising: An acquisition step, for acquiring an image using an acquisition component of the information processing device; A first generating step for generating at least one area information by using a first generating component of the information processing device, wherein the at least one area information is used to describe at least one shape information for identifying an object included in the image; a second generating step for generating association information for associating the image with the area information by using a second generating means of the information processing device; A third generating step, for generating, by using a third generating component of the information processing device, combination type information for combining the area information; A fourth generating step for generating, by using a fourth generating means of the information processing device, combined area information for identifying areas combined according to the combination type information; as well as A storing step is used to store the at least one area information, the association information, the combination type information, the combined area information and the image in a media file using a storage component of the information processing device.

15. An information processing method for processing a media file, the information processing method being executed by an information processing device, the information processing method being characterized by comprising: an acquisition step for acquiring a media file using an acquisition component of the information processing device, the media file comprising: an image; at least one region information for describing at least one shape information for identifying an object included in the image; association information for associating the image with the region information; combination type information for combining the region information; and combination region information for identifying a region combined according to the combination type information; and The processing step is used to utilize the processing component of the information processing device to process the image using the information of the media file.

16. The information processing method according to claim 14 or 15, characterized in that: The image is one of a still image, a moving image, a three-dimensional still image, and a three-dimensional moving image.

17. The information processing method according to any one of claims 14 to 16, characterized in that: The combined area information is information that can be identified independently of the area information.

18. The information processing method according to any one of claims 14 to 17, characterized in that: The combination type information is information for identifying one of the following areas as a combined area: an area obtained by synthesizing, i.e., OR, areas indicated by at least two area information; an area obtained by inverting, i.e., NOT, areas indicated by at least one area information; an area obtained by performing NOR of areas indicated by at least two area information; and an area obtained by performing a logical operation of areas indicated by at least one area information.

19. The information processing method according to any one of claims 14 to 17, characterized in that: The combination type information is one of the following information: information capable of identifying an inclusion relationship between areas indicated by at least two area information; information capable of identifying a relationship that one of the areas indicated by at least two area information includes another area as a constituent element of the other area; information capable of identifying an arrangement order of a foreground / background relationship of the areas indicated by at least two area information; information capable of identifying that an area of ​​an object indicated by one area information in at least two area information is an area hidden by an area of ​​an object indicated by another area information; and information capable of identifying a contact relationship between a region of an object indicated by one of at least two region information and a region of an object indicated by another region information, and The combination type information is information for identifying association between areas.

20. The information processing method according to any one of claims 14 to 19, characterized in that: The region information and the combined region information are information that can be associated with annotation information, respectively.

21. The information processing method according to any one of claims 14 to 20, characterized in that: The region information of the regions combined according to the combination type information is information capable of specifying the individually combined combined region information as the region information.

22. A computer program configured to cause a computer to function as each component of the information processing apparatus according to any one of claims 1 to 13.

Images