Information processing device, imaging apparatus, information processing method, and computer program

Abstract

To provide a technology for generating an image file with a structure that allows flexible specification of regions in an image and identifies the relevance of each region.SOLUTION: Association information relating an image to region information that generates one or more pieces of region information describing one or more pieces of shape information for identifying objects contained in the image is generated. Combination type information that combines the region information is generated. Combination region information for identifying combined regions in accordance with the combination type information is generated. The one or more pieces of region information, the association information, the combination type information, and the combination region information are stored in an image file.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an imaging device, an information processing method, and a computer program. [Background technology]

[0002] In recent years, cameras, smartphones, and image analysis services have made it possible to generate annotations related to parts or objects in an image (e.g., human-readable or computer-readable metadata that makes it possible to identify what an object is, metadata related to an image). For example, it is possible to detect people, faces, objects, etc. that are subjects of an image. This can be done manually, but with the development of deep learning technology, recognition by AI is also possible. In this case, information identifying the area and annotation information can be added as metadata indicating the recognized object.

[0003] Images taken with a camera or smartphone, or processed by an image analysis service, are stored on a storage device such as a memory card. Images are usually encoded to reduce the data size on the storage device. Many codec standards can be used for encoding, such as JPEG, H.264 (AVC), H.265 (HEVC), H.266 (VVC), and AV1. To store these encoded images in files, a normative structure including metadata is defined, which specifies the method of associating stored images and the metadata structure in a specific format. The ISO Base Media File Format (ISOBMFF, ISO / IEC 14496-12) is specified as one such type of file format.

[0004] ISOBMFF is used for transmission over local storage, a network, or another bitstream delivery mechanism. ISOBMFF is a well-known, flexible, and extensible file format that encapsulates and describes coded timed or untimed media data or bitstreams. There are several extensions to this file format. For example, ISO / IEC 14496-15 specifies encapsulation tools for various Network Abstraction Layer (NAL) unit-based video encoding formats. Examples of such encoding formats are Advanced Video Coding (AVC), Scalable Video Coding (SVC), High Efficiency Video Coding (HEVC), Layered HEVC (L-HEVC), and Versatile Video Coding (VVC).

[0005] Another example of file format extension is ISO / IEC 23090-2, which defines the Omnidirectional Media Application Format (OMAF). Yet another example of file format extension is ISO / IEC 23090-10 and ISO / IEC 23090-18, which define the transmission of Visual Volumetric Video-based Coding (V3C) media data and Geometry-based Point Cloud Compression (G-PCC) media data.

[0006] Another example of a file format extension is the High Efficiency Image File Format (ISO / IEC 23008-12, HEIF), which specifies an encapsulation tool for a still image or a still image sequence, such as an HEVC still image.

[0007] These file formats are object-oriented, structured standards developed by the Moving Picture Experts Group (MPEG) for storing and sharing images and image sequences.

[0008] Patent Document 1 discloses a method for identifying an area within an image to be stored in HEIF as a Region item, making the area within the image identifiable by associating it with the stored image, and adding annotation information to the identified area within the image.

[0009] Patent Document 2 discloses a method for extracting and storing an image to be stored in HEIF into multiple spatial parts, and combining these extracted parts to reconstruct it as one image data. This allows a single image representation consisting of multiple still images, called a derived image, to be recorded as an image file. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] International Publication No. 2021 / 204526 [Patent Document 2] JP 2021-119713 A Summary of the Invention [Problem to be solved by the invention]

[0011] However, in the method of Patent Document 1, the Region item identified as a region defines the region with a predefined shape, so the region cannot be flexibly defined. Although a mask shape is defined as a flexible shape description, the mask data has a large description data size. In other words, if a flexible description is attempted with a predefined shape, the description data size will be large, or it will be necessary to express it with a specific predefined shape, and flexible shape specification such as expression by combining regions will not be taken into consideration.

[0012] In addition, although the structure allows annotation information to be added to the identified regions, there is no way to indicate the relationship between regions, or to combine regions and add different annotations. It is possible to identify groups using metadata described as annotations, but these cannot be identified without analyzing a large amount of annotation information. Furthermore, it is not possible to distinguish whether a common annotation was added for grouping purposes, or whether the same annotation was added to a region that is not a group.

[0013] On the other hand, the method described in Patent Document 2 enables combinations of image data units to be expressed, but does not enable expression by combining identification areas according to subjects in the image. The present invention provides a technology for generating an image file having a structure that allows areas in an image to be flexibly specified and allows the relationship between each area to be identified. [Means for solving the problem]

[0014] One aspect of the present invention is characterized in that it comprises a first acquisition means for acquiring one or more pieces of region information describing one or more pieces of shape information for identifying an object included in an image, a second acquisition means for acquiring association information for associating the image with the region information, a third acquisition means for acquiring combination type information combining the region information, a fourth acquisition means for acquiring combination region information for identifying combined regions according to the combination type information, and a storage means for storing the one or more pieces of region information, the association information, the combination type information, and the combination region information in an image file. Effect of the Invention

[0015] According to the present invention, it is possible to generate an image file having a structure that allows regions in an image to be flexibly specified and enables the association between each region to be identified. [Brief description of the drawings]

[0016] [Figure 1]FIG. 1 is a block diagram showing an example of the hardware configuration of an image file storage device 100. [Diagram 2] A diagram showing the structure of a HEIF file. [Diagram 3] FIG. 13 is a diagram showing an example of a definition of the structure of area information. [Figure 4] A diagram showing the structure of EntityToGroupBox. [Diagram 5] FIG. 13 is a diagram showing an example of a definition of the structure of annotation information. [Figure 6] A diagram showing the structure of a CombinedRegionEntityGroupBox. [Figure 7] A diagram showing the structure of MultiplyRegionEntityGroupBox. [Figure 8] A diagram showing the structure of an InverseRegionEntityGroupBox. [Figure 9] A diagram showing the structure of LogicalOperateRegionEntityGroupBox. [Figure 10] A diagram showing the structure of InclusionRelationRegionEntityGroupBox. [Figure 11] A diagram showing the structure of TouchRelationRegionEntityGroupBox. [Figure 12] A diagram showing the structure of DepthOrderRelationRegionEntityGroupBox. [Figure 13] A diagram showing the structure of ForbidRelationRegionEntityGroupBox. [Figure 14] A diagram showing the structure of InvereRegionProperty. [Figure 15] A diagram showing the structure of LogicalOperateRegionProperty. [Figure 16] FIG. 4 is a diagram showing an example of the configuration of an image file. [Figure 17] Flowchart of the HEIF file generation process. [Figure 18] 1 is a flowchart of a playback process for playing (displaying) a HEIF file. [Figure 19] 6A and 6B are diagrams showing examples of display of region information and region annotation information. [Figure 20] 6A and 6B are diagrams showing examples of display of region information and region annotation information. [Figure 21] 6A and 6B are diagrams showing examples of display of region information and region annotation information. [Figure 22] A diagram showing the structure of a HEIF file. [Diagram 23] A diagram showing the structure of RegionLogicalDerivation. [Figure 24] FIG. 4 is a diagram showing an example of the configuration of an image file. [Diagram 25] FIG. 13 is a diagram showing the data structure of a derived area item. [Figure 26] A diagram showing the definition of CombinedRegionItem. [Figure 27] A diagram showing the definition of CompoundEntityToGroupBox. [Figure 28] FIG. 1 illustrates an example of a presentation that includes region information stored within a media file. [Figure 29] FIG. 29 shows an example of a region track 2920. [Diagram 30] FIG. 11 is a diagram showing another example of the embodiment. [Figure 31A] FIG. 31 shows an example of a region track 3110, for example an 'rgan' track. [Figure 31B] FIG. 3 shows an example of including a description of a combined region 3155 in the metadata portion of a combined region track 3150. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0018] [First embodiment] First, an example of the hardware configuration of an image file storage device 100 functioning as an information processing device according to this embodiment will be described with reference to the block diagram of Fig. 1. As shown in Fig. 1, the functional units of the image file storage device 100 are configured to be able to communicate with each other via a system bus 109. Note that, in this embodiment, a case will be described in which the functional units shown in Fig. 1 are implemented by hardware, but some of the functional units, or functional units that perform some of the functions of some of the functional units, may be implemented by software (computer program). In this case, the computer program is executed by the CPU 101, and as a result, the corresponding function is realized.

[0019] The CPU 101 executes various processes using computer programs and data stored in the RAM 103 and the ROM 102. As a result, the CPU 101 controls the operation of the entire image file storage device 100, and executes or controls various processes that will be described as being performed by the image file storage device 100.

[0020] ROM 102 is an example of a non-volatile storage device capable of permanently storing information. ROM 102 stores setting data for image file storage device 100, computer programs and data related to the startup of image file storage device 100, computer programs and data related to basic operations of image file storage device 100, etc. The data stored in ROM 102 includes, for example, parameters and display data required for the operation of each functional unit.

[0021] The RAM 103 is an example of a volatile storage device capable of temporarily storing information. The RAM 103 has an area for storing computer programs and data loaded from the ROM 102 or the non-volatile memory 110, and an area for storing captured images input from the imaging unit 104. The RAM 103 further has an area used by the image processing unit 105 when executing various processes, an area for storing data received from the outside by the communication unit 108, and a work area used by the CPU 101 when executing various processes. In this way, the RAM 103 can provide various areas as appropriate.

[0022] For example, the RAM 103 is used not only as an area for loading computer programs, but also as a storage area (output buffer) for temporarily storing data output during the operation of various functional units.

[0023] The imaging unit 104 photoelectrically converts an optical image formed on an imaging surface of an imaging element (e.g., an imaging element such as a CMOS sensor or a CCD) via an optical system (not shown), and performs various image processing (such as noise removal processing and gain processing) on ​​an analog signal obtained by the photoelectric conversion. The imaging unit 104 then A / D converts the analog signal that has been subjected to various image processing into a digital signal, and outputs the digital signal as a captured image.

[0024] The image processing unit 105 executes various image processing for the image. The image processing includes, for example, processing related to development such as gamma conversion, color space conversion, white balance processing, exposure correction, etc. The image processing unit 105 may also be capable of executing image analysis processing and synthesis processing for synthesizing two or more images.

[0025] The image processing unit 105 has an encoding / decoding unit 111, a metadata processing unit 112, a generation unit 113, and a recognition processing unit 114. For ease of understanding, this embodiment will be described assuming that processing by these functional units (encoding / decoding unit 111, metadata processing unit 112, generation unit 113, and recognition processing unit 114) is executed in one piece of hardware, the image processing unit 105. However, the processing by these functional units (encoding / decoding unit 111, metadata processing unit 112, generation unit 113, and recognition processing unit 114) may be executed by a plurality of pieces of hardware.

[0026] The encoding / decoding unit 111 is a codec for moving images and still images conforming to H.265 (HEVC), H.264 (AVC), H.266 (VVC), AV1, JPEG, etc. The encoding / decoding unit 111 executes encoding and decoding of images (still images and moving images (image sequences)) handled by the image file storage device 100.

[0027] The metadata processing unit 112 acquires image data (encoded data) encoded by the image encoding / decoding unit 111. The metadata processing unit 112 generates an image file that includes the encoded data and metadata related to the encoded data and conforms to a predetermined file format (for example, HEIF). Specifically, the metadata processing unit 112 executes an analysis process of the encoded data stored in an image file that includes a still image, an image sequence, etc., generates information about the still image or image sequence, and acquires parameters related to the encoded data. The metadata processing unit 112 executes a process of storing these pieces of information as metadata together with the encoded data in the image file. Note that the metadata processing unit 112 can also generate an image file that conforms to a format other than HEIF, such as other video file formats defined by MPEG or JPEG. Note that the encoded data to be acquired may be data previously stored in the ROM 102 or non-volatile memory 110, or data acquired via the communication unit 108 and stored in the RAM 103. The metadata processing unit 112 also generates and stores metadata according to the area information and combination information of area information generated by the generation unit 113. Furthermore, the metadata processing unit 112 analyzes metadata stored in an image file and performs metadata processing when playing back still images or image sequences. The metadata processing unit 112 also performs processing such as generating metadata related to information identifying image regions to be stored in an image file from region information and combined region information generated by the generation unit 113, and generating metadata that associates region information with images. The metadata processing unit 112 also generates metadata related to annotation information to be associated with images and region information. The metadata processing unit 112 also performs analysis processing of these metadata when playing back images.

[0028] The generating unit 113 generates area information capable of identifying the object according to the information of the object detected from the image by the recognition processing unit 114, and generates type information indicating a combination of the area information. Furthermore, the generating unit 113 generates annotation information to be added to the area information and combination information according to the combination. Note that the generation of the area information is not limited to the detection result of the object by the recognition processing unit 114, and may be information on an area specified by the user by operating the operation input unit 107.

[0029] The recognition processing unit 114 performs object recognition processing using AI or the like on the image to be stored in the image file, and performs processing to detect the subject object from the image. In the object recognition processing, the detection of the subject object may be performed by content analysis, and such processing may be executed by, for example, an external device that can communicate with the image file storage device 100 via the communication unit 108. In this case, the image file storage device 100 obtains the detection result of the subject object from the external device. The recognition processing unit 114 obtains information such as the position of the subject object in the image and the area of ​​the subject object in the image. In the object recognition processing, various detections such as multiple objects and the relationship between each object can be performed according to a recognition algorithm.

[0030] The display unit 106 is a display device having a liquid crystal display (LCD), a touch panel screen, etc., and may be a display device detachably connected to the image file storage device 100, or may be a display device integrated with the image file storage device 100. The display unit 106 displays (live view display) images (still images and moving images) being captured by the imaging unit 104, and displays information / GUI (Graphical User Interface) for various settings, etc. The display unit 106 also displays images when a generated image file is played back.

[0031] Operation input unit 107 is a user interface such as an operation button, a switch, a mouse, a keyboard, etc., and can be operated by a user to input various instructions to CPU 101. In addition, in an aspect in which display unit 106 is a touch panel screen, operation input unit 107 may include a touch panel sensor.

[0032] The communication unit 108 is a communication interface for performing data communication with an external device. The communication unit 108 may be, for example, a network interface that connects to a network and transmits and receives transmission frames. In this case, the communication unit 108 may be, for example, a PHY and MAC (transmission media control processing) that can connect to a wired LAN using Ethernet (registered trademark). Alternatively, when the communication unit 108 can be connected to a wireless LAN, the communication unit 108 may include a controller that executes wireless LAN control such as IEEE802.11a / b / g / n / ac / ax, an RF circuit, and an antenna.

[0033] The non-volatile memory 110 is a non-volatile information storage device with a large storage capacity, such as an SD card, a Compact Flash (registered trademark), a flash memory, etc. For example, the non-volatile memory 110 can store image files generated in this embodiment, and image files acquired from an external device via the communication unit 108.

[0034] 1 is merely one example of a configuration capable of realizing the operation of the image file storage device 100 described below, and can be modified / altered as appropriate. For example, in FIG. 1, the imaging unit 104 is integrated with the image file storage device 100, but it may be detachably connected to the image file storage device 100. Also, the image processing unit 105 may be a device detachable from the image file storage device 100, or may be an external device capable of communicating with the image file storage device 100 via the communication unit 108.

[0035] Next, generation of an image file by the image file storage device 100 will be described. The image file generated by the image file storage device 100 can store a plurality of images and includes information accompanying the images. In the following, a mode will be described in which HEIF is adopted as the file format of the image file, necessary information is derived to generate a compliant image file (HEIF file), and accompanying metadata is configured. However, this is not limited to this, and for example, the file format adopted for the generated image file may be, for example, another video file format defined in MPEG, an omnidirectional media application file format, a file format that handles 3D data such as point cloud data, or a format such as JPEG.

[0036] Next, the file structure of a HEIF file will be explained using Fig. 2. As shown in Fig. 2, a HEIF file 200 generally includes the following four boxes (storage areas):

[0037] The first box 201 is FileTypeBox('ftyp'), and the box 201 stores a brand name that enables a reader of the HEIF file 200 to identify the specifications of the HEIF file 200.

[0038] The second box 202 is an ExtendedTypeBox('etyp'), which stores an extended brand name that enables a reader of the HEIF file 200 to identify the specifications of the HEIF file 200. Here, the 'unif' brand definition and the like are stored, which enables track IDs, item IDs, entity group IDs, and the like to be used as uniquely identifiable values ​​at the file level.

[0039] The third box 203 is a MetaBox ('meta'), and as shown in FIG. 2, multiple types of information relating to an image are stored in different boxes in this box 203, as will be described in detail below.

[0040] The fourth box 204 is a MediaDataBox('mdat'), and the coded data (image) 241-242 is stored in the box 204 as a coded bit stream. The box 204 also stores area information 243-244 indicating an area in the image. The box 204 also stores an Exif data block 245 that stores information at the time of shooting by the imaging unit 104. In this embodiment, the box 204 is used as an area for storing the coded data 241-242, the area information 243-244, and the Exif data block 245. However, the area may also have a box structure such as 'idat' or 'imda'. In the following description, the coded data 241-242 stored in the box 204 will be referred to as "image" or "coded data" using different terms as appropriate.

[0041] When storing video, audio, or an image sequence, a MovieBox ('moov') (not shown) may be stored separately. This box stores metadata for describing various information related to the presentation, such as the video and audio stored in the image file. When the data to be stored is an image sequence, metadata is stored using a mechanism for describing various information related to the video, but unlike video, time limit information is optional.

[0042] Box 211 is a HandlerReferenceBox('hdlr'), and stores a handler type declaration for analyzing the structure of box 203. In the HEIF file 200 generated by the image file storage device 100 of this embodiment, the metadata describing the untimed data stored in box 203 targets still images. Therefore, the handler type name 'pict' is set in box 211 to identify that the target is a still image.

[0043] Box 212 is PrimaryItemBox('pitm'), and specifies the identifier (item ID) of image data that is to be the representative item among the image items stored in the HEIF file 200. The image item specified as the first priority item is played back and displayed as the image that is to be normally displayed.

[0044] Box 213 is an ItemLocationBox('iloc') and stores information indicating the storage location of each information item, including an image item in the HEIF file 200. Box 213 typically describes the storage location of an image item as a byte offset from the beginning of the HEIF file 200 and the data length from the beginning. That is, the information in box 213 makes it possible to identify the positions of the encoded data, region information, and Exif data blocks stored in box 204. Furthermore, for derived items, the information in box 213 indicates that no data exists in box 204. In this case, it indicates that there is no data structure, or that the data for the derived item is stored in box 217 in box 203.

[0045] Box 214 is an ItemInfoBox('iinf'), which defines basic information (item information) such as an item ID and an item type indicating the item type for all items included in the HEIF file 200. The item information specifies not only image items such as encoded image items and derived image items, but also area items indicating areas within an image and Exif information items indicating Exif data blocks.

[0046] Box 215 is an ItemReferenceBox('iref') and stores information describing the association between items included in the HEIF file 200. In an aspect where an image item is a captured image, box 215 is used to describe the association between the image item and its capture information (Exif data, etc.). In an aspect where multiple image items have a derived image relationship, box 215 is used to describe the association between the image items. In an aspect where an area within an image item is indicated, box 215 is used to describe the association between the image item and an area item. An item reference type is specified for the association of each item, making it possible to identify the type of item reference. In addition, the reference relationship between each item is described by describing the item ID specified in box 214 in from_item_ID and to_item_ID.

[0047] Box 216 is an ItemPropertiesBox ('iprp') that stores various attribute information (item properties) for information items included in the HEIF file 200. More specifically, box 216 includes ItemPropertyContainerBox ('ipco'), which is a box 221 that describes the attribute information, and ItemPropertyAssociation ('ipma'), which is a box 222 that indicates the association between the attribute information and each item. Box 221 may store attribute information such as entry data indicating an HEVC parameter set required for decoding an HEVC image item, and entry data indicating the width and height of an image item in pixels. Attribute information that can specify any text information as an item property can also be specified. UserDescriptionProperty ('udes') shown in FIG. 5 is an example of attribute information that can store any text information. UserDescriptionProperty shown in FIG. 5 includes the four-character code 'udes' shown in definition 501, and UserDescriptionProperty is identified using this four-character code. The udes property includes user-defined name information (name) 503, annotation description information (description) 504, and tag information (tags) 505, and can be associated with an item or entity group. The udes property further includes lang 502, and the language used in the udes property is expressed by a language tag string conforming to IETF RFC5646. When multiple udes properties are associated with the same item or entity group, they represent options specified in different languages, and the reader can select the appropriate one. Here, the name information 503 is information that indicates, in a human-readable manner, the name of the item or entity group with which the udes property is associated. Moreover, the annotation description information 504 is information that indicates, in a human-readable manner, a description of the item or entity group with which the udes property is associated, or a text to be presented as an annotation.The tag information 505 is information indicating a tag that is user-defined or automatically assigned and corresponds to an item or entity group with which the udes property is associated, and is configured to be able to include multiple tags separated by commas, for example. By associating the udes property with an area item, it becomes possible to associate annotation information as a text description with a specific area indicated by an area item associated with an image item.

[0048] It is also possible to specify a TransformativeProperty that is intended to transform an image before displaying it as attribute information that can be specified as an item property. For example, this may store data indicating rotation information for rotating an image before displaying it, or data indicating clipping information for cropping an image before displaying it.

[0049] Next, box ('ipma') 222 uses the ID (item ID) of the information item to store, for each item, entry data indicating an association with the attribute information stored in box 221. Note that if there is no attribute information to be associated with an item, such as an Exif data block information item, entry data indicating the association is not stored.

[0050] Box 217 is an ItemDataBox('idat') and stores data related to items included in the HEIF file 200. Box 217 stores, for example, a data structure for describing a derived image item. For example, in the case of an item whose item type is 'grid' shown in box 214, the data structure of a grid derived image item defined by reconstructing an input image in a predetermined grid order is specified. An item reference of a dimg reference type is specified for the input image of the derived image item using box 215. Note that in the case of a derived item that does not have a data structure, for example an identity derived image item 'iden', box 217 does not store any data structure.

[0051] Box 218 is GroupListBox('grpl') and stores metadata for grouping and storing entities such as items and tracks included in the HEIF file 200. Box 218 stores a box that is an extended definition of EntityToGroupBox shown in FIG. 4 for each grouping type parameter. EntityToGroupBox includes grouping_type shown in definition 401. Grouping_type includes a four-character code defined for each grouping type, and this four-character code is used to identify the grouping type of EntityToGroupBox. The grouping type is a concept that specifies the relationship between multiple entities included in a group. EntityToGroupBox includes group_id 402 for uniquely identifying the entity group itself, and num_entities_in_group 403 indicating the number of entities included in the entity group. EntityToGroupBox also includes the number of entity_ids 404 specified in num_entities_in_group. In entity_id 404, an item ID for identifying an item defined in box 214 or a track ID for identifying a single track of a presentation included in a MovieBox (not shown) can be specified. Furthermore, for an entity group of a specific group type, it is possible to specify a group_id that identifies other entity groups. Furthermore, EntityToGroupBox is configured to be extensible and defined for each grouping type, and is defined as a structure in which extension parameters according to the grouping type can be defined in part 405. In this way, EntityToGroupBox makes it possible to handle entities such as multiple image items and tracks contained in a group as meaningful group units depending on the grouping type. In the box configuration shown in Fig. 2, boxes 231 and 232 (CombinedRegionEntigyGroupBox('cbrg')) are stored that are obtained by extending EntityToGroupBox with the grouping type 'cbrg' as one of the group types that group region information.As described above, boxes 231 and 232 are extended definitions of EntityToGroupBox, and therefore contain the definitions 402 to 404 contained in EntityToGroupBox, and grouping_type 401 contains 'cbrg' as the four-character code (4CC) that identifies CombinedRegionEntityGroupBox.

[0052] Additionally, the item ID indicating the area item is specified in entity_id, and each area is combined. Note that the group ID of an entity group that groups separately defined areas can also be specified in entity_id. This allows combinations of area items and area item groups, and combinations between area item groups. Note that detailed definitions of the 'cbrg' entity group and other grouping types will be described later.

[0053] Next, the structure of the region information 243-244 stored in the box 204 will be described with reference to FIG. 3 showing its definition. The item with the item type 'rgan' shown in the iinf box defines a region item intended to identify a region in an image. For the association between the region information and the encoded data, an item reference of the cdsc reference type is specified using the iref box, and the region in the image is defined. As shown in FIG. 3, one region information 243-244 includes version information 301 of the data structure, and flags 302 for switching and handling the process or structure depending on the value. In this embodiment, the version information 301 is specified as 0. The region information also includes data size information 303 indicating the size of the parameter (field_size) handled in the data structure. In this embodiment, the size handled in the data structure of the region information 243 or region information 244 is configured to be switchable between 16 bits and 32 bits, and one of the data sizes is determined based on the value of the flags.

[0054] In addition, as shown in FIG. 3, spatial size information indicating the two-dimensional size of the reference space for defining the area related to the area information 243, 244 is included. As shown in FIG. 3, the spatial size information includes reference_width 304 indicating the width of the reference space and reference_height 305 indicating the height of the reference space. Considering that a HEIF file can store images of multiple image sizes and that the image size can be changed by editing, it is not efficient to store the area information 243, 244 for each image size of the stored images. For this reason, in this embodiment, a reference space is introduced to assign the image size targeted by the area information 243, 244 to the entire image, and various information of the area is defined for the reference space, thereby relatively defining the area independent of the image size of the image. For example, in an aspect in which the reference space is 1024 pixels x 512 pixels, the area specified for an image of the same image size is defined in the image with values ​​equal to the position, width, and height indicated by area shape information 307 described later. On the other hand, an area specified for a 2048 pixel x 1024 pixel image is defined in the image by a position, width, and height that are double the values ​​indicated by area shape information 307 described below. That is, area information 243, 244 defines the area as the relative position and shape in the entire image, and the spatial size information determines the two-dimensional size of the reference space to be assigned to the entire image.

[0055] As shown in FIG. 3, the region count information 306 indicates the number of regions (region_count) defined by the region information 243, 244, and region shape information 307 indicates the shape (geometry_type) of each of the defined regions. In the region information 243, 244 of this embodiment, any of a point, a rectangle, an ellipse, a polygon, a continuous line, a referenced mask, and an inline mask can be selected as the shape of the region, and the shape is specified by the value of geometry_type. In this embodiment, seven types of two-dimensional shapes can be selected as the region, but this is not limited to this. In other words, it can be easily understood that, as long as any space in an image can be specified, for example, a line, a triangle, a free curve, a 3D shape, etc. may be used as the region shape information 307.

[0056] Here, the method of describing specific parameters for identifying the region in the region shape information 307 varies depending on the shape. When the shape of the region is a point (geometry_type is 0), the region is identified by position information (x, y) 308 indicating the coordinates of the point in the reference space. When the shape of the region is a rectangle (geometry_type is 1), the region is identified by position information (x, y) 309 indicating the coordinates of the upper left point (reference point) of the rectangle in the reference space, and shape definition information 310 indicating the width (width) and height (height) of the rectangle. When the shape of the region is an ellipse (geometry_type is 2), the region is identified by position information (x, y) 311 indicating the coordinates of the center of the ellipse in the reference space, and shape definition information 312 indicating the length of the radius (radius_x) in the x-axis direction and the radius (radius_y) in the y-axis direction of the ellipse. Also, when the shape of the region is a polygon (geometry_type is 3) or a continuous line (geometry_type is 6), the region is specified by vertex number information 313 indicating the number of vertices of the polygon in the reference space, and position information (px, py) 314 indicating the coordinates of each vertex. The difference between the shapes of a polygon (geometry_type is 3) and a continuous line (geometry_type is 6) is that a shape connecting the start point and end point of the position information indicating the coordinates of each vertex is a polygon (geometry_type is 3), and a shape not connecting the start point and end point is a continuous line (geometry_type is 6). Also, when the shape of the region is a reference mask (geometry_type is 4), another item defined as a mask image item is associated with the coded image using the reference type mask of the iref box. The region is specified by position information (x, y) 315 indicating the coordinates of the top left point (reference point) of the reference mask in the reference space, and shape definition information 316 indicating the width and height of the reference mask. If 0 is specified for the width and height, the size information is associated with the reference mask image item as an item property. If the shape of the region is an inline mask (geometry_type is 5), the image portion corresponding to pixels with a value of 1 in the mask image stored in data 321 indicates the region.The region is specified by position information (x, y) 317 indicating the coordinates of the top left point (reference point) of the region (rectangle) in the reference space, and shape definition information 318 indicating the width and height of the region. In addition, as shown in FIG. 3, mask_coding_method 319 indicating the coding method applied to the mask included in the data is included. When the value of mask_coding_method 319 is 0, no coding is applied to the mask, and when the value of mask_coding_method 319 is 1, it indicates that deflate compression defined in RFC1951 is applied. Other values ​​are reserved and not specified. mask_coding_parameters 320 indicates additional coding parameters required to properly process the coded mask data. When the value of mask_coding_method 319 is 1, mask_coding_parameters 320 indicates the number of bytes of coded mask array data. The value of mask_coding_parameters is reserved when the value of mask_coding_method is greater than 1. Data321 contains the coded or uncompressed representation of the mask, including the pixels of the inline mask, in raster scan order. Each pixel is represented by one bit, and eight pixels are packed into a byte. The byte packing is in big endian order. If the width of the mask is not a multiple of eight pixels, there shall be no padding at the end of each line. Only the last data byte shall be padded with bits set to zero.

[0057] Next, a definition is shown for setting a region for any image item stored in the HEIF file 200 using region information 243, 244 configured according to such a definition, and storing the region in association with a region annotation. A region annotation is composed of metadata associated with one or more regions of an image item, or other image items. In addition, by grouping region items into entity groups according to the grouping type in box 218, it is possible to handle regions as region entity groups combined according to the grouping type. Grouped regions can be uniquely identified by a group ID.

[0058] In the image file storage device 100 of this embodiment, by adding area information 243, 244 and information related to boxes 231, 232 to the boxes below box 203, it is possible to associate area annotations with areas and combination areas in specific image items.

[0059] First, in the HEIF file 200 of this embodiment, the encoded data 241-242 and the area information 243-244 are stored in the box 204, and in order to distinguish between them, an item type is set for each data in the box 214. For example, the item type is set to 'hvc1' for the encoded data 241, and 'rgan' for the area information 243. The storage location in the HEIF file 200 of the area information 243-244 stored in the box 204 is stored in the box 213, similar to the encoded data 241-242. As described above, the area information 243, 244 is information for relatively specifying an area in the reference space, and therefore it is necessary to specify the image item for which the area is set. Information on which image item an area is set for by one piece of area information 243 is stored in the box 215. That is, in order to describe the association between the items set in the box 214, for example, information identifying an image item related to the coded data 241 that sets an area using the area information 243 for an area item related to the area information 243 is stored in the box. In addition, the area item can be set in the derived image by associating it with a derived image item configured as a derived image. In addition, when the area item is treated as an area entity group, a grouping type for grouping the areas is determined, and grouping_type is determined. In order to treat the areas specified in each area item as an area obtained by logically adding up the areas, cbrg is specified as the group type. In addition, the ID of the area item is specified as the entity that constitutes the entity group. This makes it possible to specify the logically added areas as one area group. In addition, in order to associate the image items with the grouped area groups, information on which image items one area group sets an area for is stored in the box 215. Note that since each area item included in the entity group is associated with an image item, the area group may not be associated with an image item.

[0060] However, for a region thus established for a particular image item, region annotations for that region may be established by associating any or all of the following with a region item or region entity group:

[0061] The first is a region annotation using item properties. The udes property stored in box 221 in box 216 is associated with a region item or region entity group by box 222. By associating the udes property with a region item or region entity group, it becomes possible to associate annotation information as a text description.

[0062] The second is a region annotation using a metadata item, in which a metadata item such as Exif that stores metadata in mdat is associated with a region item or a region entity group by a reference type cdsc in box 215.

[0063] The third is region annotation using other image items or entity groups. Region items or region entity groups are associated with other image items or entity groups by reference type eroi. This allows for providing different image representations for specific regions. Such associations allow for region annotations to be set for specific regions within an image.

[0064] Next, a definition of a method for specifying a combined region of region items that can be stored in the HEIF file 200 will be described. FIG. 6 is a diagram showing the data structure of CombinedRegionEntityGroupBox, which is an entity group that groups regions that can be stored in the box 218 of the HEIF file 200. This CombinedRegionEntityGroupBox is an extension of EntityToGroupBox and has grouping_type'cbrg' 601. No additional parameters specific to this entity group type are defined. Such a CombinedRegionEntityGroup can be defined as follows. It should be readily understood that the name of this entity group and the 4CC that identifies this entity group may be an alias or a different 4CC. For example, an alias for indicating the combination may be UnionOfRegionsEnityGroupBox, and it may be identifiable by 'unrg' or the like.

[0065] A CombinedRegionEntityGroup is identified by the 'cbrg' grouping_type and represents a set of regions to create a combined region by combining each region represented by the contained entities through a logical OR. The entity_id is specified as an entityToGroupBox representing another combined region when unique IDs are used by the region item, region metadata track or unif brand. Specifying other combined regions allows for more flexible region specification. The region is formed by logical ORing these regions. That is, all regions represented by any of the regions represented by entity_id are identified as the region represented by CombinedRegionEntityGroup. Multiple CombinedRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0066] It is recommended that one entity specified in entity_id indicates one region (for example, region_count in the region item is 1). In this case, it is recommended that the number of entities specified (num_entities_in_group) is 2 or more. On the other hand, if one entity indicates two or more regions (for example, region_count in the region item is 2 or more), it may be combined with the regions indicated by other entities by a logical OR of all the regions. In this case, the number of entities specified (num_entities_in_group) may be 1. As another method, if the number of regions indicated by the entities specified in the group is the same (the region_count of all entities is the same), each region may be combined in the order in which the regions are indicated. The method of combining various regions according to the number of regions may be separately controlled by the value of flags included in the definition of CombinedRegionEntityGroupBox. The region indicated by the CombinedRegionEntityGroup and each region indicated by the entities included in the CombinedRegionEntityGroup can be individually associated with an image item, and each region in the image can be identified. In addition, different annotation information can be individually associated with each region by the method defined in the region annotation.

[0067] FIG. 7 is a diagram showing the data structure of MultiplyRegionEntityGroupBox, which is an entity group that groups regions that can be stored in box 218 of HEIF file 200. This MultiplyRegionEntityGroupBox is an extension of EntityToGroupBox and has grouping_type 'mprg' 701. No parameters specific to this entity group type are defined. Such a MultiplyRegionEntityGroup may be defined as follows. It should be readily understood that the name of this entity group and the 4CC that identifies this entity group may be an alias or a different 4CC. For example, an alias for indicating the combination may be RegionsIntersectionEnityGroupBox, and it may be identifiable by 'isrg' or the like.

[0068] MultiplyRegionEntityGroup is identified by the 'mprg' grouping_type and represents a set of regions to create a combined region by combining each region represented by the contained entities through a logical conjunction. The entity_id is specified as an entityToGroupBox representing another combined region when a unique ID is used by the region item, region metadata track, or unif brand. Specifying another combined region allows for more flexible region specification. A region is formed by logical conjunction of these regions. In other words, the common region included in all of the regions represented by entity_id is identified as the region represented by MultiplyRegionEntityGroup. Multiple MultiplyRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0069] It is recommended that the number of regions indicated by one entity specified in entity_id (for example, region_count in the region item is 1). In this case, it is recommended that the number of entities specified (num_entities_in_group) is 2 or more. On the other hand, if the number of regions indicated by one entity is 2 or more (for example, region_count in the region item is 2 or more), it may be combined with the regions indicated by other entities by logical product of all the regions. In this case, the number of entities specified (num_entities_in_group) may be 1. As an alternative, if the number of regions indicated by one entity is 2 or more, they may be treated as one region and combined with the regions indicated by other entities by logical product. In other words, a common region between different region entities is specified as a combined region. As a further alternative, if the number of regions indicated by the entities specified in the group is the same (the region_count of all entities is the same), each region may be combined in the order of indication of the region. The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of MultiplyRegionEntityGroupBox. Each region represented by the MultiplyRegionEntityGroup and each of the entities contained within it can be individually associated with an image item, can identify a region within the image, and can have different annotation information associated with it, depending on how the region annotation is defined.

[0070] FIG. 8 is a diagram showing the data structure of an InverseRegionEntityGroupBox, which is an entity group that groups regions that can be stored in box 218 of a HEIF file 200. This InverseRegionEntityGroupBox is an extension of EntityToGroupBox and has grouping_type 'ivrg' 801. No additional parameters specific to this entity group type are defined. Such an InverseRegionEntityGroup may be defined as follows. It should be readily understood that the name of this entity group and the 4CC that identifies this entity group may be an alias or a different 4CC. For example, an alias for indicating the combination may be NegationRegionEnityGroupBox, and it may be identifiable by 'ngrg' or the like.

[0071] InverseRegionEntityGroup is identified by the 'ivrg' grouping_type and indicates a set of regions to create a combined region by combining each region indicated by the contained entities through logical negation or negative OR. entity_id is specified as the region item, region metadata track, or, if unique IDs are used by the unif brand, EntityToGroupBox indicating another combined region. Specifying another combined region allows for more flexible region specification. A region is formed by logical negation or negative OR of these regions. In other words, a region that is not included in any of the regions indicated by entity_id is identified as a region indicated by InverseRegionEntityGroup. Multiple InverseRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0072] It is recommended that an entity specified in entity_id indicates only one region (for example, region_count in a region item is 1). In this case, if the number of entities specified (num_entities_in_group) is one, the region obtained by logically negating that region will be specified as the combined region. In other words, the region other than the regions indicated by the included entities will be the target region. On the other hand, if two or more entities are specified, the region obtained by logically negating the regions indicated by the included entities will be specified as the combined region. In other words, the regions indicated by all entities that are not specified as any region will be the target region.

[0073] On the other hand, if the number of regions indicated by one entity is two or more (for example, the region_count in the region item is two or more), it may be combined with the regions indicated by other entities by a NOR of all the regions. As another method, if the number of regions indicated by the entities specified in the group is the same (the region_count of all entities is the same), the respective regions may be combined in the order in which they are indicated. The method of combining various regions according to the number of regions may be separately controlled by the value of flags included in the definition of InverseRegionEntityGroupBox. The region indicated by the InverseRegionEntityGroup and each region indicated by the entities included in the InverseRegionEntityGroup can be associated with an image item individually, and each region in the image can be identified. Also, different annotation information can be associated with each individually by the method defined in the region annotation.

[0074] Similarly, individual EntityToGroupBox grouping types may be defined to specify other logical combinations of region information, such as exclusive-or, not-and, not-exclusive-or, etc.

[0075] 9 is a diagram showing the data structure of LogicalOperateRegionEntityGroupBox, which is an entity group that groups regions that can be stored in box 218 of HEIF file 200. This LogicalOperateRegionEntityGroupBox is an extension of EntityToGroupBox and has grouping_type 'lorg' 901. LogicalOperateRegionEntityGroupBox includes logical_operation shown in definition 902 as an additional parameter specific to the group type. Such a LogicalOperateEntityGroup can be defined as follows. It should be readily understood that the name of this entity group and the 4CC identifying this entity group may be an alias or a different 4CC.

[0076] LogicalOperateEntityGroup is identified by 'lorg' grouping_type and indicates a set of regions to create a combined region by combining each region indicated by the contained entities through a logical operation. entity_id is specified as a region item, a region metadata track, or, if a unique ID is used by the unif brand, EntityToGroupBox is specified to indicate another combined region. Specifying another combined region allows for more flexible region specification. These regions are formed by logical operation. The operator of the logical operation is provided by the logical_operation parameter. In other words, the region resulting from performing a logical operation on each region indicated by entity_id using the operation method indicated by the logical_operation parameter is identified as the region indicated by LogicalOperateRegionEntityGroup. Multiple LogicalOperateRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0077] logical_operation indicates the type of logical operation and is an unsigned integer indicating the logical operator applied to the area entities contained in the group. A value of 0 indicates the logical combination of areas by a logical OR operation. A value of 1 indicates the logical combination of areas by a logical AND operation. A value of 2 indicates the logical combination of areas by logical NOT or NOR. A value of 3 indicates the logical combination of areas by XOR. A value of 4 indicates the logical combination of areas by NAND. A value of 5 indicates the logical combination of areas by NEXOR. Other values ​​are reserved.

[0078] The number of regions and entities indicated by one entity specified in entity_id can be handled in the same manner as above, and the method can be defined. In addition, the method of combining various regions according to the number of regions can be controlled separately by the value of flags included in the definition of LogicalOperateRegionEntityGroupBox. This behavior can be determined by the combination of the value of logical_operation and the value of flags.

[0079] The region represented by the LogicalOperateRegionEntityGroup and each of the regions represented by the entities contained in the LogicalOperateRegionEntityGroup can be individually associated with an image item, can identify a region within the image, and can have different annotation information associated with it, depending on how the region annotations are defined.

[0080] In this embodiment, the operator of the logical operation is included in the LogicalOperateRegionEntityGroupBox and is provided by the logical_operation parameter, but it may be controlled by using flags. Furthermore, it may be associated as a property with the group of LogicalOperateRegionEntityGroup by using the above-mentioned item property association structure. The data structure in this case will be described with reference to FIG. 15. LogicalOperateRegionProperty('lorg') shown in FIG. 15 is an example of attribute information capable of storing logical operator information. The LogicalOperateRegionProperty shown in FIG. 15 includes the four-character code 'lorg' shown in definition 1501, and this four-character code is used to identify the LogicalOperateRegionProperty. The lorg property includes logical_operation 1502 that provides a logical operation method, and can be associated with an area item or an area entity group. The definition of logical_operation 1502 is the same as that described in LogicalOperateRegionEntityGroupBox. The area item or area entity group to which the lorg property is associated applies the logical operation specified in logical_operation. The area item may be a derived area item. In addition, in this embodiment, logical operations that apply to domain items or domain entity groups have been shown, but they may also be applied to different types of items or entities defined separately. For example, an operation such as swapping the left and right stereo pairs may be applied by applying logical negation to a stereo entity group that indicates a stereo pair. Also, by defining it as an item property in this way, it is possible to directly associate it with a domain item.

[0081] So far, we have explained the group type that logically combines areas specified as entity groups. Next, we will explain the grouping type that indicates the relationship between each entity included in the group.

[0082] FIG. 10 is a diagram showing the data structure of an InclusionRelationRegionEntityGroupBox, which is an entity group that groups regions that can be stored in box 218 of a HEIF file 200. This InclusionRelationRegionEntityGroupBox is an extension of EntityToGroupBox and has grouping_type 'irrg' 1001. No additional parameters specific to this entity group type are defined. Such an InclusionRelationRegionEntityGroup may be defined as follows. Note that it is easily understood that the name of this entity group and the 4CC that identifies this entity group may be an alias or a different 4CC. For example, an alias for indicating a combination may be CompoundRegionEnityGroupBox, and it may be identifiable by 'corg' or the like.

[0083] InclusionRelationRegionEntityGroup indicates a set of regions to create a combined region relationship to indicate the relationship between regions that indicate the inclusion of included entities, identified by the 'irrg' grouping_type. entity_id is specified as an entityToGroupBox indicating another combined region when a unique ID is used by the region item, region metadata track, or unif brand. Specifying another combined region allows for more flexible region specification. It is identified as a region where the second or subsequent entity_ids are included in the region indicated by the first entity_id. Multiple InclusionRelationRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0084] It is recommended that an entity specified in entity_id indicates one region (for example, region_count in the region item is 1). In this case, it is recommended that the number of entities specified (num_entities_in_group) is 2 or more. On the other hand, if an entity indicates two or more regions (for example, region_count in the region item is 2 or more) and it is the first entity_id, the relationship may be defined as the first specified region as region information includes all subsequent regions and regions indicated by the second and subsequent entity_ids. In this case, the number of entities specified (num_entities_in_group) may be 1. Alternatively, the region indicated by the first entity_id may be treated as one region, and defined so that the second and subsequent entity_ids are included. In this case, it is recommended that the number of entities specified (num_entities_in_group) is 2 or more. Alternatively, if the number of regions indicated by the entities specified in the group is the same (the region_count of all entities is the same), the order in which the regions are indicated may be defined such that each region specified by the first entity_id includes the regions specified by the second and subsequent entity_ids. The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of InclusionRelationRegionEntityGroupBox. The region indicated by the InclusionRelationRegionEntityGroup and each region indicated by the entities included in the InclusionRelationRegionEntityGroup can be associated with an image item individually. Also, different annotation information can be associated with each individually by the method defined in the region annotation.In addition, since the InclusionRelationRegionEntityGroup is a combination type that indicates the relevance between regions, if each region indicated by an entity included in the InclusionRelationRegionEntityGroup is associated with an image, the InclusionRelationRegionEntityGroup may not be associated with an image.

[0085] Alternatively, such an entity group may be defined as

[0086] A CompoundRegionEntityGroup ('corg') associates one or more region items with one main region item. It indicates a logical composition relationship between a main object covered by the region of the main entity and other objects covered by regions described by one or more other entities, where the main object is logically composed of other objects. For example, a CompoundRegionEntityGroup can be used to associate a main region corresponding to a car with regions corresponding to tires (wheels), indicating that the car is logically composed of wheels.

[0087] The entities in a compound domain entity group are domain items. The number of entities in a compound domain entity group MUST be at least two. The first entity_id value MUST identify the main domain item. The second entity_id value MUST identify the domain that covers the main object that is logically composed of the objects covered by the domains described by the entity_ids that follow.

[0088] This logical composition relationship does not convey any information at the geometry level: a main region signalled as composed of other regions by a composite entity group may or may not geometrically contain the other regions.

[0089] An example of application of this InclusionRelationRegionEntityGroup (or CompoundRegionEntityGroup) is when a tree is included as an object in an image, the object region is identified, and the fruit is included as an object in the same image, and the object region is identified as a separate region. This makes it possible to indicate which tree the fruit is included in the region identified as fruit. In addition, by defining the overall region that identifies a person (the first region item specified in InclusionRelationRegionEntityGroup) and the shapes that identify hands, feet, head, etc. as the region items included therein (the second and subsequent region items specified in InclusionRelationRegionEntityGroup), and defining region items of point shapes that indicate the skeleton or the joint connection points and configuring them to be included in this entity group, it is possible to describe information such as human detection, skeleton estimation, posture estimation, and behavior prediction detection, and also to describe the relationship between regions. For example, the first area specified can simply detect a person and associate the name, etc. as an annotation, indicate which joint or body part the bones and joint connections belong to, and add annotations related to actions such as running or reaching out to the group as annotations for the relationships shown using this EntityGroup. By combining this information, it is possible to add individual annotations to information detected by separate AI algorithms, etc.

[0090] 11 is a diagram showing the data structure of TouchRelationRegionEntityGroupBox, which is an entity group that groups regions that may be stored in box 218 of HEIF file 200. This TouchRelationRegionEntityGroupBox is an extension of EntityToGroupBox and has grouping_type 'tcrg' 1101. No additional parameters specific to this entity group type are defined. Such a TouchRelationRegionEntityGroup may be defined as follows. Note that it will be easily understood that the name of this entity group and the 4CC identifying this entity group may be an alias and a different 4CC.

[0091] TouchRelationRegionEntityGroup is identified by the 'tcrg' grouping_type and indicates a set of regions to create a combined region relationship to indicate the association between the regions that the contained entities are in a touching relationship. The entity_id is two and if unique IDs are used by the region item, region metadata track, or unif brand, EntityToGroupBox is specified to indicate other combined regions. Specifying other combined regions allows for more flexible region specification. Two regions in an image with this entity group associated indicate that the subject objects are in a touching relationship. This does not simply indicate that the regions are touching or overlapping, but that the subject objects in the image are in a touching relationship. Multiple TouchRelationRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0092] It is recommended that an entity specified in entity_id indicates one region (for example, region_count in a region item is 1). On the other hand, if the number of regions included in entity_id is two, the number of entities specified (num_entities_in_group) may be one, in which case it indicates that the two regions included in one entity are in contact with each other.

[0093] In addition, when the number of regions indicated by one entity is two or more (for example, the region_count in the region item is two or more) or when the number of entities is two or more, all the regions of the first entity and all the regions indicated by the second entity may be defined as being in a contact relationship. Furthermore, if the number of regions indicated by the entities specified in the group is the same (the region_count of all entities is the same), the order in which the regions are indicated may be defined so that the region specified by the first entity_id and the region specified by the second or subsequent entity_id are in a contact relationship. The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of TouchRelationRegionEntityGroupBox. The region indicated by TouchRelationRegionEntityGroup and each region indicated by the entities included in TouchRelationRegionEntityGroup can be associated with an image item individually. Also, different annotation information can be associated with each of them individually by the method defined in the region annotation. Note that since TouchRelationRegionEntityGroup is a combination type that indicates the relationship between regions, if each region indicated by an entity included in TouchRelationRegionEntityGroup is associated with an image, TouchRelationRegionEntityGroup may not be associated with an image.

[0094] An example of application of this TouchRelationRegionEntityGroup is when a person is included as a subject in an image, the region of the subject is identified, and a ball held by the person is included as a subject in the same image, and the region of the subject is identified as a separate region. Region entities indicating the person and the ball are stored as a group. This makes it possible to identify that the person is holding the ball. For example, it is possible to identify whether the regions overlap after the ball is thrown, or whether the person is holding the ball before throwing it. In this embodiment, a grouping type indicating a contact relationship has been described, but a grouping type indicating non-contact may also be used. In addition, both a contact relationship and a non-contact relationship may be defined by defining flags or additional parameters in the TouchRelationRegionEntityGroupBox. FIG. 12 is a diagram showing the data structure of DepthOrderRelationRegionEntityGroupBox, which is an entity group that groups regions that can be stored in the box 218 of the HEIF file 200. This DepthOrderRelationRegionEntityGroupBox is an extension of EntityToGroupBox, and has grouping_type'dorg'1201. DepthOrderRelationRegionEntityGroupBox includes depth_direction shown in definition 1202 as an additional parameter specific to the group type. Such a DepthOrderRelationRegionEntityGroup can be defined as follows. It should be readily understood that the name of this entity group and the 4CC identifying this entity group may be an alias and a different 4CC.

[0095] A DepthOrderRelationRegionEntityGroup is identified by the 'dorg' grouping_type and represents a set of regions to create a combined region relationship to indicate the relationship between regions due to foreground / background context of the contained entities. The entity_id is specified by a region item, a region metadata track, or, if unique IDs are used by the unif brand, an EntityToGroupBox representing another combined region. Specifying another combined region allows for more flexible region specification. The entity_id values ​​representing regions within a DepthOrderRelationRegionEntityGroup are listed in depth_direction order. Multiple DepthOrderRelationRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0096] depth_direction indicates the depth (foreground-background) direction and is an unsigned integer indicating the depth orientation of the area entities contained in the group. A value of 0 indicates that the listed area entities are listed from foreground to background. A value of 1 indicates that the listed area entities are listed from background to foreground. All other values ​​are reserved.

[0097] It is recommended that one entity specified in entity_id indicates one region (for example, region_count in the region item is 1). It is also recommended that the number of entities specified (num_entities_in_group) is 2 or more. On the other hand, if one entity indicates two or more regions (for example, region_count in the region item is 2 or more), it indicates that the multiple regions included in one entity are specified in the order indicated by depth_direction in the order of their storage. Alternatively, multiple regions included in one entity may be defined as being at the same position, that is, having the same depth. When one entity includes multiple regions, the number of entities included in the group (num_entities_in_group) may be 1. Alternatively, if the number of regions indicated by the entities specified in the group is the same (the region_count of all entities is the same), it may be defined so that each region is lined up in the order of region count for each region specified by entity_id in the direction indicated by depth_direction. The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of DepthOrderRelationRegionEntityGroupBox. Additionally, additional parameters may be defined to indicate whether the individual regions are of the same depth or of the next depth. The regions represented by the DepthOrderRelationRegionEntityGroup and the regions represented by the entities contained in the DepthOrderRelationRegionEntityGroup may be associated with image items individually, and may be associated with different annotation information, depending on the method defined by the region annotation.Since the DepthOrderRelationRegionEntityGroup is a combination type indicating the relationship between regions, as long as each region indicated by an entity included in the DepthOrderRelationRegionEntityGroup is associated with an image, the DepthOrderRelationRegionEntityGroup may not be associated with an image.

[0098] FIG. 13 is a diagram showing the data structure of ForbidRelationRegionEntityGroupBox, which is an entity group that groups regions that can be stored in box 218 of HEIF file 200. This ForbidRelationRegionEntityGroupBox is an extension of EntityToGroupBox and has grouping_type 'fbrg' 1301. No additional parameters specific to this entity group type are defined. Such a ForbidRelationRegionEntityGroup may be defined as follows. Note that it is easily understood that the name of this entity group and the 4CC that identifies this entity group may be an alias or a different 4CC. For example, an alias for indicating the combination may be OclusionRegionEnityGroupBox, and it may be identifiable by 'ocrg' or the like.

[0099] ForbidRelationRegionEntityGroup is identified by 'fbrg' grouping_type, and indicates a set of regions for creating a combined region relationship to indicate a relationship between regions that indicates that the second specified entity is partially or completely hidden by the first specified entity. The number of entity_id is two, and when a unique ID is used by the region item, region metadata track, or unif brand, EntityToGroupBox is specified to indicate another combined region. By specifying another combined region, more flexible region specification is possible. Two regions in an image to which this entity group is associated indicate that one region has a hidden relationship with an object indicated by the other region. For example, if the object indicated by one region is hidden by the object indicated by the other region, the hidden object does not appear as a subject in the image. In other words, a hidden region is set based on information that can be inferred from another image, etc., and it indicates that it is hidden by the object indicated by the other region. Multiple ForbidRelationRegionEntityGroups with different group_id values ​​can be specified in the same file.

[0100] It is recommended that an entity specified in entity_id indicates one region (for example, region_count in the region item is 1). On the other hand, if the number of regions included in entity_id is two, the number of entities specified (num_entities_in_group) may be one. In this case, it indicates that the object in the image region indicated by the first region of the two regions included in one entity is hidden by the object in the image region indicated by the second region. In addition, if an entity indicates two or more regions (for example, region_count in the region item is two or more) or the number of entities is two or more, it may be defined that the region of the first entity is hidden by all the regions indicated by the second and subsequent regions. Furthermore, if the number of regions indicated by the entities specified in a group is the same (the region_count of all entities is the same), it may be defined so that the order of the regions indicates that the region specified by the first entity_id is hidden by the region specified by the second or subsequent entity_ids. The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of ForbidRelationRegionEntityGroupBox. The region indicated by the ForbidRelationRegionEntityGroup and each region indicated by the entities included in the ForbidRelationRegionEntityGroup can be associated with an image item individually. Also, different annotation information can be associated with each individually by the method defined in the region annotation. Note that since the ForbidRelationRegionEntityGroup is a combination type that indicates the relationship between regions, if each region indicated by the entities included in the ForbidRelationRegionEntityGroup is associated with an image, the ForbidRelationRegionEntityGroup may not be associated with an image.

[0101] In the above-described definition of the entity group according to each combination type, a method was shown in which the definition according to the number of areas held by the area item that is the input area is performed by flags in each definition. As another method of describing a combination area, a method of realizing the definition by defining the data structure of different entity groups without relying on the control by the value of flags even if one area item specified in the input area or an entity group of the combination area indicates multiple areas will be described with reference to FIG. 27. In the example shown in FIG. 27, a combination area indicating an inclusion relationship between areas is described as an example, but other combination types can also be defined by a similar mechanism. The definition of CompoundEntityToGroupBox shown in FIG. 27 shows a data structure that can be stored in the GroupListBox('grpl') box of box 218 as the data structure of a combination area entity group.

[0102] The data structure shown in FIG. 27 is an example of an extension of a compound region entity group ('corg'). The extended compound region entity group ('corg') is a configuration that allows the description of relationships between multiple compound region groups in one compound region entity group. In a region item, when describing multiple regions in one region item, it was possible to describe them by specifying a value of 2 or more in region_count. Similarly, this method is a definition that allows the description of multiple regions in relationships between regions. In order to identify the first input region (main region) for each region relationship of the number (group_count) defined in the extended compound region entity group, the index (compound_region_item_index) of each input region item and its index (compound_region_index) in the list of regions defined by this input region item are specified. In addition, in order to identify the second and subsequent input regions for each region relationship, the number of regions to be input for each region relationship (comprised_count) is specified. Also, the region to input for each relation is specified by the index of each input region item (region_item_index) and its index (region_index) in the list of regions defined by this input region item.

[0103] Here, version is defined as 0 or 1. Future extensions can be specified by specifying a new version value for this version. Here, flags is 0, and all flag values ​​are reserved. Here, region_count is the number of region relations defined by the composite region entity group. Here, group_count is the number of region groups defined by the composite region entity group. Here, compound_region_item_index is the index of a region item in the list of region items specified in the composite region entity group, and indicates a region item that contains one of the first input regions (main regions) of the composite region. Here, compound_region_index is the index of a region in the region item indicated by the compound_region_item_index field that is one of the composite regions. Here, comprised_count is the number of regions that make up one composite region. Here, region_item_index is the index of a region item in the list of region items specified in the composite region entity group, and indicates a region item that contains one of the second or subsequent input regions of the composite region. Here, region_index is the index of a region in the region item indicated by the region_item_index field that is one of the composite regions.

[0104] In this way, by defining the data structure of the combination domain entity group, it is easy to configure the combination domain entity group regardless of how many domains the domain item that is the input domain is composed of. In addition, when multiple combination domains need to be given the same annotation, it is possible to give a common annotation like existing domain items without having to give an annotation for each combination domain entity group. This data structure can also be applied to other combination types, and in a combination domain entity group that has a parameter such as RegionLogicalDerivation, a similar definition can be made by adding a parameter to this data structure.

[0105] Here, an example of an image file output by the image file storage device 100 according to this embodiment will be described with reference to Fig. 16. Note that in this embodiment, the image file is configured to store in the file a region in which region items are combined by two CombinedRegionEntityGroups as a file data structure.

[0106] In the example of FIG. 16, as shown in the description 1604 corresponding to the box 204, descriptions 1630 and 1631 corresponding to the HEVC encoded data (HEVC Image Data) 241 to 242 are stored. Furthermore, an image file is illustrated that stores descriptions 1632 and 1633 corresponding to the region item data 243 to 244 and a description 1634 corresponding to the Exif data block 245. As shown in the descriptions 1632 and 1633, the region information 243 to 244 conforms to the definition shown in FIG. 3, and each specifies a rectangular region in a reference space with an image size of 4032 pixels x 3024 pixels. In addition, the region specified by the description is specified with the coordinates (x0, y0) of the reference point in the reference space of the region, the image size (w0xh0), the coordinates (x7, y7), and the image size (w7xh7).

[0107] Description 1601 corresponds to box 201, and stores 'mif1' as the type value major-brand of the brand definition that conforms to the HEIF file, and 'heic' as the type value compatible-brands of the compatible brand definition.

[0108] Description 1602 corresponds to box 202, and stores 'unif' as the type value compatible-brands of the extended brand definition conforming to the HEIF file, indicating that the ID value at the file level is a uniquely identifiable value.

[0109] Next, in description 1603 corresponding to box 203, various pieces of metadata describing the untimed data stored in the example output file are shown.

[0110] Description 1610 corresponds to box 211, and the handler type of the MetaDataBox (meta) specified is 'pict'. Description 1611 corresponds to box 212, and stores 5 as the item_ID, and specifies the ID of the image to be displayed as the first priority image.

[0111] Description 1612 corresponds to box 214, and indicates item information (item ID (item_ID) and item type (item_type)) for each item. Each item can be identified by item_ID, and indicates what type of item the item identified by item_ID is. In the example of FIG. 16, since 14 items are stored, entry_count is 14, and 14 types of information are arranged in description 1612, and each item ID and item type are specified. In the illustrated image file, the first to fourth information corresponding to descriptions 1640 to 1641 is an HEVC encoded image item of type hvc1. Moreover, the fifth information shown in description 1642 is a grid derived image item of item type grid. Moreover, the sixth information shown in description 1643 is an HEVC encoded image item of item type hvc1 that becomes a thumbnail image. The seventh to thirteenth information corresponding to descriptions 1644 to 1645 is an area item of item type rgan that indicates an area. The 14th information corresponding to the description 1646 is a metadata item of item type Exif indicating an Exif data block.

[0112] Description 1613 corresponds to box 213, and specifies the storage location of each item in the HEIF file and information on the data size. For example, an encoded image item with item_ID 1 indicates that the offset in the file is O1 and the size is L1 bytes, and the data location in mdatBox is specified.

[0113] Description 1614 corresponds to box 215 and indicates the reference relationship (association) between each item. The item reference shown in description 1650 specifies dimg, which indicates that the reference type is a derived image. It also indicates that the grid derived image item with item_ID5 specified in from_item_ID references the HEVC encoded image items with item_ID1 to 4 specified in to_item_ID. This specifies the encoded image items with item_ID1 to 4 as input images to the derived image item. It also indicates that the item reference shown in description 1651 specifies thmb, which indicates a thumbnail relationship, as the reference type. It also indicates that the HEVC encoded image item with item_ID6 specified in from_item_ID references the grid derived image item with item_ID5 specified in to_item_ID. This indicates that the HEVC encoded image item with item_ID6 is a thumbnail image of the grid derived image item with item_ID5. The item references shown in descriptions 1652, 1653, 1654, 1655, and 1656 specify cdsc, which indicates a content description relationship, as the reference type. Description 1652 indicates that the area information item with item_ID50 specified in from_item_ID references the grid derived image item with item_ID5 specified in to_item_ID. This means that the area information item with item_ID50 indicates a partial area within the grid derived image item with item_ID5. Similarly, description 1653 indicates that the area information item with item_ID56 specified in from_item_ID references the grid derived image item with item_ID5 specified in to_item_ID. This means that the area information item with item_ID56 indicates a partial area within the grid derived image item with item_ID5. Description 1654 indicates that the Exif data block item with item_ID 1000 specified in from_item_ID references the grid derived image item with item_ID 5 specified in to_item_ID.This indicates that the Exif data block item of item_ID1000 describes information at the time of shooting of the grid derived image item of item_ID5. In the description 1655, item_ID100 specified in from_item_ID is interpreted as the group ID of the entity group by the unif brand specification in the etyp box of the description 1602. It is indicated that the grid derived image item of item_ID5 specified in to_item_ID is referenced from the area information entity group of group_ID100 specified in from_item_ID. This indicates that the combination area indicated by the area information entity group of group_ID100 indicates a partial area in the grid derived image item of item_ID5. Note that the configuration of associating the entity group with the image item is not essential. In other words, when the image item and the area items constituting the entity group are associated with each other, the combination area in the image can be identified without associating the entity group with the image item. Similarly, in description 1656, item_ID101 specified in from_item_ID is interpreted as the group ID of the entity group due to the unif brand specification in the etyp box of description 1602. It is indicated that the area information entity group with group_ID101 specified in from_item_ID refers to the grid derived image item with item_ID5 specified in to_item_ID. As a result, the combination area indicated by the area information entity group with group_ID101 indicates a partial area within the grid derived image item with item_ID5.

[0114] Description 1615 corresponds to box 217, and specifies the data structure of a grid derived image item with item ID 5. In the HEIF file of this embodiment, rows_minus_one of the grid derived image is specified as 2, and columns_minus_one is specified as 2. This indicates that a 2x2 input image is configured as a grid derived image. Note that the number of items (4) specified in ref_count as the number of input images for item reference shown in description 1650 and the number of grids (2x2(4)) are specified as matching values. In addition, 4032 is specified in output_width, and 3024 is specified in output_height, and an image with a width of 4032 pixels and a height of 3024 pixels is configured.

[0115] Description 1616 and description 1617 correspond to box 218, and specify a group of entities. In the HEIF file of this embodiment, two CombinedRegionEntityGroups are specified, and description 1616 and description 1617 correspond to boxes 231 and 232, respectively. In description 1616, 100 is specified for group_id, and the IDs of five region information items, item_id 50, 51, 52, 54, and 55, are specified as entity_id. As a result, the regions indicated by item_id 50, 51, 52, 54, and 55 are combined as a combination type cbrg, and all regions included in any of the five regions are identified as regions indicated by CombinedRegionEntityGroup. In description 1617, 101 is specified for group_id, and the IDs of two region information items, item_id 53 and 56, are specified as entity_id. As a result, the areas indicated by item_id 53 and 56 are combined as a combination type cbrg, and all areas contained in either of the two areas are identified as areas indicated by CombinedRegionEntityGroup.

[0116] Description 1618 corresponds to box 216 and includes description 1620 which corresponds to box 221 and description 1621 which corresponds to box 222 .

[0117] Description 1620 lists attribute information that may be used for each item or entity group as entry data. As shown in the figure, description 1620 includes the first and second entries indicating encoding parameters, the third, fourth, and fifth entries indicating the display pixel size of the item, and the sixth to fourteenth entries indicating annotation description information. In the annotation description information, lang is American English (en-US) in all entries, the sixth to twelfth entries are set to "human" as name, and the thirteenth and fourteenth entries are set to "family". The sixth to twelfth entries are set as description, respectively, to "Hanako", "Taro", "Shohei", "Mike", "Naomi", "Ginnji", and "Susan", which indicate the names of the people indicated by the region. In addition, the thirteenth and fourteenth entries are set as description to "Taro family" and "Mike family", which indicate the family. In tags, "AutoRecognition" is set to tag the region as an automatically recognized region. The attribute information listed in the description 1620 is associated with each item or entity group stored in the HEIF file in the entry data of the description 1621 corresponding to the box 222. In the example of FIG. 16, the image items with item_IDs 1 to 4 are associated with a common 'ispe' (property_index is 4), indicating that they are images with the same image size of 2016 pixels x 1512 pixels. Similarly, the image items with item_IDs 1 to 4 are associated with a common 'hvcC' (property_index is 1), indicating that they have the same encoding parameters. Meanwhile, the items or entity groups with item_IDs 5 to 15 are associated with different item properties. The derived image item with item_ID 5 is associated with 'ispe' (property_index is 3), indicating that it has an image size of 4032 pixels x 3024 pixels.

[0118] The image item with item_ID 6 is associated with 'ispe' (property_index is 5), which indicates that the image size is 768 pixels x 576 pixels. Similarly, the image item with item_ID 6 is associated with 'hvcC' (property_index is 2), which indicates the coding parameters. The region information item with item_ID 50 is associated with 'udes' (property_index is 6), which indicates annotation information for the region. The region information item with item_ID 56 is associated with 'udes' (property_index is 12), which indicates annotation information for the region. In this way, annotation information for the region is given by associating the 'udes' property. The region information entity group with item_ID (group_id) 100 is associated with 'udes' (property_index is 13), which indicates annotation information for the combination region. The region information entity group with item_ID (group_id) 101 is associated with 'udes' (property_index is 14), which indicates annotation information for the combination region. In this way, by associating the 'udes' property with an entity group, annotation information can be added to the combination area. Note that since no item properties are associated with the Exif data block item with item_ID 1000, no corresponding entry information is stored.

[0119] Next, the process of generating an HEIF file with a file structure that can store annotation information associated with each region by combining and identifying region information that identifies regions within an image will be described using the flowchart in Figure 17.

[0120] 17 is performed by CPU 101 performing various control processes using computer programs and data read from ROM 102 and non-volatile memory 110 to RAM 103. The generation process according to the flowchart in Fig. 17 is started, for example, when CPU 101 detects that a user has operated operation input unit 107 to input an instruction related to shooting. However, the event that serves as a trigger for starting the process according to the flowchart in Fig. 17 is not limited to a specific event.

[0121] In step S1701, the CPU 101 controls the imaging unit 104 and the image processing unit 105 to acquire an image (image data) to be stored in a HEIF file. Note that the method of acquiring the image is not limited to a specific method, and for example, the image may be acquired from the non-volatile memory 110 or from an external device via the communication unit 108.

[0122] In step S1702, under the control of CPU 101, recognition processing unit 114 performs a detection process for detecting an object captured as a subject in the image acquired in step S1701. This detection process is a process for detecting an object, such as a person or an object, from the image. Note that the method for detecting an object from an image is not limited to a specific method.

[0123] For example, a matching process may be performed using an image of a specific person registered in advance to check and detect whether or not the person is captured in the image acquired in step S1701.

[0124] Furthermore, depending on the contents registered in advance, not only specific people but also attribute information of people may be included, and such attribute information may also be used to perform detection processing for detecting objects from an image.

[0125] The recognition processing unit 114 may also detect associations based on a plurality of detection results, or perform detection processing of combined regions. For example, when detecting a human body, the recognition processing unit 114 may detect the head, torso, right hand, left hand, right foot, left foot, etc. individually, and then detect them as a combined human body. This results in different detection results depending on the recognition processing algorithm executed by the recognition processing unit 114. In addition to detection by the recognition processing unit 114, the user may detect the region of each object while checking the image using image editing software or the like.

[0126] In step S 1703 , under the control of CPU 101 , generation unit 113 generates area information for the area of ​​the object detected by recognition processing unit 114 , and stores the generated area information in an output buffer provided in RAM 103 .

[0127] In step S1704, under the control of the CPU 101, the encoding / decoding unit 111 performs encoding processing to encode the image acquired in step S1701, generates encoded data of the image, and stores the encoded data in the output buffer. Also, under the control of the CPU 101, the metadata processing unit 112 generates metadata to be stored in the image file, and stores the generated metadata in the output buffer.

[0128] In step S1705, under the control of the CPU 101, the metadata processing unit 112 generates metadata when storing the area information generated by the generation unit 113 for the area of ​​the object detected by the recognition processing unit 114. At this time, the metadata processing unit 112 determines whether it is necessary to group area items of the detection area (area where the object is detected) and store them as a combination area. This may involve data conversion processing such as optimization of the area information generated in step S1703. For example, if it is determined that storage as a combination area performing logical negation would be more efficient, the area information is modified and generated as area information indicating an inverted area.

[0129] If it is determined that it is necessary to group the area items of the detection area and store them as a combination area, the process proceeds to step S1706. On the other hand, if it is not determined that it is necessary to group the area items of the detection area (area where an object is detected) and store them as a combination area, the process proceeds to step S1709.

[0130] In step S1706, CPU 101 determines the grouping type for combining regions, for example, the grouping type of EntityToGroupBox described above.

[0131] In step S1707, CPU 101 generates group information for grouping the area information according to the grouping type determined in step S1706.

[0132] In step S1708, CPU 101 generates metadata for adding region annotation information (annotation) to the group information generated in step S1707, and associates it with the group information. As described above, this is realized by generating text description information using the udes property, and associating it with the entity group as an item property.

[0133] In step S1709, CPU 101 generates metadata for adding region annotation information to the regions detected as individual regions, and associates the metadata with the region information.

[0134] In step S1710, the CPU 101 generates a HEIF file that stores the generated metadata and encoded data. More specifically, the final metadata of the HEIF file is constructed based on the information stored in the output buffer by the metadata processing unit 112. Then, the information of the box 201 related to the HEIF file, the information of the box 203 storing the final metadata, and the information of the box 204 storing the image and region information are combined. The CPU 101 then writes the HEIF file generated by the combination from the RAM 103 to the non-volatile memory 110 and stores it. Note that the output destination of the HEIF file is not limited to the non-volatile memory 110, and may be transmitted to an external device via the communication unit 108, for example.

[0135] In this way, the image file storage device 100 according to this embodiment performs object detection processing on an image to be stored in an image file, and generates region information for identifying objects that are subjects in the image. The image file storage device 100 then generates metadata for identifying the generated region information as a combined region, making it possible to identify individual regions and combined regions. By adding annotation information to each identifiable region as metadata, it becomes possible to add region annotations to each region and combined region.

[0136] In the present embodiment, the image to be stored in the image file is obtained by shooting using the imaging unit 104, but the present invention is not limited to this. It goes without saying that the series of images may be images stored in advance in the ROM 102 or the non-volatile memory 110, or images received via the communication unit 108. In this case, the images may include a HEIF file in which one still image is stored, or may include a still image file such as JPEG. Alternatively, the images may be images encoded into a HEIF file in which multiple still images are stored, or may be unencoded RAW images.

[0137] Next, the manner in which a HEIF file is played back will be described. It will be easily understood that a HEIF file can be used in any device, not limited to the image file storage device 100 that generated the HEIF file. When used, a processor such as a CPU of the device reads the metadata of the HEIF file to be processed, making it possible to play back and change the encoded data stored in the HEIF file.

[0138] Hereinafter, as one aspect of image file playback processing, playback processing relating to playback (display) of HEIF files in the image file storage device 100 will be described with reference to the flowchart in FIG.

[0139] The processing according to the flowchart in Fig. 18 is performed by CPU 101 performing various control processes using computer programs and data read from ROM 102 or non-volatile memory 110 to RAM 103. The playback processing according to the flowchart in Fig. 18 is started, for example, when CPU 101 detects that a user has operated operation input unit 107 to input an instruction to play a HEIF file while image file storage device 100 is set to playback mode. However, the event that serves as a trigger for starting the processing according to the flowchart in Fig. 18 is not limited to a specific event.

[0140] In step S1801, CPU 101 acquires an image file (HEIF file) that is the target of a playback instruction. The method of acquiring the image file is not limited to a specific method, and the image file may be acquired from non-volatile memory 110 or from an external device via communication unit 108, for example.

[0141] In step S1802, under the control of the CPU 101, the metadata processing unit 112 obtains metadata and encoded data from the HEIF file obtained in step S1801, and analyzes the metadata to obtain the configuration of the HEIF file.

[0142] In step S1803, CPU 101 identifies a representative item based on the information in box 212 of the metadata acquired in step S1802, and causes encoding / decoding unit 111 to decode encoded data 241 indicated by the representative item. As a result, the encoding / decoding unit 111 acquires corresponding encoded data (encoded data 241) from the metadata related to the image item designated as the representative image, and stores the decoded image obtained by decoding the encoded data in a buffer on RAM 103.

[0143] In step S1804, under the control of the CPU 101, the metadata processing unit 112 acquires area metadata associated with the image to be reproduced that is designated as the representative image.

[0144] In step S1805, under the control of CPU 101, metadata processing unit 112 determines whether or not the area metadata acquired in step S1804 is associated as a group. "Whether or not the area metadata acquired in step S1804 is associated as a group" can be determined by analyzing box 218 and box 215.

[0145] If it is determined that the area metadata acquired in step S1804 is associated as a group (grouped area information associated with the reproduced image data is stored), the process proceeds to step S1806.

[0146] On the other hand, if the area metadata acquired in step S1804 is not associated as a group (area metadata related to the image to be reproduced is not grouped), the process proceeds to step S1807.

[0147] In step S 1806 , under the control of CPU 101 , generation unit 113 generates region information according to the grouping type of the grouped region information, and stores the generated region information in a buffer on RAM 103 .

[0148] In step S1807, the CPU 101 causes the display unit 106 to display the decoded image (representative image) stored in the buffer on the RAM 103, with the associated region information and the combined region information generated in step S1806 superimposed on the representative image.

[0149] In step S1808, if region annotation information corresponding to each region is added, CPU 101 displays the region annotation information (annotation) so that the relevance to the region information can be identified. Note that, in the present embodiment, a method of displaying the region annotation information and the region information superimposed on the representative image has been shown, but displaying the region information and the region annotation information may be optional. In other words, it may be determined whether to display them based on an instruction from a UI operation or the like.

[0150] 19, 20 and 21 show examples of region information and region annotation information displayed together with the representative image in steps S1807 and S1808. The images shown in Figures 19, 20 and 21 are examples of UI screens displayed in steps S1807 and S1808.

[0151] The shaded region indicated by 1901 indicates a region detected as the head region of a human body. Also, 1911 indicates region annotation information associated with the region 1901, in which the region shape is an ellipse and “Head” is associated as the region annotation information.

[0152] The dotted polygonal line indicated by 1902 indicates a region detected as the right arm region of the human body. Reference numeral 1912 indicates region annotation information associated with the region 1902, in which the region shape is Polyline and "Right arm" is associated as the region annotation information.

[0153] The dotted polygonal line indicated by 1903 indicates a region detected as the left arm region of the human body. Reference numeral 1913 indicates region annotation information associated with the region 1903, in which the region shape is Polyline and “Left arm” is associated as the region annotation information.

[0154] The lattice region indicated by 1904 indicates a region detected as the torso region of a human body. Also, 1914 indicates region annotation information associated with the region 1904, in which the region shape is Rectangular and Body is associated as the region annotation information.

[0155] The dotted polygonal line indicated by 1905 indicates an area detected as the right leg area of ​​the human body. Also, 1915 indicates area annotation information associated with the area 1905, in which the area shape is Polyline and Right leg is associated as the area annotation information.

[0156] The dotted polygonal line indicated by 1906 indicates an area detected as the left leg area of ​​the human body. Also, 1916 indicates area annotation information associated with the area 1906, in which the area shape is Polyline and “Left leg” is associated as the area annotation information.

[0157] These regions 1901 to 1906 are then combined as a combined region. In this case, the cbrg region entity group grouping makes it possible to identify the regions as one region, and the region annotation information associated with this combined region is indicated by 1921, with cbrgRegionGroup7 being a region combining Region1 to Region6 and Human being associated as region annotation information. In this way, it is possible to express the combined region without detecting the regions that make up each part individually and associating the combined region as a separate region. Furthermore, by making the combined region identifiable, it is possible to associate a region annotation with the combined region as well.

[0158] Next, in the UI screen shown in FIG. 20, the regions (individual regions) 2001 to 2006 indicating each part are the same as the regions 1901 to 1906 shown in FIG. 19. FIG. 20 shows an example in which region data 2007 is generated to separately identify the region indicating the human body indicated as the combined region, instead of the method of generating a combined region by combining each part. Region annotation information is associated with this human body region, and as shown in 2017, the region shape is Rectangular, and Human is associated as the region annotation information. In this way, the region of each part and the region indicating the whole can be identified as separate regions, and it is possible to associate the region annotation information individually, but the relationship between each part and the whole region cannot be shown. Therefore, the relationship between the regions can be identified as a combined region by InclusionRelationRegionEntityGroup indicating the inclusion relationship between the regions. As a result, the inclusion relationship is indicated as a combined region indicating that the regions 2001 to 2006 are included in the region 2007. Grouping by the irrg region entity group makes it possible to identify the combined region, and the region annotation information associated with this combined region is shown in 2021. This combined region is an area in which Region 7 includes Regions 1 to 6, and "Parts of Human" is associated with irrgRegionGroup10 as annotation region information. In this way, the regions that make up each part are detected individually, and the region that shows the whole is also detected separately, making it possible to identify these as a combined region that shows the relationship between the combined regions. It is also possible to associate region annotation information with the combined region.

[0159] 21 is a diagram showing another example of applying the cbrg entity group. The UI screen shown in FIG. 21 is an example of a screen display of the application example shown in FIG.

[0160] Area 2101 is an area indicated by the area item indicated by item_ID50 in Fig. 16. Reference numeral 2111 indicates area annotation information associated with area 2101, and indicates the description "Hanako" indicated by the udes item property, which is the sixth entry in the ipco box associated with item_ID50 in Fig. 16.

[0161] Area 2102 is an area indicated by an area item indicated by item_ID51 (not shown in Fig. 16). Reference numeral 2112 indicates area annotation information associated with area 2102, and indicates the description "Taro" indicated by the udes item property, which is the seventh entry in the ipco box in Fig. 16 associated with item_ID51.

[0162] Area 2103 is an area indicated by an area item indicated by item_ID52 (not shown in Fig. 16). Reference numeral 2113 indicates area annotation information associated with area 2103, and indicates the description "Shohei" indicated by the udes item property, which is the eighth entry in the ipco box in Fig. 16 associated with item_ID52.

[0163] Area 2104 is an area indicated by an area item indicated by item_ID53 (not shown in Fig. 16). Reference numeral 2114 indicates area annotation information associated with area 2104, and indicates the description "Mike" indicated by the udes item property, which is the ninth entry in the ipco box in Fig. 16 associated with item_ID53.

[0164] Area 2105 is an area indicated by an area item indicated by item_ID54 (not shown in Fig. 16). Reference numeral 2115 indicates area annotation information associated with area 2105, and indicates the description "Naomi" indicated by the udes item property, which is the tenth entry in the ipco box in Fig. 16 associated with item_ID54.

[0165] Area 2106 is an area indicated by an area item indicated by item_ID55 (not shown in Fig. 16). Reference numeral 2116 indicates area annotation information associated with area 2106, and indicates the description "Ginnji" indicated by the udes item property, which is the 11th entry in the ipco box in Fig. 16 associated with item_ID55.

[0166] Area 2107 is an area indicated by the area item indicated by item_ID56 in Fig. 16. Reference numeral 2117 indicates area annotation information associated with area 2107, and indicates the description "Susan" indicated by the udes item property, which is the 12th entry in the ipco box associated with item_ID56 in Fig. 16.

[0167] In addition, in FIG. 16, two cbrg entity groups are stored in the grpl box as a cbrg combination area that combines these areas, and 2122 indicates the area annotation information associated with the combination area indicated by group ID 100 of the cbrg entity group.

[0168] Region annotation information 2121 is a combination of regions 2101, 2102, 2103, 2105, and 2106, which are identified as a single region and associated with region annotation information. These regions are then shown as solid-line regions on the UI screen, and region annotation information is displayed in 2121. As shown by the 14th entry in the ipma box in Fig. 16, the 13th entry in the ipco box, the udes property, is associated with the cbrg entity group, which is an entity group with group ID 100. The description "Taro family" indicated by the udes property is displayed as region annotation information.

[0169] Similarly, area annotation information 2122 identifies areas 2104 and 2107 as a single area by combining them, and area annotation information is associated with them. These areas are shown as dotted line areas in the UI screen information, and area annotation information is displayed in 2122. As shown by the 15th entry in the ipma box in Fig. 16, the 14th entry in the ipco box, the udes property, is associated with the cbrg entity group, which is an entity group with group ID 101. The description "Mike family" indicated by the udes property is displayed as area annotation information.

[0170] According to the present embodiment described above, when one or more pieces of region information that identify an object that is related to a part of an image and is stored in an image file in association with the image, it is possible to efficiently specify a flexible region by storing combination type information that combines the region information and combination region information that identifies a combination region according to the combination type information. That is, in the past, when a complex shape was to be expressed as region information, it was necessary to indicate the region as bit mask data, which resulted in a large data size of the region information. However, according to the present embodiment, by combining multiple pieces of region information, a complex region can be specified as lightweight region information without relying on bit mask data. In addition, by making it possible to identify regions by combining them according to their types, it is possible to show the relationship between individually specified regions. This makes it possible to efficiently show information between regions that was previously difficult to show the relationship between, and the relationship between regions that could not be obtained without analyzing a large amount of region annotation information.

[0171] In this embodiment, the combination area information according to the combination type stored in the image file specifies the combination type using grouping type information and parameters with each area as one entity group. Then, by associating the grouped entity group with the image, the subject in the image can be identified by combining multiple areas. Furthermore, area annotation information can be associated with the combination area. However, the grouping type is only an example, and other combination types can be defined, and any combination type for combining and handling the areas can be used. In addition, when grouping the areas, it is possible to specify other grouped combination areas within the group. This allows for more flexible area specification. In addition, when treating individual areas as a group, the individual areas can belong to different groups. In addition, one subject object can be identified by belonging to multiple area groups, and further area annotation can be performed for each of them. Note that the four-character code for identifying the grouping type of the combination area shown in the embodiment is only an example, and any grouping type having an arbitrary four-character code reserved for each grouping type can be used.

[0172] [Second embodiment] In each of the following embodiments including this embodiment, the difference from the first embodiment will be described, and unless otherwise specified below, it is assumed that the embodiment is the same as the first embodiment. In the first embodiment, the method of grouping together with grouping type information as an entity group is shown as a method of identifying area information as a combination area, but the present invention is not limited to this method.

[0173] In this embodiment, a method of identifying a combination area will be described using the structure of a derived item that derives and handles information defined as an item. A derived item is a mechanism for performing a different expression by performing an operation on an item using metadata. For example, a derived image item realizes a different image expression by performing an operation on one or more input images, such as the grid image shown in the first embodiment. A grid image is a derived image that configures an image in which a plurality of input images that are sub-images are arranged on a grid to form one image. In addition, image overlays and identity derived images are specified. In ISO / IEC23008-12 2nd Edition FDIS, an identity derived area item is specified as an area item after applying a transformative item property in an area item. An identity derived area item is specified as a mechanism for performing a transformation to an area before associating with an image item. A transformation operation by a transformative property is specified as an operation after associating with a normal image item, so it is specified as a mechanism for first transforming an area item and then associating it with an image item. In this embodiment, a method for realizing the above by defining a derived area item type capable of expressing a combination area as a derived area item will be described.

[0174] The present embodiment differs from the first embodiment in that the structure of the generated HEIF file is as shown in FIG. 22. The block diagram in FIG. 1, the RegionItem data structure in FIG. 3, and the Udes property in FIG. 5 are common to the first embodiment. The flowchart in FIG. 17, the flowchart in FIG. 18, and the UIs in FIG. 19, 20, and 21 are common except that derived area items are used instead of entity groups. Hereinafter, the data structure of the HEIF file for identifying combination areas according to combination types using derived area items will be described with reference to FIG. 22.

[0175] As shown in Fig. 22, a HEIF file 2200 generally includes the following three boxes (storage areas). The first box 2201 (FileTypeBox('ftyp')) is common to box 201 described in the first embodiment. The HEIF file according to this embodiment does not include box 202 (ExtendedTypeBox('etyp')) required in the first embodiment, but may include it.

[0176] In the second box 2202 (MetaBox('meta')), multiple types of information relating to the image are stored in separate boxes, as described in detail below.

[0177] The third box 2203 (MediaDataBox('mdat')) is common to the box 204 described in the first embodiment.

[0178] Note that, in cases where a HEIF file stores video, audio, or an image sequence, a MovieBox ('moov') (not shown) may also be stored, which is the same as in the first embodiment.

[0179] A box 2211 (HandlerReferenceBox('hdlr')) stores a declaration of a handler type for analyzing the structure of the box 2202. The configuration of the box 2211 is common to the configuration of the box 211 described in the first embodiment.

[0180] Box 2212 (PrimaryItemBox('pitm')) specifies the identifier (item ID) of an image that is to be the representative item among the image items stored in the HEIF file 2200. The configuration of box 2212 is the same as the configuration shown in box 212 described in the first embodiment.

[0181] Box 2213 (ItemLocationBox('iloc')) stores information indicating the storage location of each information item, including an image item in the HEIF file 2200. Box 2213 typically describes the storage location of an image item as a byte offset from the beginning of the HEIF file 2200 and the data length from the beginning. That is, the information in box 2213 makes it possible to identify the positions of each piece of encoded data 2231-2232, area information 2233-2234, and Exif data block 2235 stored in box 2203. Furthermore, for a derived item, the information in box 2213 indicates that no data exists in box 2203. In this case, it indicates that there is no data structure, or that the data of the derived item is stored in box 2217 in box 2202.

[0182] Box 2214 (ItemInfoBox('iinf')) defines basic information (item information) such as item ID and item type indicating the item type for all items included in the HEIF file 2200. In the item information, not only image items such as encoded image items and derived image items, but also area items indicating areas within an image and Exif information items indicating Exif data blocks are specified. Also, information on derived area items indicating combined areas is defined. As derived area items indicating combined areas, combination type information is provided as the item type of the items specified as each derived area item. For example, the combination type RegionCombinedDerivation is a derived area item indicated by the item type 'cbrg'. By specifying 'cbrg' as the item type, it is specified that the area item applies the RegionCombinedDerivation operation. Also, information on the area items that are inputs constituting the combined area is provided by box 2215 shown below. This enables combinations according to the combination types of area items and area items, and combinations that indicate the association between area items that are inputs of derived area items. In addition, when there is one input domain item as a derived domain item, the derived domain item may use the item type 'iden'. In this case, the domain item can be derived using the Transformative item property defined separately. An identity derived domain item is a derived domain item with an item_type value of 'iden', and the derived domain item does not have an item body. Detailed definitions of these 'cbrg' derived domain items and derived domain item combination types for other derived domain items are described later.

[0183] Box 2215 (ItemReferenceBox('iref')) stores information describing the association between items included in the HEIF file 2200. In an aspect where an image item is a captured image, box 2215 is used to describe the association between the image item and its capture information (Exif data, etc.) item. In an aspect where multiple image items have a derived image relationship, box 2215 is used to describe the association between the image items. In an aspect indicating an area within an image item, box 2215 is used to describe the association between the image item and an area item. In an aspect where an area is specified by combining area items, box 2215 is used to describe the association between the area item and a derived area item.

[0184] The item reference type is specified for each item association, and the type of item reference can be identified. Also, the reference relationship between each item is described by describing the item ID specified in box 2214 in from_item_ID and to_item_ID. The item association in the derived domain item is indicated by the item reference type 'drgn', and the ID of the specified item is specified in box 2214 as the type of the derived domain item in from_item_ID. Also, the ID of one or more domain items that are input to the derived domain item are specified in to_item_ID. The reference space and domain defined by the derived domain item are obtained by applying an operation according to the type of the derived domain item to the reference space and domain of the domain item that is the input of the derived domain item. The details of the operation performed to obtain the reference space and domain are identified by the item type. Note that the reference_count of the 'drgn' item reference in the 'iden' derived domain item is equal to 1.

[0185] Box 2216 (ItemPropertiesBox('iprp')) stores various attribute information (item properties) for information items included in the HEIF file 2200. More specifically, box 2216 includes box 2221 (ItemPropertyContainerBox('ipco')) that describes the attribute information, and box 2222 (ItemPropertyAssociation('ipma')) that indicates the association between the attribute information and each item.

[0186] Box 2221 may store attribute information such as, for example, entry data indicating an HEVC parameter set required for decoding an HEVC image item, and entry data indicating the width and height of an image item in pixels. Also, attribute information capable of specifying any text information as an item property can be specified. UserDescriptionProperty('udes') shown in FIG. 5 is an example of attribute information capable of storing text information. By associating the udes property with an area item or a derived area item, it becomes possible to associate area annotation information as a text description with a specific area indicated by an area item or a derived area item associated with an image item.

[0187] It is also possible to specify TransformativeProperty, which is intended to transform and display an image when outputting it as attribute information that can be specified as an item property. This may store, for example, data indicating rotation information for rotating and displaying an image, or data indicating clipping information for cropping and displaying an image. This TransformativeProperty may be applied to an area item.

[0188] Box 2222 uses the ID (item ID) of the information item to store, for each item, entry data indicating an association with the attribute information stored in box 2221. Note that if there is no attribute information associated with an item, such as an Exif data block information item, entry data indicating the association is not stored.

[0189] Box 2217 (ItemDataBox('idat')) stores data related to items included in the HEIF file 2200. Box 2217 stores, for example, a data structure for describing a derived image item. For example, in the case of an item whose item type is 'grid' shown in box 2214, a data structure for a grid derived image item defined by reconstructing an input image in a predetermined grid order is specified. For the input image of the derived image item, a dimg reference type item reference is specified using box 2215. Similarly, for derived area items that specify an item data structure, a data structure for describing the derived area item is stored in box 2217. Note that if the derived item does not have a data structure, for example, the identity derived image item 'iden', no data structure is stored in box 2217. Similarly, for derived area items of a type that does not specify a data structure, no data structure is stored in box 2217. On the other hand, for future expansion, even if a derived area item does not have a data structure such as a specific parameter, a data structure indicating parameters indicating a version and flags may be stored. This has the advantage that it becomes easier to add parameters when updating a definition.

[0190] Next, we will explain the definition of setting an area for any image item stored in the HEIF file 2200 using area information 2233-2234 configured according to the definition of area information described in the first embodiment, and storing the area in association with an area annotation.

[0191] A region annotation consists of metadata associated with one or more regions of an image item or other image items. It also allows regions to be treated as derived region items that are combined according to the derived region item type by derivation-converting the input region items with derived region items according to the combination type. The derived region can be uniquely identified by the item ID of the derived region item.

[0192] In the image file storage device 100 of this embodiment, by adding area information 2233-2234 and information related to derived area items to the boxes following box 2202, it is possible to associate area annotations with areas and combination areas in specific image items.

[0193] First, in the HEIF file 2200 of this embodiment, the encoded data 2231-2232 and the area information 2233-2234 are stored in the box 2203, and in order to distinguish between them, an item type is set for each data in the box 2214. For example, the item type is set to 'hvc1' in the case of the encoded data 2231, and 'rgan' in the case of the area information 2233. The storage location in the HEIF file 2200 of the area information 2233-2234 stored in the box 2203 is stored in the box 2213, similar to the encoded data 2231-2232. As described above, the area information 2233-2234 is information for relatively specifying the area in the reference space, so it is necessary to specify the image item for which the area is set. Information regarding which image item the area is set for by one piece of area information 2233 is stored in the box 2215. That is, in the box 2215, in order to describe the association between the items set in the box 2214, for example, for the area item related to the area information 2233, information identifying the image item related to the coded data 2231 that sets the area using the area information 2233 is stored in the box. Also, the area item can set the area in the derived image by associating it with the derived image item configured as the derived image. Also, when the area item is treated as an area by performing a conversion operation as a derived area item, the derived area item type that is the conversion operation of the derived area item is determined, and the item_type to be set in the box 2214 is determined. In order to treat the areas specified in each area item as an area obtained by logically adding them together, cbrg is specified as the item_type. Also, the ID of the area item that is the input of the derived area item is described using the box 2215 to describe the association between items. Specifically, the item ID of the item specified as the derived area item, which is indicated by the item reference type 'drgn', is specified as from_item_ID. Furthermore, the item ID of the area item that is the input is specified as to_item_ID. When multiple area items are to be input, they can be specified by listing them in to_item_ID.In this way, information for constructing a derived area item is specified. This makes it possible to specify a logically added area as one derived area item. Furthermore, in order to associate an image item with a derived area item, information on which image item the derived area item sets an area for is stored in box 2215.

[0194] However, for a region thus established for a particular image item, region annotations for that region may be established by associating any or all of the following with the region item or derived region item:

[0195] The first is a region annotation using an item property. The udes property stored in box 2221 in box 2216 is associated with a region item or derived region item by box 2222. By associating the udes property with a region item or derived region item, it becomes possible to associate region annotation information as a text description.

[0196] The second is a region annotation using a metadata item, in which a metadata item such as Exif that stores metadata in mdat is associated with a region item or a derived region item by a reference type cdsc in box 2215.

[0197] The third is region annotation using other image items or entity groups. Region items or derived region items are associated with other image items or entity groups by reference type eroi. This allows for providing different image representations for specific regions. Such associations allow for region annotations to be set for specific regions within an image.

[0198] Next, a definition of a method for specifying a combination area of ​​area items that can be stored in the HEIF file 200 will be described. Area derivation of RegionCombineDerivation can be defined as a derived area item that can be stored in the HEIF file 200. The item body of RegionCombineDerivation is empty, and structure data is not defined. On the other hand, as described above, for future expansion, even a derived area item that does not have a data structure such as a specific parameter may store a data structure indicating parameters indicating a version and a flag. In this case, the storage location of the data structure can be indicated by information indicating the storage location in the file in the HEIF file 2200 (byte offset, data length, etc.) in the description of the derived area item in box 2213 (ItemLocationBox('iloc')). Also, as the storage location, a data structure held by the derived area item is usually described in box 2217 (ItemDataBox('idat')). A data structure having only a version parameter and a flag parameter as shown in FIG. 25 is stored in this box as the data structure of the derived area item. This has the advantage that it is easy to add parameters or switch by version when updating the definition as shown below. On the other hand, when only the initial version is used, blank data is stored, resulting in 16 bits of wasted data.

[0199] The type of such a derived region item, RegionCombineDerivation, can be defined as follows. It is easy to understand that the name of this derived region item and the 4CC of the item type that identifies this derived region item may be an alias or a different 4CC. For example, an alias to indicate the combination may be UnionRegionDerivation, and it may be identifiable as 'unrg', etc.

[0200] An item with item_type value 'cbrg' defines a derived area item to create a combined area that combines one or more (or two or more) input areas by disjunction. The input areas are listed in a SingleItemTypeReferenceBox of type 'drgn' for this derived area item in its ItemReferenceBox. The input areas may be specified as area items or derived area items. In the SingleItemTypeReferenceBox of type 'drgn', the value of from_item_ID identifies the derived area item of type 'cbrg' and the value of to_item_ID identifies the input area items. The combined (or union or union) derived area item is associated with the image items whose area is defined using an item reference of type 'cdsc' from the combined (or union or union) derived area item to the image item.

[0201] The region generated from this derived region item is the union of all the regions of each input region item after they are applied to the referenced image item. Specifying a derived region item as an input region allows for more flexible region specification. Forming a region by logically ORing these input regions allows all regions represented by any of the regions represented by the input regions to be identified as the region represented by the RegionCombineDerivation derived region item.

[0202] Within the same file, multiple RegionCombineDerivation derived region items can be specified with different item_id values.

[0203] It is recommended that one region item or derived region item specified in the input region indicates one region (for example, region_count in the region item is 1). In this case, it is recommended that the number of input regions specified (ref_count in box 2215) is 2 or more. On the other hand, if one input region indicates two or more regions (for example, region_count in the region item is 2 or more), it may be combined with other input regions by logical OR of all regions. In this case, the number of input regions specified may be 1. As another method, if the number of regions indicated by the region items or derived region items specified in the input region of the derived region item is the same (region_count of all input regions is the same), each region may be combined in the input order of the region. The method of combining various regions according to the number of regions may be controlled by the value of flags included in the definition of the item information entry in box 2214 separately. Alternatively, flags stored in the data structure shown in FIG. 25 may be used. The region indicated by the RegionCombineDerivation derived region item and each region indicated by the input region item can be associated with an image item individually, and each region in the image can be identified. Also, different region annotation information can be associated with each separately, depending on the method defined by the region annotation.

[0204] A region derivation of RegionMultiplyDerivation can be defined as a derived region item that can be stored in the HEIF file 200. The item body of RegionMultiplyDerivation is empty, and no structure data is defined. The type of such a derived region item, RegionMultiplyDerivation, can be defined as follows. It should be readily understood that the name of this derived region item and the 4CC of the item type that identifies this derived region item may be an alias or a different 4CC. For example, an alias for indicating the combination may be RegionIntersectionDerivation, and it may be identifiable by 'isrg' or the like.

[0205] An item with item_type value 'mprg' defines a derived area item to create a combined area that combines one or more input areas by logical intersection. The input areas are listed in a SingleItemTypeReferenceBox of type 'drgn' for this derived area item in its ItemReferenceBox. In the SingleItemTypeReferenceBox of type 'drgn', the value of from_item_ID identifies the derived area item of type 'mprg' and the value of to_item_ID identifies the input area items. An intersection (or intersection or intersection) derived area item is associated with the image item whose area is defined using an item reference of type 'cdsc' from the intersection (or intersection or intersection) derived area item to the image item.

[0206] The region generated from this derived region item is the intersection of all the regions of each input region item after they are applied to the referenced image item. The input region may be specified as a region item or a derived region item. Specifying a derived region item as the input region allows for more flexible region specification. By forming a region by logically multiplying these input regions, the common region contained in all of the regions represented by the input regions is identified as the region represented by the RegionMultiplyDerivation derived region item.

[0207] Within the same file, multiple RegionMultiplyDerivation derived region items with different item_id values ​​can be specified.

[0208] It is recommended that one region item or derived region item specified in the input region indicates one region (for example, region_count in the region item is 1). In this case, it is recommended that the number of input regions specified (ref_count in box 2215) is 2 or more. On the other hand, if one input region indicates two or more regions (for example, region_count in the region item is 2 or more), it may be combined with other input regions by logical ANDing all the regions. In this case, the number of input regions specified may be 1. As another method, if the number of regions indicated by the region items or derived region items specified in the input region of the derived region item is the same (region_count for all input regions is the same), each region may be combined in the input order of the regions. The method of combining various regions according to the number of regions may be separately controlled by the value of flags included in the definition of the item information entry in box 2214. The region indicated by the RegionMultiplyDerivation derived region item and each region indicated by the input region item can be associated with an image item individually, and each region in the image can be identified. In addition, different region annotation information can be associated with each individually by the method defined in the region annotation.

[0209] A region derivation of RegionInverseDerivation can be defined as a derived region item that can be stored in the HEIF file 200. The item body of RegionInverseDerivation is empty, and no structural data is defined. The type of such a derived region item, RegionInverseDerivation, can be defined as follows. It should be readily understood that the name of this derived region item and the 4CC of the item type that identifies this derived region item may be an alias or a different 4CC. For example, an alias for indicating the combination may be RegionNegationDerivation, and it may be identifiable by 'ngrg', etc.

[0210] An item with item_type value 'ivrg' defines a derived area item for creating a combined area that combines one or more input areas by logical negation or NOR. The input areas are listed in a SingleItemTypeReferenceBox of type 'drgn' for this derived area item in its ItemReferenceBox. In the SingleItemTypeReferenceBox of type 'drgn', the value of from_item_ID identifies the derived area item of type 'ivrg' and the value of to_item_ID identifies the input area items. An inverse (or inverse, or negation, or negation) derived area item is associated with the image item whose area is defined using an item reference of type 'cdsc' from the inverse (or inverse, or negation, or negation) derived area item to the image item.

[0211] The area generated from this derived area item is the inverse of all the areas of each input area item after they are applied to the referenced image item. The input area may be specified as an area item or a derived area item. Specifying a derived area item as the input area allows for more flexible area specification.

[0212] By forming a region by logically negating or negative-oring these input regions, a region that is not included in any of the regions indicated by the input regions is identified as a region indicated by the RegionInverseDerivation derived region item.

[0213] Within the same file, multiple RegionInverseDerivation derived region items can be specified with different item_id values.

[0214] It is recommended that one region item or derived region item specified in an input region indicates one region (for example, region_count in a region item is 1). In this case, if the number of input regions specified (ref_count in box 2215) is one, the region obtained by logically negating that region is specified as the combination region. In other words, the region other than the region indicated by the input region becomes the target region. On the other hand, if there are two or more input regions, the region obtained by logically negating the regions indicated by the input regions is specified as the combination region. In other words, the region indicated by all input regions and not specified by any of the regions becomes the target region.

[0215] On the other hand, if the number of regions indicated by one input region is two or more (for example, the region_count in the region item is two or more), they may be combined with the regions indicated by other input regions by a logical NOR of all the regions. As another method, if the number of regions indicated by the region items or derived region items specified in the input region of the derived region item is the same (the region_count of all input regions is the same), the regions may be combined in the order of input of the regions. The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of the item information entry in box 2214. The region indicated by the RegionInverseDerivation derived region item and each region indicated by the input region item can be associated with an image item individually, and each region in the image can be identified. In addition, different region annotation information can be associated with each region individually by the method defined in the region annotation.

[0216] When there is one region item that serves as the input region, logical negation or negated logic may be specified by defining the InverseRegionProperty shown in FIG. 14 and associating it with the region item. In this case, logical negation or negative OR may be specified using the iden region derivation. InverseRegionProperty('ivrg') shown in FIG. 14 is an example of TransformativeProperty attribute information that can specify logical negation or negated logic of a region. The InverseRegionProperty shown in FIG. 14 includes the four-character code 'ivrg' shown in definition 1401, and this four-character code is used to identify the InverseRegionProperty. The ivrg property is a property that is composed of only the four-character code, version information, and flags, and does not include any other parameters.

[0217] Fig. 23 shows the data structure of RegionLogicalDerivation as a derived region item that can be stored in the HEIF file 200, and region derivation by logical operations can be defined. The data structure of RegionLogicalDerivation is the data structure of a derived region item stored in box 2217, and as shown in Fig. 23, includes version shown in definition 2301, flags shown in definition 2302, and logical_operation shown in definition 2303. Such a derived region item type RegionLogicalDerivation can be defined as follows. It should be easily understood that the name of this derived region item and the 4CC of the item type identifying this derived region item may be an alias or a different 4CC.

[0218] An item with item_type value 'lorg' defines a derived area item to create a combined area that combines one or more input areas through a logical operation. The input areas are listed in a SingleItemTypeReferenceBox of type 'drgn' for this derived area item in its ItemReferenceBox. In the SingleItemTypeReferenceBox of type 'drgn', the value of from_item_ID identifies the derived area item of type 'lorg' and the value of to_item_ID identifies the input area items. A logical derived area item is associated with the image item whose area is defined using an item reference of type 'cdsc' from the logical derived area item to the image item.

[0219] The area generated from this derived area item is an area that applies the logical operation specified for each input area item after it is applied to the referenced image item. An area item or a derived area item is specified as the input area. Specifying a derived area item as the input area allows for more flexible area specification. An area is formed by logical operation of these input areas. The operator for the logical operation is provided by the logical_operation parameter of definition 2303. In other words, the area that results from performing a logical operation on each area indicated by the input area using the calculation method indicated by the logical_operation parameter is identified as the area indicated by the RegionLogicalDerivation derived area item.

[0220] Within the same file, multiple RegionCombineDerivation derived region items can be specified with different item_id values.

[0221] logical_operation indicates the type of logical operation and is an unsigned integer indicating the logical operator to apply to the input fields as the derived field transformation. A value of 0 indicates logical combination of fields by logical OR operation. A value of 1 indicates logical combination of fields by logical AND operation. A value of 2 indicates logical combination of fields by logical NOT or NOR. A value of 3 indicates logical combination of fields by XOR. A value of 4 indicates logical combination of fields by NAND. A value of 5 indicates logical combination of fields by NEXOR. Other values ​​are reserved.

[0222] The number of regions indicated by one region item or derived region item specified in the input region can be defined in the same manner as above. In addition, the method of combining various regions according to the number of regions may be controlled separately by the value of definition 2302 (flags) included in the definition of RegionLogicalDerivation. The behavior can be determined by the combination of the value of logical_operation and the value of flags.

[0223] Each region represented by the RegionLogicalDerivation and each region represented by the input region item can be associated with an image item separately, each identifying a region within the image, and each can have different region annotation information associated with it, depending on how it is defined by the region annotation.

[0224] In this embodiment, the method of providing the operator of the logical operation by the parameter logical_operation included in the data structure of RegionLogicalDerivation has been shown, but it may also be controlled using flags. When there is one region item that is the input region, it may be associated with the region item using the LogicalOperateRegionProperty shown in FIG. 15 to specify a combination region by logical operation. The lorg property is a TransformativeProperty, and the associated region item applies the logical operation specified in logical_operation. The region item may be a derived region item. In this case, it may be configured to specify the logical operation using iden region derivation.

[0225] Similarly, item types may be defined as derived operation types for individual derived area items to specify combinations of area information using other logical operations. For example, exclusive OR, NOT AND, and NOT exclusive OR can be specified. Furthermore, when there is one area item (ref_count in box 2215) that is the input area, and one area item indicates multiple areas (for example, region_count in the area item is 2 or more), item properties according to these logical operations may be defined to specify combined areas. In this case, it can also be defined by performing iden area derivation.

[0226] So far, we have explained the derived area item type that logically combines areas specified as derived area items. Next, we will explain the area derivation operation type that indicates the relationship between each input area included in a derived area item.

[0227] A region derivation of RegionInclusionRelationDerivation can be defined as a derived region item that can be stored in the HEIF file 200. The item body of RegionInclusionRelationDerivation is empty, and no structural data is defined. The type of such a derived region item, RegionInclusionRelationDerivation, can be defined as follows. It should be readily understood that the name of this derived region item and the 4CC of the item type that identifies this derived region item may be an alias or a different 4CC. For example, an alias for indicating the combination may be CompoundRegionDerivation, and it may be identifiable by 'corg' or the like.

[0228] An item with item_type value 'irrg' defines a derived area item that specifies an inter-area relationship indicating containment of one or more input areas. The input areas are listed in a SingleItemTypeReferenceBox of type 'drgn' for this derived area item in its ItemReferenceBox. In the SingleItemTypeReferenceBox of type 'drgn', the value of from_item_ID identifies a derived area item of type 'irrg' and the value of to_item_ID identifies the input area items. A containment (or compound, or intermix) derived area item is associated with the image item whose area is defined using an item reference of type 'cdsc' from the containment (or compound, or intermix) derived area item to the image item.

[0229] The area generated from the derived area item indicates the relationship between the areas that the first input area item of each input area item after being applied to the referenced image item includes the second and subsequent input area items. An area item or a derived area item is specified for the input area. By specifying a derived area item as the input area, more flexible area specification is possible. The area indicated by the first input area is identified as an area including the second and subsequent input areas. In this embodiment, the definition of RegionInclusionRelationDerivation is defined as a derived area indicating an inclusion relationship in which the first input area includes the second and subsequent input areas. However, a configuration that enables flexible specification may be made using the data structure of the derived area item. Specifically, a parameter may be defined so as to identify which input area is included in which input area, or whether multiple input areas are included in other input areas. For example, ref_count is used to specify a parent-child relationship indicating an inclusion relationship by a parameter for each input area.

[0230] Multiple RegionInclusionRelationDerivation derived region items with different item_id values ​​can be specified in the same file.

[0231] It is recommended that one region item or derived region item specified in an input region indicates one region (for example, region_count in the region item is 1). In this case, it is recommended that the number of input regions specified (ref_count in box 2215) is 2 or more. On the other hand, if one input region indicates two or more regions (for example, region_count in the region item is 2 or more) and it is the first input region, the region information may be defined as a relationship in which the first specified region includes the subsequent regions and all the regions indicated by the second and subsequent input regions. In this case, the number of input regions specified may be 1. Alternatively, the region indicated by the first input region may be treated as one region, and it may be defined so that the second and subsequent input regions are included. In this case, it is recommended that the number of input regions specified (ref_count in box 2215) is 2 or more. As another method, if the number of regions indicated by the region items or derived region items specified in the input region of a derived region item are the same (region_count for all input regions is the same), the relationship may be shown as each region being included in the first region in the input order of the regions. The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of the item information entry in the box 2214. The region indicated by the RegionInclusionRelationDerivation derived region item and each region indicated by the input region item can be individually associated with an image item. Also, different region annotation information can be individually associated with each region by the method defined in the region annotation. In this embodiment, a derived region item for indicating the relationship between the regions is defined to indicate the relationship between the region items that are the input regions, but it is also possible to describe the relationship by simply using an item reference between the region items without defining it as a derived region item. In this case, for example, it is described as an item reference from the region item specified first in the derived region item to another region item.

[0232] A region derivation of RegionTouchRelationDerivation can be defined as a derived region item that can be stored in the HEIF file 200. The item body of RegionTouchRelationDerivation is empty, and no structure data is defined. The type of such a derived region item, RegionTouchRelationDerivation, can be defined as follows: Note that it will be readily understood that the name of this derived region item and the 4CC of the item type that identifies this derived region item may be an alias or a different 4CC.

[0233] An item with item_type value 'tcrg' defines a derived area item that specifies an association between areas indicating that one or more input areas have a touching relationship. There are two input areas and they are listed in a SingleItemTypeReferenceBox of type 'drgn' for this derived area item in its ItemReferenceBox. In the SingleItemTypeReferenceBox of type 'drgn', the value of from_item_ID identifies the derived area item of type 'tcrg' and the value of to_item_ID identifies the input area item. A touching (or touch, or common boundary) derived area item is associated with the image item whose area is defined using an item reference of type 'cdsc' from the touching (or touch, or common boundary) derived area item to the image item.

[0234] The areas generated from this derived area item indicate that the areas specified in each input area item are in a touching relationship after being applied to the referenced image item. The input area may be an area item or a derived area item. Specifying a derived area item as an input area allows for more flexible area specification. Two areas in an image related by this derived area item identify that their subject objects are in a touching relationship. This does not simply indicate that the areas are touching or overlapping, but rather that the subject objects in the image are in a touching relationship.

[0235] Within the same file, multiple RegionTouchRelationDerivation derived region items with different item_id values ​​can be specified.

[0236] It is recommended that one region item or derived region item specified in an input region indicates one region (for example, region_count in a region item is 1). On the other hand, if one input region indicates two regions (for example, region_count in a region item is 2), the number of input regions specified (ref_count in box 2215) may be one. In this case, it indicates that the two regions included in one input region are in a contacting relationship. In addition, if one input region indicates two or more regions (for example, region_count in a region item is 2 or more) or the number of input regions is two or more, it may be defined that all regions in the first input region and all regions indicated by the second input region are in a contacting relationship. Furthermore, if the number of regions indicated by region items or derived region items specified in input regions is the same (region_count for all input regions is the same), it may be defined so that the order in which the regions are indicated indicates that the region specified by the first input region and the region specified by the second or subsequent input regions are in a contacting relationship. The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of the item information entry in the box 2214. The region indicated by the RegionTouchRelationDerivation derived region item and each region indicated by the input region item can be individually associated with an image item. Also, different region annotation information can be individually associated with each region by the method defined in the region annotation. In this embodiment, the derived region item type indicating a contact relationship has been described, but it may be a type indicating non-contact. Also, it may be possible to define both a contact relationship and a non-contact relationship by defining flags or additional parameters in the RegionTouchRelationDerivation derived region item data structure. In this embodiment, a derived region item for indicating the relationship between the regions is defined to indicate the relationship between the region items that are the input regions, but it may be described simply by using item references between the region items without defining it as a derived region item.In this case, for example, the derived area item may be described as an item reference from the item designated as the input area item to the other input area item, or item references may be described mutually.

[0237] A region derivation of RegionDepthOrderRelationDerivation can be defined as a derived region item that can be stored in the HEIF file 200. The data structure of RegionDepthOrderRelationDerivation is not shown, but includes parameters similar to those of the entity group shown in FIG. 12. The type of such a derived region item, RegionTouchRelationDerivation, can be defined as follows: It should be readily understood that the name of this derived region item and the 4CC of the item type identifying this derived region item may be an alias or a different 4CC.

[0238] An item with item_type value 'dorg' defines a derived region item that specifies a foreground-background contextual relationship between one or more input regions. The input regions of a RegionDepthOrderRelationDerivation are listed in a SingleItemTypeReferenceBox of type 'drgn' for this derived region item within an ItemReferenceBox in the order indicated by depth_direction. In the SingleItemTypeReferenceBox of type 'drgn', the value of from_item_ID identifies the derived region item of type 'dorg' and the value of to_item_ID identifies the input region items. A context (or depth, or foreground-background) derived region item is associated with the image item whose region is defined using an item reference of type 'cdsc' from the context (or depth, or foreground-background) derived region item to the image item.

[0239] The regions generated from this derived region item indicate the relationship between the regions, that is, the region before and after the regions specified in each input region item after being applied to the referenced image item. A region item or a derived region item is specified as the input region. Specifying a derived region item as the input region allows for more flexible region specification.

[0240] depth_direction indicates the depth (foreground-background) direction and is an unsigned integer indicating the depth orientation of the input regions of the derived region. A value of 0 indicates that the listed input regions are listed from foreground to background. A value of 1 indicates that the listed input regions are listed from background to foreground. All other values ​​are reserved.

[0241] Within the same file, multiple RegionDepthOrderRelationDerivation derived region items with different item_id values ​​can be specified.

[0242] It is recommended that one region item or derived region item specified in an input region indicates one region (for example, region_count in a region item is 1). It is also recommended that the number of region items or derived region items that become input regions is 2 or more. On the other hand, if one input region indicates two or more regions (for example, region_count in a region item is 2 or more), the number of input regions specified (ref_count of an iref box) may be 1. In this case, it indicates that the multiple regions included in one input region are specified in the direction indicated by depth_direction in the order of their storage. Alternatively, multiple regions included in one input region may be defined as being at the same position, that is, having the same depth. When one input region includes multiple regions, the number of input regions may be 1. Alternatively, if the number of regions indicated by the input regions is the same, they may be defined as being lined up in the direction indicated by depth_direction in the order specified by each region in the order of region counts. The method of combining various regions according to the number of regions may be controlled separately by defining flags or additional parameters in the RegionDepthOrderRelationDerivation derived region item data structure. In addition, an additional parameter may be defined and specified to indicate whether the depth of each individual region is the same or the next depth. Note that in this embodiment, a derived region item is defined to indicate the relationship between the regions, and the relationship between the region items that are the input regions is indicated, but it is also possible to describe the relationship simply using item references between region items without defining it as a derived region item. In this case, for example, it is described as an item reference from the region item specified first in the derived region item to the other region items. In this case, it is possible to identify the order of the depths by the order in which the item references are specified.

[0243] A region derivation of RegionForbidRelationDerivation can be defined as a derived region item that can be stored in the HEIF file 200. The item body of RegionForbidRelationDerivation is empty, and no structure data is defined. The type of such a derived region item, RegionForbidRelationDerivation, can be defined as follows. It should be readily understood that the name of this derived region item and the 4CC of the item type that identifies this derived region item may be an alias or a different 4CC. For example, an alias for indicating the combination may be OclusionDerivation, and it may be identifiable by 'ocrg' or the like.

[0244] An item with item_type value 'fbrg' defines a derived area item that specifies an association between areas indicating that the second input area is in a partial or full occlusion relationship to the first input area. There are two input areas and they are listed in a SingleItemTypeReferenceBox of type 'drgn' for this derived area item in an ItemReferenceBox. In the SingleItemTypeReferenceBox of type 'drgn', the value of from_item_ID identifies the derived area item of type 'fbrg' and the value of to_item_ID identifies the input area item. An occlusion relationship (or occlusion, or obstruction) derived area item is associated with the image item whose area is defined using an item reference of type 'cdsc' from the occlusion relationship (or occlusion, or obstruction) derived area item to the image item.

[0245] The area generated from this derived area item indicates the relationship between the areas specified in each input area item after being applied to the referenced image item, that is, the areas are in an obscuring relationship. An area item or a derived area item is specified as the input area. By specifying a derived area item as the input area, more flexible area specification is possible. Two areas in an image related by this derived area item are identified as being in a relationship in which one area hides an object indicated by the other area. For example, if all objects indicated by one area are hidden by objects indicated by the other area, the hidden object does not appear as a subject in the image. In other words, a hidden area is set based on information that can be inferred from another image, etc., and it is indicated that it is hidden by an object indicated by another area.

[0246] Multiple RegionForbidRelationDerivation derived region items with different item_id values ​​can be specified in the same file.

[0247] It is recommended that one region item or derived region item specified in an input region indicates one region (for example, region_count in the region item is 1). On the other hand, if one input region indicates two regions (for example, region_count in the region item is 2), the number of input regions specified (ref_count in box 2215) may be 1. In this case, it indicates that the object indicated by the first region of the two regions included in one input region is hiding the object indicated by the second region. In addition, if one input region indicates two or more regions (for example, region_count in the region item is 2 or more) or the number of input regions is two or more, it may indicate that the object indicated by the region indicated by the first input region is hiding the object indicated by the region specified by the second or subsequent input regions.

[0248] The method of combining various regions according to the number of regions may be controlled separately by the value of flags included in the definition of the item information entry in the box 2214. The region indicated by the RegionForbidRelationDerivation derived region item and each region indicated by the input region item can be individually associated with an image item. Also, different region annotation information can be individually associated with each region by the method defined in the region annotation. In this embodiment, a derived region item for indicating the relationship between the regions is defined to indicate the relationship between the region items that are the input regions, but it is also possible to describe the relationship by simply using an item reference between the region items without defining it as a derived region item. In this case, for example, it is described as an item reference from the region item specified first in the derived region item to another region item.

[0249] In the above-described definition of the derived area item according to each combination type, a method of defining the number of areas held by the area item that is the input area by flags has been shown. As another method of describing the combined area, even if one area item or derived area item specified in the input area indicates multiple areas, a method of realizing the above by defining the data structure of different derived area items without relying on the control by the value of flags will be described with reference to FIG. 26. In the example shown in FIG. 26, a combined area by logical sum is described as an example, but other combination types can also be defined in a similar manner. The definition of CombinedRegionItem shown in FIG. 26 shows a data structure that can be stored in the idat box as the data structure of the derived area item.

[0250] The data structure shown in FIG. 26 is an example of an extension of a combined area item. The extended combined derived area item has a configuration that allows multiple combined areas to be described in one combined derived area item. In the case of an area item, when multiple areas are described in one area item, it was possible to describe them by specifying a value of 2 or more for region_count. Similar to this method, this definition allows multiple areas to be described in the combined area. For each of the number of areas (region_count) defined in the extended combined derived area item, the number of input areas (input_count) that make up the combined area is given. Each input area item is specified by the index (region_item_index) of the area item that defines the input order to the combined area, and its index (region_index) in the list of areas defined by this input area item.

[0251] Here, version is defined as 0 or 1. Future extensions can be specified by specifying a new version value for this version. Here, flags is 0 and all flag values ​​are reserved. Here, region_count is the number of regions defined by the combined region item. Here, input_count is the number of regions that are input regions for each combined region defined in the combined region item. Here, region_item_index is the index of an input region item in the list of input region items specified in the SingleItemTypeReferenceBox of type 'drgn' of this derived region item, and indicates one of the input regions that make up one of the combined regions defined by this derived region item. Here, region_index is the index of the region in the input region item notified by the region_item_index field that is an input region that makes up one of the combined regions.

[0252] By defining the data structure of the derived area item in this way, it becomes easy to configure the derived area item regardless of how many areas the input area item is composed of. Also, when multiple combined areas need to be given the same annotation, it is possible to give a common annotation in the same way as with existing area items, without having to give annotations to each derived area item.

[0253] This data structure can be adapted to other combination types, and similar definitions can be made for derived region items that have parameters such as RegionLogicalDerivation by adding parameters to this data structure.

[0254] Here, an example of an image file output by the image file storage device 100 according to this embodiment will be described with reference to Fig. 24. Note that in this embodiment, the image file is configured to store in the image file a region in which region items are combined using two RegionCombineDerivation derived region items as a file data structure.

[0255] In the example of FIG. 24, as shown in description 2403 corresponding to box 2203, descriptions 2430 and 2431 corresponding to HEVC encoded data (HEVC Image Data) 2231 to 2232 are stored. Furthermore, an image file is illustrated that stores descriptions 2432 and 2433 corresponding to region item data 2233 to 2234 and description 2434 corresponding to Exif data block 2235. As shown in descriptions 2432 and 2433, region information 2233 to 2234 conforms to the definition shown in FIG. 3, and each specifies a rectangular region for a reference space with an image size of 4032 pixels x 3024 pixels. In addition, the region specified by the description specifies the coordinates (x0, y0) of the reference point in the reference space of the region, the image size (w0xh0), the coordinates (x7, y7), and the image size (w7xh7).

[0256] Description 2401 corresponds to box 2201, and stores 'mif1' as the type value major-brand of the brand definition that conforms to the HEIF file, and 'heic' as the type value compatible-brands of the compatible brand definition.

[0257] Description 2402 corresponding to box 2202 shows various information of metadata describing the untimed data stored in the example output file. Description 2410 corresponds to box 2211, and the handler type of the MetaDataBox(meta) specified is 'pict'. Description 2411 corresponds to box 2212, and stores 5 as the item_ID, specifying the ID of the image to be displayed as the first priority image.

[0258] Description 2412 corresponds to box 2214, and indicates item information (item ID (item_ID) and item type (item_type)) for each item. Each item can be identified by item_ID, and indicates what type of item the item identified by item_ID is. In the example of FIG. 24, since 16 items are stored, entry_count is 16, and 16 types of information are arranged in description 2412, and each item ID and item type are specified. In the illustrated image file, the first to fourth information corresponding to descriptions 2440 to 2441 is an HEVC encoded image item of type hvc1. Also, the fifth information shown in description 2442 specifies a grid derived image item of item type grid. Also, the sixth information shown in description 2443 is an HEVC encoded image item of item type hvc1 that becomes a thumbnail image. The seventh to thirteenth information corresponding to descriptions 2444 to 2445 is an area item of item type rgan indicating an area. The 14th and 15th pieces of information corresponding to the descriptions 2446 and 2447 are derived region items of item type cbrg indicating the region derivation of RegionCombineDerivation. The cbrg region derivation items specified in the descriptions 2446 and 2447 respectively make the combined region identifiable using item ID 57 and item ID 58.

[0259] The 16th information corresponding to the description 2448 specifies a metadata item of the item type Exif indicating an Exif data block.

[0260] Description 2413 corresponds to box 2213, and specifies the storage location of each item in the HEIF file and information on the data size. For example, an encoded image item with item_ID 1 indicates that the offset in the file is O1 and the size is L1 bytes, and the location of the data in box 2203 is specified.

[0261] Description 2414 corresponds to box 2215 and indicates the reference relationship (association) between each item. In the item reference shown in description 2450, dimg is specified, which indicates that the reference type is a derived image. It is also shown that the grid derived image item with item_ID5 specified in from_item_ID refers to the HEVC encoded image items with item_ID1 to 4 specified in to_item_ID. As a result, the encoded image items with item_ID1 to 4 are specified as input images to the derived image item.

[0262] In the item reference shown in description 2451, thmb, which indicates a thumbnail relationship, is specified as the reference type. It is indicated that the HEVC encoded image item with item_ID6 specified in from_item_ID refers to the grid derived image item with item_ID5 specified in to_item_ID. This indicates that the HEVC encoded image item with item_ID6 is a thumbnail image of the grid derived image item with item_ID5.

[0263] In the item references shown in descriptions 2452, 2453, 2456, 2457, and 2458, cdsc, which indicates a content description relationship, is specified as the reference type. Description 2452 indicates that the area information item with item_ID50 specified in from_item_ID references the grid derived image item with item_ID5 specified in to_item_ID. As a result, the area information item with item_ID50 indicates a partial area within the grid derived image item with item_ID5. Similarly, description 2453 indicates that the area information item with item_ID56 specified in from_item_ID references the grid derived image item with item_ID5 specified in to_item_ID. As a result, the area information item with item_ID56 indicates a partial area within the grid derived image item with item_ID5. Description 2456 indicates that the cbrg derived area item with item_ID57 specified in from_item_ID refers to the grid derived image item with item_ID5 specified in to_item_ID. As a result, the combination area indicated by the cbrg derived area item with item_ID57 indicates a partial area in the grid derived image item with item_ID5. Similarly, description 2457 indicates that the cbrg derived area item with item_ID58 specified in from_item_ID refers to the grid derived image item with item_ID5 specified in to_item_ID. As a result, the combination area indicated by the cbrg derived area item with item_ID58 indicates a partial area in the grid derived image item with item_ID5. Description 2458 indicates that the Exif data block item with item_ID1000 specified in from_item_ID refers to the grid derived image item with item_ID5 specified in to_item_ID. This indicates that the Exif data block item with item_ID1000 describes information about the time when the grid derived image item with item_ID5 was shot.

[0264] The item references shown in the descriptions 2454 and 2455 specify drgn, which indicates that the reference type is a derived area. The description 2454 indicates that the cbrg derived area item with item_ID 57 specified in the from_item_ID refers to the area information items with item_ID 50, 51, 52, 54, and 55 specified in the to_item_ID. This specifies the area information items with item_ID 50, 51, 52, 54, and 55 as input areas to the derived area item. This combines the areas shown in the area information items with item_ID 50, 51, 52, 54, and 55 as a combination type cbrg, and all areas included in any of the five areas are specified as areas shown by the cbrg derived area item. The description 2455 indicates that the cbrg derived area item with item_ID 58 specified in the from_item_ID refers to the area information items with item_ID 53 and 56 specified in the to_item_ID. This specifies the area information items with item_IDs 53 and 56 as input areas to the derived area item. This combines the areas shown in the area information items with item_IDs 53 and 56 as a combination type cbrg, and all areas included in either of the two areas are specified as areas shown by the cbrg derived area item.

[0265] Description 2415 corresponds to box 2217, and specifies the data structure of a grid derived image item with item ID 5. In the HEIF file of this embodiment, rows_minus_one of the grid derived image is set to 2, and columns_minus_one is set to 2. This indicates that a 2x2 input image is configured as a grid derived image. Note that the number of items (four) specified in ref_count as the number of input images for item reference shown in description 2450 and the number of grids (2x2(4)) are specified as matching values. Also, 4032 is specified in output_width, and 3024 is specified in output_height, so that an image with a width of 4032 pixels and a height of 3024 pixels is configured. Note that since it does not have the cbrg derived area item data structure shown in this embodiment, it does not have a data structure to be stored in box 2217. However, in the case of a derived area item that stores a data structure, such as the RegionLogicalDerivation derived area item, the data structure is stored in box 2217 of description 2415.

[0266] Description 2416 corresponds to box 2216, and includes description 2420 corresponding to box 2221 and description 2421 corresponding to box 2222. Description 2420 lists attribute information that can be used for each item or entity group as entry data. As shown in the figure, description 2420 includes 1st and 2nd entries indicating encoding parameters, 3rd, 4th, and 5th entries indicating display pixel size of the item, and 6th to 14th entries indicating annotation description information. In the annotation description information, lang is American English (en-US) in all entries, "human" is set as name in the 6th to 12th entries, and "family" is set in the 13th and 14th entries. "Hanako", "Taro", "Shohei", "Mike", "Naomi", "Ginnji", and "Susan" are set as description, respectively, indicating the names of the people indicated by the area. In addition, "Taro family" and "Mike family" are set as description in the 13th and 14th entries. "AutoRecognition" is set to tags to indicate that the area was automatically recognized.

[0267] The attribute information listed in the description 2420 is associated with each item or entity group stored in the HEIF file in the entry data of the description 2421 corresponding to the box 2222. In the example of FIG. 24, the image items with item_IDs 1 to 4 are associated with a common 'ispe' (property_index is 4), indicating that they are images with the same image size of 2016 pixels x 1512 pixels. Similarly, the image items with item_IDs 1 to 4 are associated with a common 'hvcC' (property_index is 1), indicating that they have the same encoding parameters. Meanwhile, the items or entity groups with item_IDs 5 to 15 are associated with different item properties. The derived image item with item_ID 5 is associated with 'ispe' (property_index is 3), indicating that it has an image size of 4032 pixels x 3024 pixels.

[0268] The image item with item_ID 6 is associated with 'ispe' (property_index is 5), which indicates that the image size is 768 pixels x 576 pixels. Similarly, the image item with item_ID 6 is associated with 'hvcC' (property_index is 2), which indicates coding parameters. The region information item with item_ID 50 is associated with 'udes' (property_index is 6), which indicates region annotation information for the region. The region information item with item_ID 56 is associated with 'udes' (property_index is 12), which indicates region annotation information for the region. In this way, by associating the 'udes' property, region annotation information for the region is given. Furthermore, the derived region item with item_ID 57 is associated with 'udes' (property_index is 13), which indicates region annotation information for the combined region. The derived region item with item_ID 58 is associated with 'udes' (property_index is 14), which indicates region annotation information for the combined region. In this way, by associating the 'udes' property with the entity group, the annotation information for the combined area is realized. Note that since no item property is associated with the Exif data block item with item_ID 1000, no corresponding entry information is stored.

[0269] In the above-mentioned embodiment, the combination area information according to the combination type stored in the image file specifies the combination type as a derived area using the derivative conversion type information and parameters of the input area using the area derivation. Then, by associating the derived area item with the image, it is possible to identify the subject in the image by combining multiple areas. Furthermore, it is possible to associate area annotation information with the combination area. Here, the type of derivative conversion is one example, and other combination types can also be defined, and it is sufficient that the derivative conversion is used to combine and handle the areas. Furthermore, when the area is derivative converted, it is possible to specify other derived areas as the input area. This allows for more flexible area specification. Furthermore, when treating an individual area as a derived area with an input area as the individual area, it is possible to make the individual area an input area for a different derived area. According to this, one subject object can be identified as belonging to multiple derived areas, and furthermore, it is possible to annotate each of the areas. Note that the four-character code for identifying the derived area item type of the combination area shown in the embodiment is one example, and it is sufficient that the item type of the derived area has an arbitrary four-character code reserved for each derived area item type.

[0270] [Third embodiment] In the above-described embodiment, the method of identifying area information as a combination area according to the type is described by using an entity group and a derived area, but the present invention is not limited to this. In the present embodiment, a method of identifying a combination area will be described by improving the data structure of area information defined as an item.

[0271] Specifically, flags302 in the definition of the region information shown in FIG. 3 or a separate additional parameter is defined, and the combination type shown in the above-mentioned embodiment is defined as an identifiable value. For example, when flags is used, if flags&2 is valid, it is specified that it is a cbrg type combination region. In this case, multiple region items cannot be combined, but it is possible to define multiple regions in one region item (region_count is 2 or more) and specify them as a combined region. Also, for combination types that require separate parameters, the parameters can be associated using item properties.

[0272] In this way, according to the present embodiment, when one or more pieces of region information that identify objects that are related to a part of an image and are stored in an image file in association with the image, it is possible to efficiently perform flexible region specification by storing combination type information that combines the region information and combination region information that identifies a combination region according to the combination type information. In this case, although it is not possible to associate region annotation information with individual regions, it is possible to flexibly specify one region as one region item.

[0273] [Fourth embodiment] In the above embodiment, the method of identifying area information in a two-dimensional image as a combined area according to the type and associating it with the two-dimensional image is shown, but the present invention is not limited to this. That is, it is possible to define area information of a three-dimensional shape for a three-dimensional graphic image and combine the methods shown in the above embodiment.

[0274] This makes it possible to individually identify 3D region information not only for 2D images but also for 3D images composed of point cloud data, mesh data, etc., and further to combine and identify these regions according to their type.

[0275] Thus, according to this embodiment, when one or more pieces of three-dimensional region information that identify a three-dimensional object represented by a three-dimensional image related to a part of the three-dimensional image are stored in an image file in association with the three-dimensional image, combination type information that combines the three-dimensional region information and combination region information that identifies a combination region according to the combination type information are stored, making it possible to flexibly and efficiently specify regions.

[0276] [Fifth embodiment] In the above-described embodiment, the images are mainly still images, and the area information is identified by associating it with the still image. Then, the method of identifying the areas as combined areas according to the type and associating it with the still image is described. However, the present invention is not limited to this.

[0277] That is, it is possible to define region information for a video or image sequence, and to specify a combination region according to the type using a metadata structure for storing the video. In a video or image sequence, the region corresponding to the object detected from the image by the recognition processing unit 114 may move within the image over time. Therefore, in a certain image, regions may overlap, and regions may appear or disappear (e.g., occlusion by other regions, or objects outside the image or frame).

[0278] The storage method is to describe metadata about videos and image sequences using the moov box, a metadata structure defined in ISOBMFF. This describes a timed sequence of media data belonging to a video presentation in a trak box specified in the moov box. For example, a sequence of video frames, a sequence of subparts of video frames, a sequence of audio samples, or a sequence of timed metadata samples. Within each track, each time unit of data is called a sample. This may be a frame or subpart of a frame of video, audio or timed metadata, or an image in an image sequence. A sample is defined as all media data associated with the same presentation time in a track.

[0279] The region information is stored in an mdat box that stores the encoded data of the video. The metadata of the timed sequence that describes the region information is configured as a track called a metadata track that stores timed metadata. The metadata track can be associated with a video track using a tref box to indicate a region in the video. A plurality of combination regions can be described in a plurality of region metadata tracks, and these tracks can be used as a track group to specify a combination region according to a track group type. Region annotation information can be associated using a track group description box. When a region related to a plurality of different objects is specified in one region metadata track, each region can be configured to be identifiable by an ID or the like in the region information stored in the mdat box, and grouped using a sample group structure. Group type information can also be specified for the sample group, and region annotation information can be added to the combination region according to the group type using a sample group description box.

[0280] It is also possible to define samples for describing combination regions and store them as the same or separate metadata track. Now, with reference to Figures 28, 29, 30 and 31, a method of defining region information for a video or image sequence and specifying combination regions according to the type using a metadata structure for storing video and storing them in a file will be described.

[0281] FIG. 28 shows an example of a presentation including region information stored in a media file, where the mapping between regions and their associated annotations is included in samples of a region track, according to some embodiments. Media data consisting of a timed sequence of images or video frames is stored in a track 2810, called a source track. In this source track, each timed unit of the timed sequence is encapsulated in a sample (2811, 2812, 2813). For example, a source track can be a video track (e.g., a track containing a handler box with a handler type of 'vide') or an image sequence track (e.g., a track containing a handler box with a handler type of 'pict'). Some of the samples 2811, 2812, 2813 may have region information specified and may have annotations associated with them. Regions may be detected by image analysis algorithms or may be selected by a user. The geometry, i.e. the position and / or shape, of regions within samples 2811, 2812 or 2813 of a source track 2810 is described in a region track 2820, which can be considered a metadata track, e.g. a track containing a handler box with handler type 'meta'.

[0282] The metadata track 2820 consists of sample entries with e.g. a coding name or the four-letter sample entry type 'rgan' dedicated to the 'region annotation' track or 'region' track. The metadata track 2820 is called a 'region track'. In a region track, the samples define one or more regions of an image that are included in the sample of the associated image sequence or video track (also called source track).

[0283] The region track 2820 is associated with a source track 2810 with trackID 1 which contains the sample and the region is defined using a track reference of type 'cdsc' from the region track to the source track as shown at 28252 .

[0284] The region track 2820 shown in FIG. 28 is composed of samples 2821 describing two regions with region identifiers equal to 1 and 2. These regions move between successive samples. For example, the positions and possibly the shapes of regions 1 and 2 differ in sample 2822. Then, these regions with region identifiers 1 and 2 disappear, and a new region with region identifier equal to 3 appears as in sample 2823. The region identifiers and the corresponding region position and shape information are generated by the generation unit 113 according to the information of the object detected from the image by the recognition processing unit 114. Note that the region information and the like are generated according to the information of the region specified by the user operating the operation input unit 107, not limited to the detection result of the object detected by the recognition processing unit 114. The generated region identifiers and region information are stored in a file as metadata by the metadata processing unit 112. There are annotations called region annotations in the region track, and they may be associated with one or more samples of the region track, such as samples 2821, 2822, and 2823. A region annotation may be an item such as 2840, e.g. an image item, a metadata item, another track 2830, or samples 2831, 2832, 2833 of another track 2830, or a simple label or text such as the example in 2824. Simple annotations as shown in 2824 are provided as SampleGroupDescriptionEntry in a SampleGroupDescriptionBox, or as item properties (not shown) of the region track 2820 itself. In 2824, the SampleGroupDescriptionBox can have any value of grouping_type indicating the type of property described by the SampleGroupDescriptionEntry. In 2824, the annotation consists of a textual user description indicated by the four-letter code 'udes' as the grouping_type value. If the annotation of the region is configured in a separate track such as the annotation track 2830, this track can be a metadata track, a video track, an audio track, a 3D video track, or an image sequence track depending on the type of annotation.The region track 2820 is associated with a region annotation track 2830 using a track reference type 'anot' as shown in 28251. Using the same mechanism (28251), the region track 2820 can reference annotation items such as 2840. In this case, the media file needs to be branded with 'unif' as in the example shown in the first embodiment. Each region of samples 2821, 2822, or 2823 of the region track 2820 is associated with a region identifier. The same region identifier can be used between consecutive samples to identify the same moving region between consecutive samples. The scope of the region identifier can be persistent across multiple samples in the region track, or the scope of the region identifier can be the entire media file. Limiting the scope to the region track makes it easier to edit the file. As a specific example, the creation of a new region track does not require analysis of the region identifiers of other region tracks to detect possible conflicts, and any region identifier can be used in the newly created region track. On the other hand, it may be easier to combine regions or describe relationships between regions if the region identifier is scoped to the entire media file. This is explained in more detail below when discussing combined regions or region relationships. If a region identifier scope is present in the media file (either the entire media file or multiple samples in a region track), the client or reader is guaranteed that the same region identifier in different samples identifies the same region. If the media file does not indicate a region identifier scope, the same region identifiers across samples may or may not be used by the content creator, and the client or reader has no guarantee that the same region identifier will identify the same moving region across different samples.

[0285] The information specifying the scope of a region identifier in a region track may be stored in the sample entry of the region track using a flag indicating whether the scope of the region identifier is persistent for all samples associated with this sample entry. For example, this is done in the RegionSampleEntry describing the samples of the region track or in one of its sub-boxes such as RegionTrackConfigBox. Alternatively, for sample entries, the specification of the scope of the region identifier to the region track may be specified in the track header box, for example as a specific flag value. The storage of the identification information when the region identifier is scoped to the whole media file is identified using a higher level box structure than the track (for example, as a brand of 'ftyp' or 'styp', having the same scope or describing the same source track, or within a track group of the region track in an entity group). The HEIF specification defines the payload of a sample in a region track. The data (or payload) of a sample in a region track describes the shape, position, size, etc. of the region.

[0286] These region information define the shape, position, and size of the region in the reference space that is mapped to samples 2811, 2812, 2813 of the source track 2810 with which the region track 2820 is associated. For example, the shape of the region can be a point, line, rectangle, polygon, binary mask, etc., which is indicated in the sample's payload by the geometry_type parameter. The HEIF specification also defines a means to associate annotations to regions of samples in a region track; for example, the sample-to-region-id-mapping ('regm') sample group can be used (not shown).

[0287] FIG. 29 shows an example of a region track 2920. For example, samples 2921 and 2922 contain regions 1 and 2, and the union of these two regions as third regions 2927 and 2928. This is an example of a combination of regions. The combination methods shown in the first and second embodiments, such as logical OR, AND, and NOT, can be applied as well. In an embodiment, the region track 2920 is extended to support regions resulting from operations on regions. For clarity and brevity, a region resulting from an operation or transformation into one or more regions is called a "combined region" or a "derived region". For clarity and brevity, a region used in a transformation or operation is also called an "input" region, a "simple" region, or a "basic" region. In this embodiment, a region track is defined as a metadata track of 'rgan' sample entry type with samples that define one or more regions in the images of the associated image sequence or video track. While these one or more regions can be simple regions (or basic regions), they can also be combined regions (or derived regions). Additionally, simple (or basic) regions can be combined with combined (or derived) regions, or combined with each other. Each region in a region track sample, regardless of its type, is associated with a region identifier. This allows for associating annotations with regions by region identifiers, for example using existing mechanisms provided by HEIF. Combined or derived regions are distinguished from simple or basic regions in a region track sample by specific values ​​of geometry_type.

[0288] There are several further ways how to describe a combination of one or more regions within an image of an image sequence or a video track. The first way is that the type of geometry (geometry_type) directly indicates the type of combination or derivation.

[0289] These types can be specified without any restrictions on combination types such as logical sum, logical product, and logical negation shown in embodiments 1 and 2. Next, specific integer values ​​of geometry_type are reserved to represent combinations of one or another type to enable interoperability between media file writers (or storage modules or packaging modules) and readers (or analysis modules or media players). The payload (or data) of a sample from a region track can be expanded as follows: aligned (8) class RegionSample { unsigned int field_size = ((RegionTrackConfigBox.flags & field_length_size) == field_length_size) ? 32 : 16; / / this is a temporary, non-parsable variable unsigned int(32) region_count; for (r=0; r < region_count; r++) { unsigned int(32) region_identifier; unsigned int(8) geometry_type; if (geometry_type == 0) { / / point signed int(field_size) x; signed int(field_size) y; } else if (geometry_type == 1) { / / rectangle signed int(field_size) x; signed int(field_size) y; unsigned int(field_size) width; unsigned int(field_size) height; } / *Other basic domain shape types are the same as in Figure 3, so omitted* / else if (geometry_type == 6) { / / union of regions unsigned int (8) num_regions; (for int n=0; n < num_regions, n++) { unsigned int(32) region_id; / / regions that are part of the union } } else if (geometry_type == 7) { / / intersection of regions unsigned int (8) num_regions; (for int n=0; n < num_regions, n++) { unsigned int(32) region_id; / / regions that are part of the Intersection } else if (geometry_type == 8) { / / negation of regions unsigned int (8) num_regions; (for int n=0; n < num_regions, n++) { unsigned int(32) region_id; / / regions that are part of the negation } / * From now on, other combination types can be specified in the same way.* / }

[0290] The value of the geometry_type of each combination region in the above RegionSample may be any value that can be specified.

[0291] Here, num_regions indicates the number of input regions for the derivation or combination. Here, "region_id" indicates the region_identifier of the input region for the derivation or combination. For combined or derived regions, there is no data regarding shape information (shape, position, size, etc.) (optional). If necessary, the shape information of combined or derived regions can be inferred from the input regions. For example, the shape can be easily inferred if all input regions have a geometry type specified as rectangular shape and are configured as rectangular bounding boxes arranged in a grid, or as peripheral regions regularly arranged around a central region.

[0292] Input regions and combined regions may be interleaved within a sample payload, for example when object detection or recognition is performed in one pass. This means that the order of the base region geometry types and combined region geometry types specified within a sample can be any order. Combined regions may also be defined at the end of a sample payload, for example when object detection or recognition is performed in multiple passes. Optionally, the integer value of geometry_type corresponding to the region combination may be preceded by a most significant bit specified as 1, or by a reserved range corresponding to the larger of the 8-bit integer values ​​allowed for geometry_type. For example, the reserved range of values ​​specifying the region combination may be in the range 11110001 to 11111111 (in binary notation), or 241 to 255 (in decimal notation). This allows 15 types for the combinations and 241 types for other geometry types that are base regions describing the shape of the regions. The range can be adapted to assign more values ​​to the combinations as needed.

[0293] In the second specification method, geometry_type uses a specific integer value (for example, 0xFF) to indicate that the region is a combined region or derived region, and combination type information of the combined region or derived region is further specified using combination_type. As with the above, the types that can be specified in combination_type are combination types such as logical sum, logical product, and logical negation shown in embodiments 1 and 2, and can be specified without any restrictions. In this embodiment, the sample payload is updated as follows. aligned (8) class RegionSample { unsigned int field_size = ((RegionTrackConfigBox.flags & field_length_size) == field_length_size) ? 32 : 16; / / this is a temporary, non-parsable variable unsigned int(32) region_count; for (r=0; r < region_count; r++) { unsigned int(32) region_identifier; unsigned int(8) geometry_type; if (geometry_type == 0) { / / point signed int(field_size) x; signed int(field_size) y; } else if (geometry_type == 1) { / / rectangle signed int(field_size) x; signed int(field_size) y; unsigned int(field_size) width; unsigned int(field_size) height; } / *Other basic domain shape types are the same as in Figure 3, so omitted* / else if (geometry_type == 0xFF) { / * reserved value for generic combination * / unsigned int (8) combination_type; unsigned int (8) num_regions; for (int n=0; n < num_regions, n++) { unsigned int(32) region_id; / / regions that are part of the combination }

[0294] Here, combination_type indicates the type of region derivation or region combination (e.g., logical OR, logical AND, logical negation, containment relationship, contact relationship, etc.). Here, num_regions indicates the number of input regions for the derivation or combination. Here, region_id indicates the region_identifier of the input region for the derivation or combination. Optionally, in the first or second specification method, the sample entry of the region track 'rgan' can include additional information such as parameters and flags. For example, the HEIF specification defines "combination_present" as a flag value that can be specified in the RegionTrackConfigBox. If it is set to a specific value, it indicates that the region track may contain region combinations or derivations, and if it is not set to this specific value, it indicates that the region track contains only basic or simple regions. aligned(8) class RegionTrackConfigBox () extends FullBox ('rgaC', version=0, flags) { unsigned int((field_length_size + 1) * 16) reference_width; unsigned int((field_length_size + 1) * 16) reference_height; unsigned int(1) combination_present; unsigned int(7) reserved; }

[0295] Here, if combination_present is set, it indicates that the sample referenced by the sample entry containing this box may contain a combination of regions. If it is not set, the parser or reader can identify that the sample referenced by the sample entry containing this box does not contain a combination of regions. By retaining this information at the sample entry level, the parser or reader can determine that some combination of regions is present from the beginning of the file. In the case of a legacy player that does not support combination of regions, it may not process the region track, or may ignore and process some of the samples that have the geometry_type corresponding to the combination of regions, and may not be able to recognize or interpret it. In this case, by specifying the region from the simple region or basic region and storing it (without interleaving the simple region and the combined region), storing the combined region at the end of the sample allows processing to be interrupted when the combined region appears, making it possible to process only the simple region or basic region. It is also possible to process all of the simple region or basic region.

[0296] The third method of specification is to use a flag value specification when including a region combination or derived combination_present in the region track, and indicate the combination_type at the same level as the region_identifier in the payload of the sample in the region track, as in the second method of specification. aligned (8) class RegionSample { unsigned int field_size = ((RegionTrackConfigBox.flags & field_length_size) == field_length_size) ? 32 : 16; / / this is a temporary, non-parsable variable unsigned int(32) region_count; for (r=0; r < region_count; r++) { unsigned int(32) region_identifier; unsigned int(8) geometry_type; if (RegionTrackConfigBox.combination_present ==1) { unsigned int (8) combination_type; } if (geometry_type == 0) { / / point signed int(field_size) x; signed int(field_size) y; } else if (geometry_type == 1) { / / rectangle signed int(field_size) x; signed int(field_size) y; unsigned int(field_size) width; unsigned int(field_size) height; } / *Other basic domain shape types are the same as in Figure 3, so omitted* / else if (geometry_type == 0xFF) { / * reserved value for generic combination * / unsigned int (8) num_regions; for (int n=0; n < num_regions, n++) { unsigned int(32) region_id; / / regions that are part of the combination }

[0297] Here, combination_type indicates the type of region derivation or combination of regions (e.g., logical OR, AND, NOT, etc.). A specific value, e.g., 0, indicates that the combination is "uncombined" or "none". This allows avoiding specifying combination_type when a sample entry indicates that there are no combined or derived regions. Then, geometry_type uses a generic value (e.g., 0xFF), and the information describing combined or derived regions in the payload of a region track sample simply consists of a list of region_identifiers for the input regions involved in the combination or derivation.

[0298] The above several specification methods and optional methods in this embodiment are effective when a media file contains one region track. However, there are some drawbacks, especially when the region or object recognition decision is made in multiple passes. For example, it is necessary to update the sample payload and possibly the corresponding sample size and chunk offset in the sample description of the region track. Furthermore, it is not possible to handle region combinations across different region tracks. On the other hand, the method of this embodiment has the advantage that the number of region tracks 2920 can be kept small, and there may be a possibility that there is only one region track in a typical use case. These are trade-offs between the description cost and the number of specifications for the target use case. The ability to specify the combination or derivation of the regions described in the sample and having a region identifier provides a unified way to associate annotations with regions, regardless of their type, such as simple, basic, or combination, derived (see FIG. 28). Furthermore, the time axis aspect of the combination can be handled. If there is no combination declared in the sample payload, it means that there is no combination of regions in this sample.

[0299] The above-described method for extending the definition of RegionSample can also be achieved by the method for defining RegionItem shown in the third embodiment.

[0300] Another example of this embodiment is shown in Figure 30. This embodiment is an embodiment that improves on some of the above-mentioned shortcomings, particularly the need to update the payload of the samples in the area track.

[0301] In this embodiment, a combined or derived region is defined in a specific track 3050. This track is newly defined as a CombinedRegionTrack 3050, and is configured to include specific sample entries 3051. This track may use specific boxes in the sample entries as a way to provide a region combination type 3052.

[0302] A CombinedRegionTrack is a metadata track with an arbitrary four-letter code reserved for describing the sample entries 'cbrg' or "Combined Regions".

[0303] This is a type of region track whose samples define one or more combined or derived regions, taking as input regions the regions defined in one or more region tracks 3020 and / or 3030. The combined region track 3050 is associated with the region tracks whose input regions to the combined or derived regions are defined using a track reference of type 'cbrg' (3053) from the combined region track 3050 to one or more region tracks 3020 and / or 3030. The combined region track can be associated with the source track 3010 by a track reference of type 'cdsc' 3054. If 3054 is not present, the source track can be obtained via the 'cbrg' track reference type 3053 and its track reference type 'cdsc' 3021 or 3031 can be traced for at least one region track referenced by the 'cbrg' track reference type 3053. If 3054 is present, the source track can be directly identified from the combined region track 3050. The combined region track 3050, like the region tracks 3020 or 3030, can associate a region annotation with each combined or derived region defined in the samples of the associated source track. For a region track, the region annotation can be an item (e.g. an image item, a metadata item), a sample in another track (see annotation track 2930 in Fig. 28), or a SampleGroupDescriptionEntry. Each region in the samples of the combined region track is associated with a region identifier. The extent to which the region identifier of a combined region is uniquely identifiable is indicated by the persistent_region_ids flag in the sample entry of the combined region track or one of its subboxes. The region identifier can be used to associate annotations to combined regions in the sample via an association map defined in the sample-to-region-id-mapping sample group. In such a case, the values ​​of the region identifier in the sample and the region identifier in the sample group are in the same value space. The combined regions in the combined region track do not need to be described in terms of their shape, position, size, etc., since they can be calculated from the input regions.If the overall shape, position, size, etc. of the combined region are retained at the time of file generation, they may be included in the payload of the sample of the combined region track. The samples of the combined region track must be synchronized or aligned in time with the corresponding samples of the region track referenced via the 'cbrg' track reference 3053. Note that the combined region track may be a mixture of combined regions of samples and basic or simple regions. For example, when repetitive object detection or recognition is performed and additional simple or basic regions are defined by the repetition. It may be easier to specify these new regions in the combined region track than to edit the existing region track. The distinction between combined regions and basic regions can be made using the geometry_type described in the previous embodiment and its variants. For each synchronization sample 3016 of the source track or each synchronization sample 3026 or 3036 of the region track, it is desirable to have a synchronization sample 3056 in the combined region track 3050 that is aligned in time with this synchronization sample (example of time Ts in FIG. 30). There are several options for defining the combination type 3052 of a combined region track, depending on whether it contains combinations of a single type, or whether it contains a mixture of combinations and derivations of regions of different types.

[0304] The first way is for the combined region track 3050 to contain combined or derived regions of a single combination type, where the combination type is specified in the sample entry or one of its subboxes (i.e. the 'stsd' box or one of its subboxes). This avoids repeating the combination type for each region in each sample of the combined region track (e.g. 3055 and 3056). This can be done with a new version of the RegionTrackConfigBox, a new config box specific to the combined region track, or a new box included in the sample entry of type 'cbrg' (e.g. in addition to the RegionTrackConfigBox defined as version 0 in the HEIF specification).

[0305] To extend the RegionTrackConfigBox to indicate the combination type 3052 of the combined region track 3050, it can be written as follows: aligned(8) class RegionTrackConfigBox () extends FullBox ('rgaC', version, flags) { unsigned int((field_length_size + 1) * 16) reference_width; unsigned int((field_length_size + 1) * 16) reference_height; if (version > 0 ) unsigned int (8) combination_type; } aligned(8) class CombinedRegionSampleEntry extends MetadataSampleEntry ('cbrg'){ RegionTrackConfigBox config; / / mandatory with version=1 }

[0306] If combination_type 3052 is in a new config box specific to the combined region track 3050, this box can be defined as follows: aligned(8) class CombinedRegionTrackConfigBox () extends FullBox ('cbrC', version, flags) { unsigned int (8) combination_type; / / optional parameters, can be controlled by flags or version fields of the box unsigned int((field_length_size + 1) * 16) reference_width; unsigned int((field_length_size + 1) * 16) reference_height; } aligned(8) class CombinedRegionSampleEntry extends MetadataSampleEntry ('cbrg'){ CombinedRegionTrackConfigBox config; }

[0307] In the CombinedRegionSampleEntry, the config box can be omitted by specifying the default combination type for the combined region track, such as combining regions by logical OR, which is a combination type that is likely to be used in common usage. In other words, there is no need to specify the default combination type. If combination_type uses a combination type other than the default value, the config box must be present in the sample entry that describes the sample of the combined region track.

[0308] The parameters reference_width and reference_height are optional in CombinedRegionTrackConfigBox since they may be the same as the referenced region track, especially if they share the same reference space or if there is only one region track. If the referenced region track does not have the same reference space, these parameters may be specified in CombinedRegionTrackConfigBox by the encapsulation module to allow the reader to correctly position the combined region on the source image.

[0309] If combination_type3052 were to be included in a new box in a sample entry of type 'cbrg', it could be specified as follows: (The names and 4-letter codes used below are only examples, you can use any reserved name and 4-letter code as long as it does not conflict with others already in use.) aligned(8) class CombinedRegionSampleEntry extends MetadataSampleEntry ('cbrg'){ RegionTrackConfigBox config; / / with version = 0 RegionCombinationTypeBox combination_type; }

[0310] For example, RegionCombinationTypeBox is defined as follows: aligned(8) class RegionCombinationTypeBox () extends FullBox ('rcbt', version, flags) { unsigned int (8) combination_type; / / optional parameters unsigned int(8) num_input_regions; unsigned int(8) combination_shape; / / possibly other parameters avoiding repetition in the samples }

[0311] The combination type specification 3052 may also include optional parameters, whatever the means mentioned above, especially if the combination uses a fixed number of regions or results in a single shape. The optional parameters may provide a parameter indicating how many input regions are fixed (e.g., num_input_regions parameter) or a parameter indicating shape information in case of a repetition of a single shape (e.g., combination_shape parameter) so that it is not repeated in samples 3055 or 3056 of the combined regions track 3050. The value used for the shape information parameter may reuse the value defined in the geometry_type value of the region track sample (e.g., 1 for rectangle, 2 for ellipse, etc.). If the value of these parameters is specified as 0, it is reserved to indicate that a specific combination shape is not used and may optionally be present in samples 3055 or 3056 of the combined regions track 3050. Also, in the case of num_input_regions, it should be interpreted that the number of input regions may vary for each combination region, and this number of input regions is indicated in samples 3055 or 3056 of the combined regions track 3050. Then, in this first method, depending on the parameters present in the sample entry or its subboxes, a sample of the combined region track can be represented as follows: aligned (8) class RegionCombinationSample { unsigned int(8) combined_region_count; for (r=0; r < combined_region_count; r++) { unsigned int(8) combined_region_identifier; / * if not provided at sample entry level (or sub-box)* / unsigned int (8) num_input_regions; for (n=0; n < num_ input _regions, n++) { unsigned int(32) input_region_identifier; / / region that are part of the union } / / optionally shape, position, size information may be described for the combined region, if not provided at sample entry level (or sub-box) } }

[0312] num_input_regions indicates the number of input regions used in the combined region, which can be taken from the sample entry or one of its subboxes if available, or declared in the sample otherwise.

[0313] In the second way, a combined region track like 3050 contains several types of combined or derived regions. In this case, the combined type 3052 may be pre-declared in the sample entry or a sub-box of the sample entry (such as a config box or a dedicated box) and referenced in the sample. If the combined type of 3052 is pre-declared in the sample entry and referenced in the sample, the sample entry can be defined as follows: aligned(8) class CombinedRegionSampleEntry extends MetadataSampleEntry ('cbrg'){ unsigned int (8) num_combination_types; unsigned int (8) combination_type[num_combination_types] / / possibly additional parameters describing combinations that are common to several samples }

[0314] Here, num_combination_types indicates the number of combination types used in the samples that reference this sample entry.

[0315] Here, combination_type indicates a list of combination types used in the samples that reference this sample entry. The combination types are mapped to reserved values, such as 8-bit integer values. For example, the union of regions may be specified as 1, the intersection as 2, etc., and the value 0 may be reserved. These combination type values ​​are used in sample 3055 or 3056 as follows: aligned (8) class CombinedRegionSample { unsigned int(8) combined_region_count; for (r=0; r < combined_region_count; r++) { unsigned int(8) combined_region_identifier; unsigned int(8) combination_idx; unsigned int(8) num_input_regions; for (i=0; i < num_input_regions; i++) { unsigned int(8) input_region_identifier; } } }

[0316] Here, combined_region_count indicates the number of combined regions in the sample. It must be at least 1 (if it is 0, the sample does not need to be declared). Here, combined_region_identifier indicates the region identifier of the combined region. This region identifier can be used as any region identifier, such as for annotation association, recombination, or to express relationships with other regions. Here, combination_idx indicates the index in the list of combination types declared in the sample entry that this sample references. The value 0 is reserved. The value should use predefined values, such as integer values ​​for interoperability (1 for union, 2 for intersection, etc.).

[0317] num_input_regions indicates the number of regions used in the combined region, where input_region_identifier indicates the region identifier of the region used in the combined region.

[0318] If a combination type 3052 is pre-declared in a subbox of a sample entry, such as a config box, and referenced in the sample, the sample entry and config box can be defined as follows: aligned(8) class CombinedRegionSampleEntry extends MetadataSampleEntry ('cbrg'){ RegionTrackConfigBox config; / / mandatory with version=1 } aligned(8) class RegionTrackConfigBox () extends FullBox ('rgaC', version, flags) { unsigned int((field_length_size + 1) * 16) reference_width; unsigned int((field_length_size + 1) * 16) reference_height; if (version > 0) { unsigned int (8) num_combination_types; unsigned int (8) combination_type[num_combination_types]; } / / possibly additional parameters describing combinations that are common to several samples }

[0319] Here, the num_combination_types and combination_type parameters have the same meaning as in the previous configuration. This syntax uses the new version of RegionTrackBox, but you can also use a specific config box: aligned(8) class CombinedRegionSampleEntry extends MetadataSampleEntry ('cbrg'){ CombinedRegionTrackConfigBox config; } aligned(8) class CombinedRegionTrackConfigBox () extends FullBox ('cbrC', version, flags) { unsigned int (8) num_combination_types; unsigned int (8) combination_type[num_combination_types]; / / optional parameters, can be controlled by flags or version fields of the box unsigned int((field_length_size + 1) * 16) reference_width; unsigned int((field_length_size + 1) * 16) reference_height; / / possibly additional parameters describing combinations that are common to several samples }

[0320] Here, the num_combination_types and combination_type parameters have the same meaning as in the configuration described above. Here, the identification of the reference space is optionally indicated, for example, if it cannot be deduced from the reference region track. For example, the parameters related to the reference space can be omitted if there is only one region track or if all referenced region tracks have the same reference space.

[0321] Reference to a combination type in a sample is made in the same manner as described above through the combination_idx parameter which provides an index into the predefined list, in this case within the config box of the sample entry to which the sample refers.

[0322] If a combination type 3052 is pre-declared in a sub-box of a sample entry, such as a dedicated box, and referenced in the sample, the sample entry and the dedicated box can be defined as follows: aligned(8) class CombinedRegionSampleEntry extends MetadataSampleEntry ('cbrg'){ RegionTrackConfigBox config; / / with version = 0 RegionCombinationTypeBox combination_types; } For example, a RegionCombinationTypeBox can be defined as follows: aligned(8) class RegionCombinationTypeBox () extends FullBox ('rcbt', version, flags) { unsigned int (8) num_combination_types; unsigned int (8) combination_type[num_combination_types]; / / optional parameters unsigned int(8) num_input_regions; unsigned int(8) combination_shape; / / possibly other parameters avoiding repetition in the samples }

[0323] Here, the num_combination_types and combination_type parameters are as described above. Additional data for the combination type 3052, whatever the above means, may also include optional parameters common to several samples, for example if the combination uses a fixed number of regions or results in a single shape. Optional parameters may provide a parameter indicating how many input regions are fixed (e.g., num_input_regions parameter) or a parameter indicating shape information in case of a repetition of a single shape (e.g., combination_shape parameter) so that it is not repeated in samples 3055 or 3056 of the combined regions track 3050. The value used for the shape information parameter specified here may reuse the value defined in the geometry_type value of the region track samples (1 for rectangle, 2 for ellipse, etc.). The value 0 of these parameters is reserved to indicate that a particular combination shape is not used and may optionally be present in samples 3055 or 3056 of the combined regions track 3050. Also, num_input_regions should be interpreted as the number of input regions may vary for each combined region, and this number of input regions is indicated in samples 3055 or 3056 of the combined regions track 3050.

[0324] In a third method, the combination types 3052 in a combined region track such as 3050 can be described within the payload of a sample 3055 or 3056, regardless of how many combination types are used. For example, a sample in a combined region track can define one or more regions as follows: aligned (8) class RegionCombinationSample { unsigned int(8) combined_region_count; for (r=0; r < combined_region_count; r++) { unsigned int(8) combined_region_identifier; unsigned int(8) combination_type; unsigned int (8) num_input_regions; for (n=0; n < num_ input _regions, n++) { unsigned int(32) input_region_identifier; / / region that are part of the union } / / optionally shape, position, size information may be described for the combined region. } }

[0325] where combined_region_count indicates the number of combined regions in the sample. It must be at least 1 (if 0, no sample declaration is required). where combined_region_identifier indicates the region identifier of the combined region. This region identifier can be used as any region identifier, such as for associating annotations, recombining, expressing relationships with other regions, etc. where combination_type indicates the combination type. A value of 0 may be reserved, for example, to indicate that the region is a basic region (no combination). The value should be a predefined value, such as an integer value for interoperability (1 for union, 2 for intersection, etc.). num_input_regions indicates the number of regions used in the combined region. where input_region_identifier indicates the region identifier of the input region used in the combined region.

[0326] While such a definition requires repetition across samples, it allows for different combination types within the same sample, and is a flexible way of describing samples because they are self-contained. However, it does not allow you to know what combinations are used before processing the sample.

[0327] For some specific uses, provided that the number of combined regions does not change over time (e.g. for an entire presentation or a movie fragment), the combination type is of a given type, and the same number of input regions are combined, a sample Combined Regions Track can be a simple payload such as: aligned (8) class RegionCombinationSample { for (n=0; n < num_ input _regions, n++) { unsigned int(32) input_region_identifier; / / region that are part of the union } }

[0328] where num_input_regions indicates the number of regions used in the combined region, taken from the sample entry or one of its subboxes, and input_region_identifier indicates the region identifiers of the regions used in the combined region.

[0329] In this particular usage, static information (i.e., that does not change over time) can be described at the sample entry level (in the 'stsd' box or one of its sub-boxes).

[0330] FIG. 31A shows an example of a region track 3110, e.g. a 'rgan' track, whose metadata part contains a description of a combined region 3115. The region track contains samples 3111 or 3112 or 3113 that provide a description of the region. No association with a source track is shown here. The description of the combined region 3115 may be indicated as a sample group with a dedicated grouping_type. For example, the information about combined regions indicates 'crif' to identify the sample group type. A group of samples is defined in a SampleToGroupBox ('sbgp' part of 3115) with grouping_type equal to 'crif'. Grouping allows grouping samples whose regions are combined to form one or more combined regions. The associated SampleGroupDescriptionBox ('sgpd' of 3115) with grouping_type 'crif' contains one or more specific SampleGroupDescriptionEntry that provide a description of these combined regions. For example, this 'sgpd' entry can be defined as follows: class CombinedRegionGroupEntry() extends SampleGroupDescriptionEntry('crif') { unsigned int(32) entry_count; for (i=1; i<= entry_count; i++) { unsigned int(8) regionID; unsigned int(8) combination_type; unsigned int (8) num_input_regions; for (n=0; n < num_ input _regions, n++) { unsigned int(32) input_region_identifier; / / region that are part of the union }}

[0331] where entry_count is an integer indicating the number of combined regions in the entry. where regionID is an integer specifying the region identifier of the combined region. where combination_type is an integer specifying the combination type. The value 0 is reserved and the value SHOULD use predefined values ​​such as integer values ​​for interoperability (1 for union, 2 for intersection, etc.). This parameter can be optional if the combination type is a default combination such as union of regions, or is specified elsewhere such as in the track configuration or sample description ('stbl' or 'stsd' or their subboxes). where num_input_regions is an integer specifying the number of input regions used for the combined regions. where input_region_identifier indicates the region identifier of the input region used for the combined regions. It specifies the value of one region identifier in the samples of the region track.

[0332] According to this embodiment, the annotation association mechanism can be extended to consider as region identifiers not only the region identifiers contained in the samples of a region track, but also the regionID contained in this sample group description entry. If a media file contains multiple region tracks, there is a region track that can contain sample groups that describe the region combinations between the samples of this region track. In this example, the combined regions declared in the metadata part 3115 of the region track only use input regions from this region track. That is, the region combinations are only the regions contained in the track. Figure 31B shows an example where the metadata part of a combined region track 3150 contains a description of a combined region 3155. This combined region track is indicated for example by a sample entry type 'cbrg' 3151 in an 'stsd' box. The track 3150 references a region track 3160, whose samples contain region samples 3161, 3162 or 3163 that define regions, simple regions or elementary regions. The association between the region track and the source tracks is not shown, but can be specified in the same way.

[0333] A description of a combined region, for example the combination of regions 1 and 2 in samples 3161 and 3162, may be indicated as a sample group with a dedicated grouping_type, such as 'crif' in the information about combined regions in a combined region track. A group of samples is defined in a SampleToGroupBox ('cbgp' in 3155) with grouping_type 'crif'. Grouping allows grouping samples like 3156 and 3157 from combined region tracks (even if they are empty samples) that have the same combined region, such as the same combination type, the same number of input regions, regions indicated by the same region identifier, etc.

[0334] For example, if two regions with persistent identifiers from time t1 to t2 are declared as the union of a region with region identifier 3, then samples 3156 and 3157 between t1 and t2 will be part of the same sample group. The two regions may be from different region tracks or from the same region track. A combined region sample group can be mapped to a combined region description entry. If not mapped, there is no region combination for this sample group. A combined region description entry can be defined as an entry in the 'sgpd' box as follows (if the scope of the region identifier is global to the file): class CombinedRegionGroupEntry() extends SampleGroupDescriptionEntry('crif') { unsigned int(32) entry_count; for (i=1; i<= entry_count; i++) { unsigned int(8) regionID; unsigned int(8) combination_type; unsigned int (8) num_input_regions; for (n=0; n < num_ input _regions, n++) { unsigned int(32) input_region_identifier; / / region that are part of the union } }

[0335] where entry_count is an integer indicating the number of combined regions in the entry. where regionID is an integer specifying the region identifier of the combined region. where combination_type is an integer specifying the type of combination. The value 0 is reserved and the value SHOULD use predefined values ​​such as integer values ​​for interoperability (1 for union, 2 for intersection, etc.). This parameter can be optional if the combination type is a default combination such as union of regions, or is specified elsewhere such as in the track configuration or in the sample description ('stbl' or 'stsd' or their subboxes). where num_input_regions is an integer specifying the number of input regions used for the combined region. where input_region_identifier indicates the region identifier of the region used in the referencing region track. It specifies the value of one region identifier in one sample of the region track.

[0336] In this embodiment, a combined region declared in the combined region track metadata part 3152 uses input regions from one or more referencing region tracks, i.e. region combinations can be cross-track or from a single region track. The input region can be indicated by a region identifier (input_region_identifier parameter) if its scope is global to the media file, or via an input_region_identifier and trackID pair if the region identifier scope is at track level, or via an input_region_identifier and track_ref_idx pair, where track_ref_idx is the index of an entry in the track reference box of type 'cbrg' 3159 that associates the combined region 3150 with one or more region tracks 3160.

[0337] In a variant, the "CombinedRegionGroupEntry" may not include the combination_type parameter. In that case, this parameter may be provided by the grouping_type_parameter of the SampleToGroupBox with grouping_type equal to 'crif'. This means that there may be as many 'sbgp' and 'sgpd' boxes as there are combination types used. This is not suitable when a default sample grouping is applied to describe the region combinations of 3152 or 3115 that apply to all samples of a track or track fragment, since the SampleToGroupBox 'sbgp' may not be present. According to the embodiment corresponding to Fig. 31A or Fig. 31B, the region combinations can be declared after the construction of the track (at least the data part of the track in the 'mdat' box and part of the description of the samples), without the need to edit the sample payload.

[0338] In tracks providing a description of derived or combined regions (combined regions track 3150 in Fig. 31B, combined regions track 3050 in Fig. 30 or extended regions track in Fig. 29), a 'regm' sample group can be present in this track. This sample group can take into account a region_identifier corresponding to the region identifier of the combined or input region. Region identifier persistence can apply to both basic and combined regions if it is set in a flag in the sample entry (e.g. sample entry type of 'rgan' or 'cbrg' or the 4-letter code used for the region track or combined regions track). To have more fine-grained control over region identifier persistence and to distinguish between simple region persistence and combined region persistence, an additional flag value can be defined in the sample entry of the region track or combined regions track (e.g. sample entry type of 'rgan' or 'cbrg' or the 4-letter code used for the region track or combined regions track) as follows:

[0339] 0x000004 persistent_combined_region_ids indicates the range of region identifiers for the combined regions present in samples that reference this sample entry. When set, it specifies that the scope of the combined region identifiers is persistent to the sample; that is, the same region identifiers for combined regions in different samples mapped to this sample entry identify the same combined region. Otherwise, the scope of the region identifiers for combined regions is the sample. When using separate flag values, the value 0x000002 persistent_region_ids only applies to simple or basic regions.

[0340] It should be noted that the combined region sample group mechanism described with reference to FIG. 31A or FIG. 31B can also be used to express the relationship between regions. The CombinedRegionGroupEntry can be extended to indicate the relationship between regions in addition to or instead of specifying the combination type. For example, this relationship between regions can be indicated as an integer value in a list of predefined values. Alternatively, it can be expressed as a four-character code, but the number of bits required for description is increased. For example, the predefined value is specified as 0 for an inclusion relationship. This means that the first region in the list indicated by region_identifier contains the other regions in the list. Also, 1 indicates a contact relationship indicating a common boundary between the regions. Also, 2 indicates a depth relationship. 3 can indicate an occlusion relationship (e.g., the first region hides the second region), and other relationships between regions as shown in the first and second embodiments.

[0341] It is recommended to use different grouping_type values ​​to identify combined regions and to identify inter-region relationships. For example, 'crif' is the type for combined regions, and another 4-letter code is reserved to indicate region relationships, such as 'rrel' to indicate region relationships. An entry in SampleGroupDescriptionBox 'sgpd' describing inter-region relationships would look like this: class RegionRelationshipGroupEntry() extends SampleGroupDescriptionEntry(’rrel’) { unsigned int(32) entry_count; for (i=1; i<= entry_count; i++) { unsigned int(8) relation_type; unsigned int (8) num_input_regions; for (n=0; n < num_ input _regions, n++) { unsigned int(32) input_region_identifier; / / region that are part of the union } }

[0342] Here, entry_count is an integer indicating the number of relationships described in this entry. Note that for relationships between regions, it is not necessary to define region identifiers. The following parameters are then configured to specify the region identifiers of the regions that have some relationship. In the embodiment related to FIG. 31B, depending on the range of the region identifier, it is indicated by a region identifier (input_region_identifier parameter) if the range is global to the media file. Alternatively, the region can be indicated via a pair of input_region_identifier and trackID if the range of the region identifier is at the track level, or as a pair of input_region_identifier and track_ref_idx, where track_ref_idx is the index of an entry in the track reference box of type 'cbrg' shown in 3159 that associates the combined region track 3150 with one or more region tracks 3160. Alternatively, such an identification can be specified as a grouping_type_parameter in a SampleToGroupBox for the description of the combined region 'crif', or as a grouping_type_parameter in a SampleToGroupBox for the description of the region relation 'rrel'. In this case, the parameter relation_type is not present in the CombinedRegionGroupEntry or the RegionRelationshipGroupEntry.

[0343] As a result, it is possible to identify area information not only for still images but also for moving images, and further to identify these areas by combining them according to their types. It is clear that it is also possible to realize three-dimensional moving images by combining the above-mentioned embodiments.

[0344] In addition, the numerical values, processing timing, processing order, processing subject, data (information) structure / acquisition method / send destination / send source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0345] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0346] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0347] The invention of this specification includes the following information processing device, imaging device, information processing method, and computer program.

[0348] (Item 1) A first generating means for generating one or more pieces of region information describing one or more pieces of shape information for identifying an object included in an image; A second generating means for generating association information that associates the image with the area information; A third generating means for generating combination type information by combining the area information; a fourth generating means for generating combined area information for identifying a combined area according to the combination type information; a storage means for storing the one or more pieces of area information, the association information, the combination type information, and the combination area information in an image file; An information processing device comprising:

[0349] (Item 2) 2. The information processing device according to item 1, wherein the combined region information is information that can be identified separately from the region information.

[0350] (Item 3) 3. The information processing device according to item 1 or 2, wherein the combined region information is information for grouping and identifying the one or more pieces of region information.

[0351] (Item 4) 3. The information processing device according to item 1 or 2, characterized in that the combined region information is information that derives and identifies the one or more region information as a derived region.

[0352] (Item 5) The information processing device described in any one of items 1 to 4, characterized in that the combination type information is information that identifies an area obtained by performing a logical OR of each area indicated by two or more area information as a combination area.

[0353] (Item 6) The information processing device described in any one of items 1 to 4, characterized in that the combination type information is information that identifies an area obtained by performing a logical negation of an area indicated by the one or more area information as a combination area.

[0354] (Item 7) The information processing device according to any one of items 1 to 4, characterized in that the combination type information is information for identifying an area obtained by performing a negative logical OR on areas indicated by two or more area information as a combination area.

[0355] (Item 8) The information processing device described in any one of items 1 to 4, characterized in that the combination type information is information that identifies an area obtained by performing a logical operation on the area indicated by the one or more area information as a combination area.

[0356] (Item 9) 9. The information processing device according to item 8, wherein the logical operation is identified by a parameter.

[0357] (Item 10) An information processing device described in any one of items 1 to 4, characterized in that the combination type information is information that can identify the inclusion relationship of areas indicated by two or more area information, and is information that identifies the relationship between each area.

[0358] (Item 11) The information processing device described in any one of items 1 to 4, characterized in that the combination type information is information that can identify the order of the foreground and background relationships of areas indicated by two or more area information, and is information that identifies the relationship between each area.

[0359] (Item 12) The information processing device described in any one of items 1 to 4, characterized in that the combination type information is information that enables identification that, in two or more area information, a subject area indicated by one area information is an area hidden by a subject area indicated by the other area information, and is information that identifies the relationship between each area.

[0360] (Item 13) The information processing device described in any one of items 1 to 4, characterized in that the combination type information is information that enables identification of a contact relationship between a subject area indicated by one area information and a subject area indicated by another area information in two or more area information, and is information that identifies the relationship between each area.

[0361] (Item 14) 14. The information processing device according to any one of items 1 to 13, wherein the region information and the combined region information are information to which annotation information can be individually associated.

[0362] (Item 15) An information processing device described in any one of items 1 to 14, characterized in that the area information of the area combined according to the combination type information is information that allows separately combined combined area information to be specified as area information.

[0363] (Item 16) 16. The information processing device according to any one of items 1 to 15, wherein the combination type information is stored in the image file as a parameter.

[0364] (Item 17) 16. The information processing device according to any one of items 1 to 15, wherein the combination type information is stored in the image file as a property.

[0365] (Item 18) An information processing device described in any one of items 1 to 17, characterized in that the image file is configured to store in the file as a region combining region items using two CombinedRegionEntityGroups as a file data structure.

[0366] (Item 19) An imaging unit that captures the image; An information processing device according to any one of items 1 to 18. An imaging device comprising:

[0367] (Item 20) 20. The imaging device according to item 19, wherein the imaging unit captures moving images or still images.

[0368] (Item 21) an acquiring means for acquiring from an image file one or more pieces of region information describing one or more pieces of shape information for identifying an object included in the image, association information for associating the image with the region information, combination type information combining the region information, and combination region information for identifying a region combined according to the combination type information; a processing means for processing the image based on the one or more pieces of region information, the association information, the combination type information, and the combination region information; An information processing device comprising:

[0369] (Item 22) An information processing method performed by an information processing device, a first generation step in which a first generation means of the information processing device generates one or more pieces of region information describing one or more pieces of shape information for identifying an object included in an image; a second generation step in which a second generation means of the information processing device generates association information that associates the image with the region information; a third generation step in which a third generation means of the information processing device generates combination type information combining the area information; a fourth generation step in which a fourth generation means of the information processing device generates combined area information for identifying combined areas according to the combination type information; a storing step in which a storing means of the information processing device stores the one or more pieces of area information, the association information, the combination type information, and the combination area information in an image file; An information processing method comprising:

[0370] (Item 23) 23. The information processing method according to item 22, wherein the combined region information is information that can be identified separately from the region information.

[0371] (Item 24) 23. The information processing method according to item 22, wherein the combined region information is information that groups and identifies the one or more pieces of region information.

[0372] (Item 25) 24. The information processing method according to item 22 or 23, wherein the combined region information is information that derives and identifies the one or more region information as a derived region.

[0373] (Item 26) An information processing method described in any one of items 22 to 25, characterized in that the combination type information is information that identifies an area obtained by performing a logical OR of each area indicated by two or more area information as a combination area.

[0374] (Item 27) An information processing method described in any one of items 22 to 25, characterized in that the combination type information is information that identifies an area obtained by performing a logical negation of an area indicated by one or more area information as a combination area.

[0375] (Item 28) An information processing method described in any one of items 22 to 25, characterized in that the combination type information is information that identifies an area obtained by performing a negative logical OR on areas indicated by two or more area information as a combination area.

[0376] (Item 29) An information processing method described in any one of items 22 to 25, characterized in that the combination type information is information that identifies an area obtained by performing a logical operation on an area indicated by the one or more area information as a combination area.

[0377] (Item 30) An information processing method described in any one of items 22 to 25, characterized in that the combination type information is information that can identify the inclusion relationship of areas indicated by two or more area information, and is information that identifies the relationship between each area.

[0378] (Item 31) The information processing method described in any one of items 22 to 25, characterized in that the combination type information is information that can identify the order of the foreground and background relationships of areas indicated by two or more area information, and is information that identifies the relationship between each area.

[0379] (Item 32) The information processing method described in any one of items 22 to 25, characterized in that the combination type information is information that enables identification that, in two or more area information, a subject area indicated by one area information is an area hidden by a subject area indicated by another area information, and is information that identifies the relationship between each area.

[0380] (Item 33) The information processing method described in any one of items 22 to 25, characterized in that the combination type information is information that enables identification of a contact relationship between a subject area indicated by one area information and a subject area indicated by another area information in two or more area information, and is information that identifies the relationship between each area.

[0381] (Item 34) 34. The information processing method according to any one of items 22 to 33, wherein the region information and the combined region information are information to which annotation information can be individually associated.

[0382] (Item 35) An information processing method described in any one of items 22 to 34, characterized in that the area information of the area combined according to the combination type information is information that allows separately combined combination area information to be specified as area information.

[0383] (Item 36) An information processing method described in any one of items 22 to 35, characterized in that the image file is configured to store in the file as a region combining region items using two CombinedRegionEntityGroups as a file data structure.

[0384] (Item 37) An information processing method performed by an information processing device, an acquisition step in which an acquisition means of the information processing device acquires, from an image file, one or more pieces of region information describing one or more pieces of shape information for identifying an object included in the image, association information for associating the image with the region information, combination type information combining the region information, and combination region information for identifying a region combined according to the combination type information; a processing step in which a processing means of the information processing device processes the image based on the one or more pieces of area information, the association information, the combination type information, and the combination area information; An information processing method comprising:

[0385] (Item 38) A computer program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 18 and 21.

[0386] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0387] 100: Image file storage device 101: CPU 102: ROM 103: RAM 104: Imaging unit 105: Image processing unit 106: Display unit 107: Operation input unit 108: Communication unit 109: System bus 110: Non-volatile memory 111: Encoding / decoding unit 112: Metadata processing unit 113: Generation unit 114: Recognition processing unit

Claims

1. a first generating means for generating one or more pieces of region information describing one or more pieces of shape information for identifying the shape of a region included in an image; a second generating means for generating association information that associates the image with the region information; a third generating means for generating combination type information indicating a type of combination of a plurality of regions; a fourth generating means for generating combined area information for identifying each of the plurality of areas to be combined in accordance with the combination type information; a storage means for storing, in an image file, area information for each of the plurality of areas, the association information corresponding to the plurality of areas, the combination type information, and the combination area information; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the combined region information is information that can be identified separately from the region information of each of the plurality of regions.

3. 2. The information processing apparatus according to claim 1, wherein the combined area information is information for grouping and identifying area information of the plurality of areas.

4. 2. The information processing apparatus according to claim 1, wherein the combined area information is information that derives and identifies area information of each of the plurality of areas as a derived area.

5. 2. The information processing apparatus according to claim 1, wherein the combination type information is information for identifying an area obtained by performing a logical sum of the plurality of areas as a combination area.

6. 2. The information processing apparatus according to claim 1, wherein the combination type information is information for identifying an area obtained by performing a logical negation on one or more areas as a combination area.

7. 2. The information processing apparatus according to claim 1, wherein the combination type information is information for identifying an area obtained by performing a NOR on the plurality of areas as a combination area.

8. 2. The information processing apparatus according to claim 1, wherein the combination type information is information for identifying an area obtained by performing a logical operation on one or more areas as a combination area.

9. 9. The information processing apparatus according to claim 8, wherein the logical operations are identified by parameters.

10. 2. The information processing apparatus according to claim 1, wherein the combination type information is information that enables identification of an inclusion relationship between the plurality of regions.

11. The information processing apparatus according to claim 1 , wherein the combination type information is information that enables identification of the order of the foreground and background of the plurality of regions in a chronological order.

12. 2. The information processing device according to claim 1, wherein the combination type information is information that enables identification that a first subject included in a first region is hidden by a second subject included in a second region in the plurality of regions.

13. 2. The information processing device according to claim 1, wherein the combination type information is information that enables identification of a contact relationship between a first region corresponding to a first subject and a second region corresponding to a second subject in the plurality of regions.

14. 2. The information processing apparatus according to claim 1, wherein the area information and the combined area information are pieces of information to which annotation information can be individually associated.

15. 2. The information processing apparatus according to claim 1, wherein the area information of the areas combined in accordance with the combination type information is information that allows separately combined area information to be designated as area information.

16. 2. The information processing apparatus according to claim 1, wherein the combination type information is stored as a parameter in the image file.

17. 2. The information processing apparatus according to claim 1, wherein the combination type information is stored as a property in the image file.

18. 2. The information processing apparatus according to claim 1, wherein the image file is configured to store in the file as a region in which region items are combined by two CombinedRegionEntityGroups as a file data structure.

19. The information processing device according to claim 1, wherein the image file is an image file conforming to the High Efficiency Image File Format (HEIF).

20. The information processing device according to claim 19, wherein the combination type information indicates an Entity Group type that indicates a group of a plurality of area items corresponding to the plurality of areas.

21. The information processing device according to claim 20, wherein the combination type information is a grouping type of EntityToGroupBox.

22. The information processing device described in Claim 1, characterized in that the combined area information is information that identifies a combination of area information of the multiple areas using the same data structure as the area information.

23. an imaging unit that captures the image; The information processing device according to claim 1 ; An imaging device comprising:

24. 24. The imaging device according to claim 23, wherein the imaging unit captures a moving image or a still image as the image.

25. an acquiring means for acquiring, from an image file, region information describing shape information for identifying the shape of each of a plurality of regions included in the image, association information for associating the image with the region information for each of the plurality of regions, combination type information indicating a type of combination of the plurality of regions, and combined region information for identifying each of the plurality of regions combined in accordance with the combination type information; a processing means for processing the image based on the region information, the association information, the combination type information, and the combined region information; An information processing device comprising:

26. An information processing device as described in Claim 25, characterized in that the combined area information is information that groups and identifies area information of the multiple areas.

27. ​​The information processing device described in Claim 25, characterized in that the combination type information is information that identifies an area obtained by performing a logical OR on the multiple areas as a combination area.

28. An information processing device as described in Claim 25, characterized in that the combination type information is information that can identify the inclusion relationship of the multiple areas.

29. The information processing device according to claim 25, characterized in that the area information and the combined area information are information to which annotation information can be individually associated.

30. An information processing method performed by an information processing device, a first generation step in which first generation means of the information processing device generates one or more pieces of region information describing one or more pieces of shape information for identifying the shape of a region included in an image; a second generation step in which second generation means of the information processing device generates association information that associates the image with the region information; a third generation step in which third generation means of the information processing device generates combination type information indicating a type of combination of a plurality of regions; a fourth generation step in which fourth generation means of the information processing device generates combined area information for identifying each of the plurality of areas to be combined in accordance with the combination type information; a storing step in which a storing means of the information processing device stores, in an image file, area information for each of the plurality of areas, the association information corresponding to the plurality of areas, the combination type information, and the combination area information; An information processing method comprising:

31. An information processing method performed by an information processing device, an acquisition step in which acquisition means of the information processing device acquires, from an image file, region information describing shape information for identifying the shape of each of a plurality of regions included in the image, association information associating the image with the region information for each of the plurality of regions, combination type information indicating a type of combination of the plurality of regions, and combined region information for identifying each of the plurality of regions combined in accordance with the combination type information; a processing step in which a processing means of the information processing device processes the image based on the region information, the association information, the combination type information, and the combined region information; An information processing method comprising:

32. 30. A computer program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 22 and 25 to 29.

Images