Display control apparatus, display control method, and storage medium

Abstract

To provide a technique for intuitively recognizing a correspondence relation between an area in an image and annotation information to the area.SOLUTION: An image output apparatus including an output control unit configured to superimpose and output area annotation information including area information indicating an area included in an image and annotation information indicating information related to the area on the image, the output control unit superimposing and outputting, when a plurality of pieces of area annotation information are superimposed on the image, the first area annotation information associated with a first area of a first depth in front of the second area annotation information associated with a second area of a second depth deeper than the first depth.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an image output device, and more particularly to a technique for outputting an image from an image file in which one or more pieces of image data are stored. [Background technology]

[0002] The Moving Pictures Experts Group (MPEG) is working on a standard for storing a single still image, multiple still images, or an image sequence (e.g., a burst of still images) in a single file. This standard, called HEIF (High Efficiency Image File Format), enables the exchange, editing, and display of images and image sequences.

[0003] Furthermore, HEIF has a structure for storing not only display images but also non-display images that assist in displaying the display images. For example, display images and non-display images can be stored as separate image items, and the two can be associated by setting ItemReference Type='auxl' and setting the item IDs of the non-display image and display image. A depth map image is a non-display image that assists in outputting a display image. A depth map is information for expressing depth in three-dimensional space, and a depth map image is generally an image that expresses the distance from the camera to an object using grayscale shading. Using a depth map image makes it possible to generate three-dimensional images, refocused images, and the like. Patent Document 1 describes a method for generating a refocused image.

[0004] MPEG is currently considering extensions to the HEIF standard, including standardization of Region items, a metadata structure for specifying regions within an image. Region items are stored in HEIF files as metadata items that specify regions within an image, and by associating them with images, specific regions within the image can be identified. Furthermore, the structure allows annotations to be added to Region items. Currently under consideration are methods for adding annotations: storing them as item properties associated with the Region item; associating another image with the Region item; or associating metadata defined outside HEIF. By using user-defined properties identified by udes as item properties, it is possible to associate arbitrary tag information, names, and description information with regions within an image. Hereafter, annotated Region items are referred to as region annotation information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-61743 Summary of the Invention [Problem to be solved by the invention]

[0006] Possible methods for superimposing multiple pieces of area annotation information associated with a single display image on the display image include, for example, superimposing the pieces of area annotation information in the order in which the information is stored in the image file, or superimposing the pieces of area annotation information in the order of their horizontal or vertical coordinates.

[0007] However, when a large number of area annotation information pieces are associated with one display image or when there is overlap between the areas of the area annotation information pieces, if all the area annotation information pieces are superimposed uniformly, it may become difficult to intuitively recognize the correspondence between the objects indicated by the areas and the annotation information pieces.

[0008] The present invention has been made in view of the above-mentioned problems, and has as its object to provide a technique for intuitively recognizing the correspondence between an area in an image and annotation information for that area. [Means for solving the problem]

[0009] In order to achieve the above object, an image output device according to one aspect of the present invention comprises: an output control means for superimposing, on the image, area annotation information including area information representing an area included in the image and annotation information indicating information related to the area; The output control means is characterized in that, when superimposing multiple pieces of area annotation information on the image, it superimposes first area annotation information associated with a first area at a first depth in front of second area annotation information associated with a second area at a second depth greater than the first depth. [Effects of the Invention]

[0010] According to the present invention, it becomes possible to intuitively recognize the correspondence between an area in an image and annotation information for that area. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of an image output apparatus according to an embodiment. [Figure 2] FIG. 1 is a functional configuration diagram of an image output device according to a first embodiment. [Figure 3] 5 is a flowchart showing the procedure of processing performed by a metadata analysis unit of the image output device according to the first embodiment. [Figure 4] FIG. 1 is a diagram showing an example of a HEIF file structure. [Figure 5] FIG. 10 is a diagram showing an example of a display image and a depth map image. [Figure 6] FIG. 10 is a diagram showing an example of an area frame superimposed diagram and a depth table. [Figure 7] 10 is a flowchart showing the procedure of a process for superimposing a display image and area annotation information, which is performed by the image output device according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing an example of an output image in which area annotation information is superimposed on a display image. [Figure 9] 10 is a flowchart showing the procedure of a process for superimposing a display image and area annotation information, which is performed by an image output device according to a second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of an output image when the depth is specified using a scroll bar. [Figure 11] FIG. 10 is a diagram showing an example of an output image when area annotation information is specified using a check box. [Figure 12] FIG. 10 is a functional configuration diagram of an image output device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] [Embodiment 1] In the first embodiment, an example will be described in which a display image stored in a HEIF file and all area annotation information associated with the display image are superimposed and output.

[0014] <Hardware configuration> 1 is a diagram showing an example of the hardware configuration of an image output device 100 according to this embodiment. The image output device 100 is a device equipped with an image output function, such as a PC (Personal Computer), smartphone, tablet, printer, etc. In this embodiment, a description will be given assuming a form in which an image is presented to a user by an image display application installed on a PC.

[0015] The image output device 100 includes a system bus 101, a CPU 102, a ROM 103, a RAM 104, a communication interface 105, an input device 106, an output device 107, and a storage device 108. CPU stands for Central Processing Unit, ROM stands for Read Only Memory, and RAM stands for Random Access Memory.

[0016] The system bus 101 connects the various components and serves as a transmission path for various data. The CPU 102 comprehensively controls the various hardware components and controls the image output device 100. The ROM 103 stores control programs executed by the CPU 102. The RAM 104 functions as the CPU 102's main memory, work area, etc., and temporarily stores programs and data. The communication interface 105 is an interface for sending and receiving communication packets via a network, such as a wireless LAN interface, a wired LAN interface, or a public mobile communication interface, but other interfaces are also possible. The input device 106 is an interface through which a user selects an image file or inputs image output settings to the image output device 100, such as a keyboard, mouse, touchpad, touch panel, or buttons. The output device 107 is a display that presents images and displays information input by the input device 106, processed results, etc. The input device 106 and output device 107 may be integrated, such as a touch panel. The storage device 108 is a storage device for storing image files, etc. Examples of such devices include storage devices such as HDDs (Hard Disk Drives) and SSDs (Solid State Drives), and auxiliary storage devices using USBs (Universal Serial Buses) and SDs (Secure Digital).The communication interface 105, input device 106, output device 107, storage device 108, etc. do not have to be built into the image output device 100. In this case, the image output device 100 controls the output of processed information to a separate output device 107 for display.

[0017] <Functional configuration> 2 is a diagram showing an example of the functional configuration of the image output device 100 according to this embodiment. The image output device 100 includes an image file input unit 201, a metadata analysis unit 202, and an image output control unit 203.

[0018] An HEIF file containing at least one image data item is input to the image file input unit 201. The HEIF file may be input from a camera, smartphone, or the like via the communication interface 105, or may be input from a storage device 108 storing the HEIF file, such as a USB flash memory or an SD card.

[0019] The metadata analysis unit 202 analyzes the metadata (MetaBox) of the input HEIF file. HEIF is a storage format that has been expanded based on tools defined in the ISO Base Media File Format (ISOBMFF). HEIF is currently being standardized under the name "Image File Format" in ISO / IEC 23008-12 (Part 12). HEIF also defines a normative structure that includes metadata, defines a method for associating metadata with images, and defines the structure of specific metadata formats.

[0020] Metadata analysis is the process of extracting and analyzing attribute information for image data stored in a HEIF file. Image data attribute information includes, for example, first-priority image information specified in PrimaryItemBox and thumbnail image information specified in ItemReferenceBox with Type='thmb'. Other information includes display / non-display image information indicated by Flags in the ItemInfoEntryBox of an ItemInfoBox, and auxiliary image information for the display image specified with Type='auxl' in ItemReferenceBox. Auxiliary images for the display image include alpha channel images that represent pixel opacity and depth map images that represent pixel depth. Details of the metadata analysis process required for superimposing the display image and region annotation information will be described later with reference to Figure 3.

[0021] The image output control unit 203 superimposes the area annotation information on the display image and outputs the image to the output device 107. Here, the area annotation information according to this embodiment includes area information representing an area included in the display image and annotation information indicating information related to the area. Here, the area information is, for example, a rectangular area frame surrounding an object, such as area frame 8011 in FIG. 8 (described later). The annotation information is information indicating the attributes of an object within the area (for example, the type of object), such as annotation characters that indicate an annotation for the area, such as annotation information 801 in FIG. 8 (described later). Details of the process of superimposing the display image and the area annotation information will be described later with reference to FIG. 7.

[0022] <Analysis processing> Next, the processing procedure performed by the metadata analysis unit 202 of the image output device 100 according to this embodiment will be described with reference to the flowchart in Fig. 3. Fig. 4 shows an example of metadata 401 stored in an HEIF file 400, and the following description will be given with reference to the processing flow in Fig. 3 and Fig. 4.

[0023] In S301, the metadata analysis unit 202 analyzes the PrimaryItemBox 402 and obtains the first priority image item ID '1' that will be the display image. The display image does not necessarily have to be the first priority image item, but in this embodiment, the first priority image item is the display image. Furthermore, as display image information, the item type 'hvc1' that matches the item ID '1' is obtained from the InfoEntry of the ItemInfoBox 403. This indicates that the image item is encoded using HEVC. Furthermore, the ItemLocationBox 404 is analyzed to determine the location in the MediaDataBox 409 where the data with item ID '1' is stored. In this case, it is stored in the HEVCImageData 410.

[0024] Furthermore, the item attributes belonging to the AssociationEntry that matches item ID '1' are identified from ItemPropertyAssociation 408 in ItemPropertiesBox 406. Here, property_index '1' and '2' are obtained. These are index 1 'hvcC' and index 2 'ispe:width=1920,height=1080' of the item attributes listed in ItemPropertyContainerBox 407. 'hvcC' indicates the HEVC encoding parameters, and 'ispe:width=1920,height=1080' indicates that the image size is 1920 pixels wide and 1080 pixels high.

[0025] In S302, the metadata analysis unit 202 checks whether or not there is region annotation information. The region annotation information is an InfoEntry of item type 'rgan' in ItemInfoBox 403, and in this case corresponds to the seven items with item IDs '3' to '9'. If there is no region annotation information, the processing flow ends.

[0026] In S303, the metadata analysis unit 202 checks whether or not there is a depth map image. The depth map image is an ItemReference of referenceType 'auxl' in ItemReferenceBox 405, and here, the item ID '2' becomes the depth map image item ID based on from_item_ID.

[0027] An example of a display image and a depth map image is shown in Fig. 5. In contrast to a display image 501, a depth map image 502 is an image in which the depth is expressed in grayscale. Here, areas in the back where the depth is large are expressed in black, and areas in the front where the depth is small are expressed in white, but the expressions may be reversed.

[0028] As with the display image, the depth map image also obtains coding information from the InfoEntry in the ItemInfoBox 403 , the location of the image data in the MediaDataBox 409 from the ItemLocationBox 404 , and item attributes from the ItemPropertiesBox 406 .

[0029] In this embodiment, the processing flow ends if there is no depth map image. However, a generating unit (not shown) for generating a depth map image may be further provided to dynamically generate the depth map image. For example, if the displayed image is a stereo image, a method of generating the image using binocular parallax is common. In recent years, AI technology has made it possible to generate a depth map even for a monocular image. Various known technologies can be applied to generate the depth map image.

[0030] In S304, the metadata analysis unit 202 analyzes the depth map. In analyzing the depth map, the depth of each pixel in the display image is obtained as a numerical value. If the depth map image is encoded, it is decoded to obtain depth information from the luminance of each pixel.

[0031] In S305 and S306, the metadata analysis unit 202 determines the depth for all region annotation information. The region annotation information is obtained from the storage location RegionItemData411 in the MediaDataBox409 indicated by the ItemLocationBox404. The region annotation information includes information such as the size of the mapping image, the shape of the region, the coordinates in the mapping image size, and the size of the region.

[0032] The size of the mapping image does not necessarily have to match the size of the display image. If it differs from the size of the display image, it will be converted to coordinates and area sizes that match the ratio of the mapping image size and used for the display image. Here, the shape of the area is defined as geometry_type, with '1' representing a rectangle. Other shapes such as a circle or star are also acceptable. Coordinates are expressed as the X and Y coordinates of the upper left corner of the rectangle.

[0033] The upper part of Figure 6 shows an example in which the area frames of seven area annotation information areas with item IDs '3' to '9' are superimposed on the displayed image. 601 to 607 correspond to item IDs '3' to '9', respectively.

[0034] In S306, the metadata analysis unit 202 acquires the depths analyzed in S304 for the area portions in the display image extracted from the coordinates and area sizes of the area annotation information, and determines a representative depth from among them.

[0035] One method for determining the representative depth is to recognize the object (e.g., tree, person, building, car, etc.) that the area represents through image analysis of the area, and then take the average depth for the pixel portion of the object. As an example, the bottom of Figure 6 shows a depth table 608 that obtains the average depth of the object's area pixels and records the item ID, name, and depth corresponding to the figure number. The smaller the depth value, the closer the area is to the front, and the larger the value, the further back the area is.

[0036] In the example of Figure 6, the chart number 601 has an item ID of '3', a name of "Tree", and a depth of 23. The chart number 602 has an item ID of '4', a name of "Man", and a depth of 198. The chart number 603 has an item ID of '5', a name of "Man", and a depth of 48. The chart number 604 has an item ID of '6', a name of "Man", and a depth of 10. The chart number 605 has an item ID of '7', a name of "Building", and a depth of 201. The chart number 606 has an item ID of '8', a name of "Tree", and a depth of 138. The chart number 607 has an item ID of '9', a name of "Car", and a depth of 173.

[0037] Other depth determination methods include using the most frequent depth, minimum depth, maximum depth, or center pixel depth of the pixel portion of the object, or using the average depth, most frequent depth, minimum depth, maximum depth, or center pixel depth for all pixels within the region of the region annotation information without object recognition.

[0038] That is, the average value, mode, minimum value, or maximum value of the depth for any multiple pixels within a region (for example, a pixel portion constituting an object) may be determined as the depth of the region. Also, the average value, mode, minimum value, or maximum value of the depth for all pixels within a region may be determined as the depth of the region. Alternatively, the depth for all pixels within a region or the central pixel of any multiple pixels within a region (for example, a pixel portion constituting an object) may be determined as the depth of the region. The central pixel may be the pixel at the center of gravity.

[0039] In this manner, the metadata analyzer 202 analyzes the depth map information using a depth map image corresponding to the display image and determines the depth of regions included in the display image based on the depth map information.

[0040] This completes the series of processes shown in FIG.

[0041] <Superimposition processing> 7 shows a flow of processing for superimposing a display image and area annotation information performed by the image output device according to this embodiment. In S701, the image output control unit 203 arranges the display image 501. In S702, the image output control unit 203 sorts the area annotation information by depth. For example, when sorting in descending order in the depth table 608 shown at the bottom of FIG. 6, the order is item ID '7' (depth 201), '4' (depth 198), '9' (depth 173), '8' (depth 138), '5' (depth 48), '3' (depth 23), and '6' (depth 10).

[0042] In steps S703 to S705, the image output control unit 203 processes all the region annotation information in the order sorted in step S702. The depths are sorted in descending order, and an example will be described in which the region annotation information is processed starting from the deepest.

[0043] In S704, the image output control unit 203 superimposes the frame of the region of the region annotation information (for example, if the subject of the region is a person, the frame of a rectangular region surrounding the person). Here, if there is another region already superimposed and there is an overlap between the other superimposed region and the region to be superimposed, the frame and / or annotation information of the other superimposed region within the region to be superimposed is erased. In the example of FIG. 8 described later, for example, part of the region frame 8011 is erased by the region frame 8014. In other words, the region frame 811 and / or annotation information 803 of an object with a greater depth are not displayed within the display range of the region frame 8014 and / or annotation information 804 of the object in FIG. 8. In this way, the region annotation information of a region with a greater depth (region at the back) is not displayed within the display range of the region annotation information of a region with a smaller depth (region at the front).

[0044] Note that annotation information may be re-superimposed on non-overlapping portions. The thickness of the region frame may also be changed according to the depth. For example, the region frame may be superimposed so that the region frame is thinner for a deeper region with a greater depth and thicker for a closer region with a smaller depth. Furthermore, the color of the region frame may be changed according to the depth. For example, the region frame may be superimposed so that the color of the region frame is lighter for a deeper region with a greater depth and darker for a closer region with a smaller depth. Superimposing the region frame with different thicknesses and display intensities in this way makes it easier to recognize information in the foreground.

[0045] In S705, the image output control unit 203 superimposes annotation information (for example, an annotation string explaining the characteristics of the region) on the region annotation information. The annotation information obtains the item attributes defined in the ItemPropertyContainerBox 407 from the property_index associated with the item IDs '3' to '9' in the ItemPropertyAssociation 408 in FIG. 4 .

[0046] Here, the property type 'udes' defined as an item attribute with property_index '3' to '6' is identified as the UserDescription property. Language information is stored in lang, and the name describes the associated item's name in a human-readable format. In addition to these, a description of the item may be written in a human-readable format as description. User-defined tag information related to the item may also be specified as tags, separated by commas.

[0047] Here, English is specified as the language information for property_index '3' to '6', and 'Man', 'Tree', 'Building', and 'Car' are defined as the names of the items.

[0048] An example of an output image in which the display image and area annotation information are superimposed is shown in Fig. 8. Annotation information 801 to 807, which are respectively associated with the areas (601 to 607) shown in Fig. 6, are superimposed.

[0049] Annotation information 801 corresponding to region 601 is the character string "Tree" displayed in a speech bubble. Annotation information 802 corresponding to region 602 is the character string "Man" displayed in a speech bubble. Annotation information 803 corresponding to region 603 is the character string "Man" displayed in a speech bubble. Annotation information 804 corresponding to region 604 is the character string "Man" displayed in a speech bubble. Annotation information 805 corresponding to region 605 is the character string "Building" displayed in a speech bubble. Annotation information 806 corresponding to region 606 is the character string "Tree" displayed in a speech bubble. Annotation information 807 corresponding to region 607 is the character string "Car" displayed in a speech bubble.

[0050] Here, the item name defined by name in the item attributes is output as annotation information, but it is also possible to output an item description defined by description or tag information defined by tags.

[0051] Also, although the annotation information is output as a speech bubble here, it may be superimposed anywhere within the region. The superimposition method for annotation information also changes depending on the depth. For example, the thickness of the frame of the annotation information speech bubble is changed depending on the depth. In Fig. 8, for example, the speech bubbles of annotation information 801 and 804, which are located at a small depth (located on the foreground), have thick frames, while the speech bubbles of annotation information 802 and 805, which are located at a large depth (located on the background), have thin frames.

[0052] Furthermore, the character size of the annotation information is made smaller for annotation information in areas further back with greater depth, and the character size of the annotation information is made larger for annotation information in areas closer to the center with less depth. In other words, the size of the annotation characters in areas closer to the center is displayed larger than the size of the annotation characters in areas further back.

[0053] In the example of FIG. 8, annotations 801 and 804 with a small depth (located closer to the viewer) have large character sizes, while annotations 802 and 805 with a large depth (located closer to the viewer) have small character sizes.

[0054] Like the frame, the color of the text may also be changed according to the depth. For example, the display density of the annotation text in the foreground area may be greater than the display density of the annotation text in the background area. In other words, the text color may be darker for annotation information with a smaller depth (located closer to the foreground), and lighter for annotation information with a larger depth (located further back).

[0055] This completes the series of processes shown in FIG.

[0056] As described above, in this embodiment, when multiple pieces of area annotation information associated with each area of ​​a single display image are superimposed on the display image and output, the area annotation information corresponding to an area with a small depth is superimposed in front of the area annotation information corresponding to an area with a large depth.

[0057] This makes it easy to intuitively recognize the correspondence between an area (for example, a rectangular area surrounding an object) and annotation information for that area.

[0058] [Variations] When multiple pieces of area annotation information are superimposed on an image, the brightness of the area annotation information in the nearer area (area with a smaller depth) may be displayed higher than the brightness of the area annotation information in the farther area (area with a larger depth).Furthermore, the brightness of at least one of the rectangular frame (area information) surrounding the object and the annotation text (annotation information), rather than the brightness of the entire area annotation information, may be configured to be higher as the area becomes closer to the object (area with a smaller depth).

[0059] Similarly, when multiple pieces of area annotation information are superimposed on an image, the saturation of the area annotation information in the nearer area (area with a smaller depth) may be displayed higher than the saturation of the area annotation information in the farther area (area with a larger depth).Furthermore, the saturation of at least one of the rectangular frame (area information) surrounding the object and the annotation text (annotation information), rather than the saturation of the entire area annotation information, may be configured to be higher the closer the area (area with a smaller depth).

[0060] [Embodiment 2] In the second embodiment, an example will be described in which region annotation information selected from region annotation information associated with a display image in accordance with a user's specification is superimposed on a display image stored in an HEIF file and output.

[0061] The hardware configuration in this embodiment is the same as that in FIG. 1 described in the first embodiment, and therefore a description thereof will be omitted.

[0062] <Functional configuration> 12 is a diagram showing an example of the functional configuration of an image output device 1200 in this embodiment. The image output device 1200 includes an image file input unit 201, a metadata analysis unit 202, a designation unit 1201, and an image output control unit 203. The image output device 1200 further includes a designation unit 1201 in addition to the configuration of the image output device 100 shown in the first embodiment. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0063] The designation unit 1201 designates region annotation information to be superimposed on the region annotation information associated with the display image. The designation unit 1201 accepts designation of region annotation information to be output and superimposed based on parameter designation by the user. Examples of parameters designated by the user include depth information, coordinate information, and item name and tag information as item attributes. Examples of superimposing the display image and region annotation information when each parameter is designated will be described later.

[0064] <Analysis processing> The processing flow of the metadata analysis unit 202 in this embodiment is the same as that shown in FIG. 3 described in the first embodiment, and therefore a description thereof will be omitted.

[0065] <Superimposition processing> 9 shows an example of the flow of the process of superimposing the display image and the area annotation information, which is performed by the image output device 1200 in this embodiment. Here, the description will be made on the assumption that the depth and depth width are specified as the output area annotation information specification.

[0066] In S901, the designation unit 1201 acquires the depth and depth range input by the user as the designated output range via the input device 106. The depth and depth range are designated via an input dialog, scroll bar, or the like from an input device such as a keyboard, mouse, or touch panel.

[0067] S902 is similar to S701, S903 is similar to S702, and S904 is similar to S703, so the explanation will be omitted.

[0068] In S905, the image output control unit 203 determines whether the depth of the area annotation information being processed is within the specified output range, and if it is within the specified output range, proceeds to the area frame placement process of S906. The processes of S906 and S907 are the same as S704 and S705, respectively, so the background names are omitted. On the other hand, if the depth is outside the specified output range, the area annotation information being processed is skipped without being superimposed. This completes the series of processes in Figure 9.

[0069] FIG. 10 shows an example of an output image in which region annotation information is superimposed on a display image when the user specifies the depth and depth width using the scroll bar.

[0070] Output image 1000 shows the image when the user specifies the depth by manipulating scroll bar 1002 with mouse cursor 1001. Here, the depth width is fixed at 10, and the depth is determined by the position of the scroll bar. Auxiliary dialog 1003 displays the output range of depth at the current scroll bar position. The output range here is a depth range from 195 to 200. According to the depth table 608 of the area annotation information shown in Figure 6, the area annotation information of item ID '4' (name = 'Man') and item ID '7' (name = 'Building'), which are included in the depth range, is superimposed.

[0071] Output image 1004 shows the image when the user uses mouse cursor 1001 to move scroll bar 1002 downward from the state it was in when output image 1000 was displayed. Here, auxiliary dialog 1005 displays the output range of depth at the current scroll bar position. The output range here is a depth range from 15 to 25, and area annotation information for item ID '3' (name='Tree') included in the depth range is superimposed according to depth table 608 of area annotation information shown in Figure 6.

[0072] Furthermore, a filter process, such as a Gaussian filter, may be applied to the displayed image based on the depth specified by the user. In this case, the same filter process may be applied to the region annotation information. That is, based on the depth (and / or depth range) specified, the region and annotation information corresponding to the specified depth may be filtered and displayed.

[0073] Furthermore, the parameter used by the user to specify region annotation information is not limited to depth. For example, the item name of the region annotation information may be extracted and the user may specify it. FIG. 11 shows an example of an output image in which region annotation information is superimposed on a display image when the item name of the region annotation information is specified using a checkbox. The user can specify the region annotation information to be output by checking one or more of checkboxes 1102 to 1105 with mouse cursor 1101. The item name of the region annotation information is written alongside each checkbox. In this example, checkbox 1103, which also contains 'Man', is checked. Output image 1100 is a display example when checkbox 1103 is checked. In accordance with region annotation information depth table 608 shown in FIG. 6, region annotation information for item IDs '4', '5', and '6' with the item name 'Man' is superimposed. Furthermore, radio buttons may be used instead of checkboxes, or the user may specify the item by entering a character string via an input dialog. Furthermore, tag names (tags) may be used instead of item names, or a character string may be searched for in the description (description), and the region annotation information that is found may be superimposed.

[0074] Alternatively, the user may specify coordinate information and superimpose area annotation information within the range of the coordinate information. The coordinate information may be specified by tapping with a mouse or a touch panel, or may be input using an input dialog.

[0075] As described above, in this embodiment, region annotation information selected from region annotation information associated with the display image in accordance with a user's specification is superimposed on the display image and output.

[0076] For example, the area annotation information to be displayed is specified based on the specification of depth and / or depth width, the specification of coordinate information in the image, the specification of a character string included in annotation characters, or a combination of these.

[0077] This makes it possible to display a desired type of information even when there is a large amount of area annotation information, thereby improving the visibility for the user.

[0078] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0079] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0080] 100: Image output device, 201: Image file input unit, 202: Metadata analysis unit, 203: Image output control unit, 501: Display image, 502: Depth map image, 601-607: Region frame of region annotation information, 608: Depth table, 801-807: Annotation information of region annotation information, 1000, 1004, 1100: Output image

Claims

[Claim 1] An image output device, an output control means for superimposing, on the image, area annotation information including area information representing an area included in the image and annotation information indicating information related to the area; An image output device characterized in that, when superimposing multiple area annotation information on the image, the output control means superimposes first area annotation information associated with a first area at a first depth in front of second area annotation information associated with a second area at a second depth greater than the first depth.

Images