Information processing device, information processing method, and program

Abstract

To provide annotation information relating to a subregion within an image in a file different from the image data used for distribution of an image.SOLUTION: An information processor generates a play list including a network address referring to for acquisition of an image, an area information stipulating a spatial subregion in an image and annotation information which is the information to be displayed in connection with the subregion, and transmits the generated play list.SELECTED DRAWING: Figure 4a

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There are systems that distribute streaming content consisting of audio data, video data, etc. in real time, and allow users to view such content via their own terminal devices. In such systems, terminal devices have various capabilities, and content is played in various environments, so technology is required to adapt to the content playback environment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-20654 Summary of the Invention [Problem to be solved by the invention]

[0004] In a playlist, which is a file different from image data and is distributed for the purpose of distributing an arbitrary image, annotation information that is displayed in association with a partial area of ​​the video has not been added.

[0005] The present invention aims to add annotation information associated with a partial area within an image in a file that is different from the image data used for image distribution. [Means for solving the problem]

[0006] In order to achieve the object of the present invention, for example, an information processing apparatus according to one embodiment has the following configuration: That is, a network address referenced to acquire an image, area information defining a spatial partial area in the image, and a display image associated with the partial area. canThe present invention further comprises a generating means for generating a playlist including annotation information, which is information on the content of the playlist, and a transmitting means for transmitting the playlist generated by the generating means. [Effects of the Invention]

[0007] It is possible to add annotation information associated with a partial area within an image in a file that is different from the image data used for image distribution. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a system including an information processing device according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the functional configuration of an information processing apparatus according to a first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a box configuration of an image file according to the first embodiment. [Figure 4a] FIG. 4 is a view showing a display example of annotation information set by the information processing device according to the first embodiment. [Figure 4b] FIG. 3 is a diagram showing the relationship between items set by the information processing device according to the first embodiment. [Figure 5] 10 is a flowchart showing an example of a playlist generation process according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a playlist generated by the information processing device according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a playlist generated by the information processing device according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an example of a playlist generated by the information processing device according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a playlist generated by the information processing device according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing an example of a playlist generated by the information processing device according to the first embodiment. [Figure 11] FIG. 2 is a diagram showing an example of a playlist generated by the information processing device according to the first embodiment. [Figure 12]FIG. 2 is a diagram showing an example of a playlist generated by the information processing device according to the first embodiment. [Figure 13] FIG. 2 is a diagram showing an example of a playlist generated by the information processing device according to the first embodiment. [Figure 14] FIG. 10 is a diagram showing an example of a playlist generated by the information processing device according to the second embodiment. [Figure 15] 11 is a flowchart showing an example of a display process performed by a receiving device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [Embodiment 1] Fig. 1 shows an example of a system including an information processing device according to this embodiment. The information processing device 100 according to this embodiment is a transmitting device that transmits image data (images) to a receiving device 110. In the system shown in Fig. 1, the information processing device 100 is communicably connected to the receiving device 110 via a network 120. Here, the number of information processing devices 100 and receiving devices 110 is not limited to one each, and there may be a plurality of each.

[0011] The information processing device 100 generates a playlist including network addresses to be referenced for acquiring images, and transmits the playlist together with the images to the receiving device 110. The information processing device 100 may be, for example, a camera, a video camera, a mobile terminal such as a smartphone, a PC (personal computer), or a cloud server, but is not limited to these as long as it is capable of executing the functions described below. Note that the image transmitted here may be a moving image (video), but for the sake of explanation, it will be referred to as a single still image below.

[0012] The receiving device 110 receives data from the information processing device 100. The receiving device 110 also has a function of playing / displaying content such as images, and may also accept input from a user. As the receiving device 110 according to this embodiment, any desired electronic device can be used, for example, a mobile terminal such as a smartphone, a PC, a television, or the like.

[0013] The network 120 may be any of various networks, such as the Internet / intranet, a local area network (LAN), or a wide area network (WAN). The wired communication interface may be an interface conforming to the Ethernet (registered trademark) standard, but other interfaces may also be used. The wireless communication interface may be an interface conforming to a wireless LAN standard conforming to the IEEE802.11 series of standards, or an interface conforming to a WAN such as 3G / 4G / LTE, Bluetooth (registered trademark), or other standards. The wireless connection may be an infrastructure network connection or an ad hoc network connection. The network 120 may also be a combination of a wired communication path and a wireless communication path. In other words, the network 120 may have any format as long as a connection is established between the information processing device 100 and the receiving device 110 and communication is performed.

[0014] In this embodiment, a standard called MPEG (Moving Picture Experts Group)-DASH (Dynamic Adaptive Streaming over HTTP) of ISO / IEC 23009-1 is used. Furthermore, the following description will be given assuming that each process, such as the generation of a playlist described below, is performed using this MPEG-DASH standard.

[0015] The following describes the MPEG (Moving Picture Experts Group)-DASH (Dynamic Adaptive Streaming over HTTP) standard. MPEG-DASH is a video distribution standard that enables dynamic changes to the acquired stream.

[0016] In MPEG-DASH, media data is divided into segments of a predetermined length, and URLs (Uniform Resource Locators) for obtaining the segments can be written in a file called a playlist. A receiving device first obtains this playlist and then uses the information written in the playlist to request and obtain the desired segment from the transmitting device. Furthermore, by writing URLs for multiple versions of segments with different bit rates, resolutions, etc. in the playlist, a receiving device can obtain the optimal version of a segment depending on its own capabilities, communication environment, etc. Furthermore, the file format of the segments is ISO Base Media File Format (hereinafter referred to as ISOBMFF).

[0017] The structure of ISOBMFF is broadly divided into a section that stores header information and a section that stores encoded data. The header information includes information indicating the size and timestamp of the encoded data stored in the segment, and the encoded data can include video, still images, and audio data. ISOBMFF has multiple extension standards depending on the type of encoded data to be stored, and one of these extension standards is the High Efficiency Image File Format (HEIF), standardized by MPEG. HEIF, known as the "Image File Format," is currently being standardized in ISO / IEC 23008-12 (Part 12) and defines specifications for storing still images and image sequences encoded with High Efficiency Video Coding (HEVC), a codec primarily used for video. In addition to media data such as video, ISOBMFF can also store metadata such as text and XML. This metadata can be stored both statically and dynamically. Metadata that contains information in a time series is called timed metadata, and a specific example is subtitle data.

[0018] 2 is a block diagram showing an example of the functional configuration of an information processing device according to this embodiment. The information processing device 100 includes an analysis unit 101, an extraction unit 102, a generation unit 103, a conversion unit 104, a storage unit 105, a generation unit 106, and a communication unit 107. The processing performed by each functional unit will be described in detail below with reference to FIGS. 3 to 7.

[0019] The analysis unit 101 analyzes the structure of a data file. The following description will be given assuming that the data file analyzed by the analysis unit 101 is in the HEIF file format. The extraction unit 102 extracts metadata and encoded data stored in the data file based on the analysis result of the data file by the analysis unit 101.

[0020] The generating unit 103 divides the metadata and encoded data extracted by the extracting unit 102 into time lengths suitable for communication or changes the bit rate as necessary, and generates segments storing the data. The converting unit 104 can convert the extracted encoded data into a different encoding format as necessary. The generating unit 103 may store the encoded data converted by the converting unit 104 in segments. The storing unit 105 stores the data generated by the generating unit 103.

[0021] Based on the analysis result of the data file, the generation unit 106 generates a playlist including network addresses that the receiving device 110 references to acquire the data stored in the storage unit 105. The playlist includes area information that defines partial areas on an image included in the data file, and annotation information that is information to be displayed in association with the partial areas.

[0022] Here, the network addresses included in the playlist are basically URIs (Uniform Resource Identifiers). The generation unit 106 may write a URL or an IP address of the Internet or a LAN as the network address, and the format is not particularly limited as long as it can describe the location of the data.

[0023] A partial region on an image can be set by any method. For example, the generation unit 106 may perform image analysis processing on an image input to the information processing device 100 and set a region that satisfies a predetermined condition as a partial region. For example, when a predetermined object is detected by image analysis, the generation unit 106 may set a bounding box representing the object as a partial region defined by region information. Furthermore, when a predetermined event is detected by context analysis, the generation unit 106 may set a region in which the predetermined event occurs as a partial region. Furthermore, the generation unit 106 may receive input from a user and set a region specified by the user as a partial region. The position, shape, and description of the partial region in the playlist are not particularly limited, and a detailed description thereof will be given later with reference to FIGS. 6 to 13.

[0024] As described above, annotation information is information displayed in association with a partial region. For example, annotation information is information associated with a partial region and superimposed on an image, as in annotation information 1 to 3 in FIG. 6 (described later). The way in which annotation information is displayed is not limited to being superimposed on an image as long as it is clear that the annotation information is associated with a partial region. For example, annotation information may be displayed in list form on a separate screen or in a separate frame on the video screen. Annotation information may be, for example, text information composed of characters or symbols, an image or video, or may include audio. Any desired form may be used as long as it can be displayed and played back in association with a partial region. Such annotation information may be information output as a result of image analysis, information input by a user, or information acquired from an external source. In this embodiment, such annotation information is stored and defined as metadata or encoded data in each box shown in FIG. 3. That is, for example, according to the information processing device of this embodiment, it is possible to generate a playlist that assigns partial regions and annotation information to video (for example, stored on a cloud) and transmit the playlist to a user who requires the annotation information. This makes it possible to display and present to the user annotation information about an object requiring attention, such as an object to be monitored that has been detected by a surveillance camera or an object that has become hidden.

[0025] Based on the analysis results by the analysis unit 101, the generation unit 106 includes in the playlist annotation information that is to be displayed in association with a partial area of ​​the still image data that constitutes the video data stored in the HEIF file. The structure of the HEIF file analyzed by the analysis unit 101 will be described below with reference to Fig. 3. Fig. 3 is a diagram showing an example of the structure of a data file (HEIF file) that stores annotation information and is to be analyzed by the information processing device 100.

[0026] The analysis unit 101 analyzes the nested boxes that make up the HEIF file (in this embodiment), allowing the extraction unit 102 to acquire each piece of information contained in the image file. In this embodiment, the boxes in the HEIF file are identified by four-character identifiers, and each box stores information according to its purpose. Hereinafter, each box will be represented by its assigned four-character identifier. In the example of Figure 3, the HEIF file contains the boxes meta301 and mdat302.

[0027] meta (MetaDataBox) 301 is a box for storing metadata, and includes the following boxes: hdlr, dinf, iloc 305, iinf 303, iref 304, pitm, iprp 306, ipma 307, and idat 308. meta 301 can store various information, such as the ID of each item, such as each image and audio file, information about the encoding of media data, or information about how to store it in a HEIF file.

[0028] Furthermore, item data can be stored in mdat 302, but may also be stored in idat 308 within meta 301. In the example of Figure 3, items 313 and 314 are stored in idat 308 within meta 301, and items 311 and 312 are stored in mdat 302. In this example, items of still image, video, or audio information are stored in mdat 302, and items stored in idat 308 are items indicating area information or annotation information. In this embodiment, data such as video or audio is stored in mdat 302, and annotation information and area information, which are relatively small in size, such as text data, are stored in idat 308.

[0029] hdlr (HandlerReferenceBox) stores information about the handler type for identifying the structure and format of the content included in meta.

[0030] iinf (ItemInformationBox) 303 stores information indicating the type of item and identifiers used to identify all stored items, including image items for images in the HEIF file. Item information includes the ID (item ID) of each item in the HEIF file, the item type indicating the type of item, and the name of the item. iinf 303 can also store item information when storing Exif data generated when capturing image data with a digital camera, or area information indicating a partial area in image data stored as an item.

[0031] iref(ItemReferenceBox) 304 is a box that stores association information between items, such as association information between a still image and Exif data, or between a still image and area information, and defines a reference type according to the association relationship between the items. For example, the type of association between items related to area information is defined as cdsc, which is intended to provide explanatory information for the referenced item. In this embodiment, the association information includes information indicating annotation information to be displayed in association with a partial area of ​​video data (images constituting the video data). The association information may also include information relating to the association between image data and Exif data.

[0032] iloc (ItemLocationBox) 305 stores information indicating the ID (i.e., identification information for each image) and storage location of each item, such as an image and its encoded data, in a HEIF file. In each process performed by the information processing device 100, information indicating the location of the data for an item defined in a HEIF file can be obtained by referencing iloc 305, which includes a construction method that indicates the storage location for each item. For example, when the reference type defined in iref 304 is cdsc, the construction method is generally defined as "1," indicating that the item is stored in idat 308. In the example of Figure 3, item 313 or item 314 stored in idat 308 is the item that stores region information.

[0033] ItemPropertyBox (iprp) 306 stores attribute information of an image in an image file. To this end, iprp 306 includes ipco and ipma boxes. Attribute information is information related to the display of an image, such as the image's width and height, and the number and bit length of color components. In the example of FIG. 3, iprp 306 stores five properties: Property 331, Property 332, Property 333, Property 334, and Property 335. In this example, Property 331 is codec initialization information for the encoded data, Properties 332 and 335 are information indicating the image size, and Properties 333 and 334 are annotation information associated with subregions of the image.

[0034] ItemPropertyAssociationBox (ipma) 307 stores information indicating the association between the information stored in ipco and the item ID. In the example of Fig. 3, Properties 331 and 332 are associated with item 311, Properties 331 and 335 are associated with item 312, Property 333 is associated with item 313, and Property 334 is associated with item 314. That is, codec initialization information and image size information are each associated with Items 311 and 312, which are image items, and annotation information is each associated with Items 313 and 314, which are area information items.

[0035] Next, the relationship between items and properties stored in a HEIF file having the configuration described with reference to FIG. 3 will be described with reference to FIGS. 4a and 4b. In FIG. 4a, Item 311 is a main image, and Items 313 and 314 indicated by dotted lines are area information indicating partial areas of the main image. Here, the main image refers to the entire image in which a partial area is set, and the sub-image refers to an image displayed as annotation information. Properties 333 and 334 are annotation information associated with Items 313 and 314, respectively, and are displayed as information linked to the partial areas in the example of FIG. 4a. Furthermore, Item 312 is a sub-image related to the area indicated by Item 314, and may be displayed as a set with Property 334, which is also annotation information assigned to Item 314.

[0036] Figure 4b shows the relationship between items and properties stored in a HEIF file, as indicated by iref 304 and ipma 307 in Figure 3. Here, there are two pieces of region information indicating partial regions, and the items are associated with each other by the reference type cdsc described in iref 304. Similarly, a subimage, which is an image item, is associated with a region information item in iref 304, and the reference type is eroi (encoded region of interest), indicating that it is encoded region of interest information. Also in Figure 4b, properties are indicated by rectangles with rounded corners, and items are indicated by rectangles.

[0037] 5 is a diagram showing an example of processing performed by the information processing device 100 according to this embodiment to analyze an input HEIF file and generate a playlist. In MPEG-DASH, a file equivalent to a playlist is called an MPD (Media Presentation Description).

[0038] In S501, the information processing device 100 acquires a HEIF file to be analyzed. Here, the information processing device 100 acquires the HEIF file from, for example, an imaging device (not shown). In S502, the analysis unit 101 analyzes the HEIF file and acquires item IDs, which are identifiers of individual items included in the file, and item types.

[0039] In S503, the analysis unit 101 refers to the ipma 307 and acquires the reference relationship including the reference type between items based on the item ID, and then in S504, the analysis unit 101 acquires the properties associated with each item.

[0040] In S505, the analysis unit 101 determines whether the item acquired in S502 contains area information indicating a partial area, and if so, proceeds to S506; if not, it determines that no annotation information exists and terminates the process.

[0041] In S506, the analysis unit 101 determines whether annotation information exists in the properties associated with at least one area information item. If annotation information exists, the process proceeds to S507; if not, the process ends assuming that annotation information does not exist.

[0042] In S507, the generation unit 103 generates segments for distribution. Here, if there are multiple items stored in the HEIF file, for example, the generation unit 103 generates one file for each image item of still images. In S508, the generation unit 106 generates a playlist based on the annotation information and ends the process.

[0043] Next, an example of a playlist generated by the generating unit 106 will be described with reference to Fig. 6. The generating unit 106 can generate playlists such as those shown in Figs. 6 to 14 below. Fig. 6 shows an example of a playlist according to this embodiment, and is a description example in which annotation information added to a partial area of ​​image data that is a still image can be arbitrarily acquired.

[0044] A playlist 600 shown in Fig. 6 shows a portion of an MPD, and a display example 610 displays the images, region information, and annotation information described in the playlist 600. The playlist 600 describes information for acquiring each segment of a main image 601, annotation information 602 to 604, and sub-image 605. In addition, region information 606 to 608 is described as attribute information of the annotation information 602 to 604, respectively.

[0045] In the example of FIG. 6, the generation unit 106 defines the region information using a URN such as "urn:mpeg:dash:rgon:2021," and indicates the region information using a numerical value or symbol written after "value=". In this way, the generation unit 106 can define various information, including region information, using a schema for interpreting the description and can write information for acquiring that schema. Here, the generation unit 106 can write the first value after "value=" as information defining the shape (type) of the partial region. For example, if the shape of the partial region is a point, a rectangle, or a circle, the generation unit 106 writes the first value after "value=" as "1," "2," or "3," respectively.

[0046] The generation unit 106 can describe the coordinates of the partial region following the shape of the partial region. Here, the number and meaning of the coordinate description differ depending on the shape of the partial region. For example, if the shape of the partial region is a point, it is sufficient to describe one parameter indicating the vertical and horizontal coordinates (XY coordinates) within the main image as coordinate information. In the example of Figure 6, the parameter "450,400" in the region information 606 indicates the coordinates of the X axis and Y axis with the upper left corner of the main image as the origin.

[0047] Region information 607 representing a rectangular subregion may include two parameters indicating the horizontal and vertical sizes of the rectangle, in addition to a parameter indicating the coordinate of the upper left corner of the rectangle. Region information 608 representing a circular subregion may include three parameters indicating the center coordinate of the edge and the radius. It is also possible to define region information that adds an angle to an ellipse by adding a rotation angle as a parameter. The shapes of the subregion are not limited to these, and any desired shape that can be represented by parameters can be used. Note that if multiple subregions have the same shape, the generation unit 106 may collectively describe these subregions in a single element.

[0048] Furthermore, the generation unit 106 sets the representation ID of the main image to "associationId" as attribute information of the annotation information representation to associate the main image with the annotation information. The type indicating the attribute of the annotation information is written in "associationType." Here, setting "associationType" to "cdsc" indicates that it is annotation information for the main image. Furthermore, since the sub-image is an image related to a partial region of the annotation information 603, "eroi" is set in "associationType" of the annotation information 603.

[0049] Fig. 7 is a diagram for explaining an example of a method for associating a sub-image with an annotation image, in which a playlist description different from that shown in Fig. 6 is performed. Fig. 7 shows an example of a description in which annotation information added to a partial region of an image is associated with another image.

[0050] In playlist 700, region information 701 and region information 702 indicate the same region. In the example of Fig. 6, the sub-images were indirectly associated with the region information by being associated with annotation information having the region information as attribute information, but in playlist 700, the region information is assigned as attribute information of the sub-images. In other words, the sub-images are directly associated with the region information.

[0051] 6 illustrates three examples of partial region shapes: a point, a rectangle, and a circle, but an example of a more complex shape using a polygon or a bit mask that defines a shape arbitrarily in pixel units will be described with reference to Fig. 8. Fig. 8 is an example of a description of a playlist generated by the generation unit 106, similar to Fig. 6.

[0052] In the example of FIG. 8, as shown in display example 810, playlist 800 defines one polygonal region (annotation 1) and one pixel designated region (region of arbitrary shape) (annotation 2) as partial regions within the main image. In playlist 800, descriptions can be basically made in the same way as in playlist 600. Here, annotation 1 indicates that the shape of the partial region is polygonal by setting the first value 801 of "value=" to "4," and the following value 802, "5," indicates the number of vertices of the polygon. The following value 803 is the coordinates of each of the five vertices, and a total of 10 parameters are described as the X and Y coordinates. Note that the generation unit 106 can define a straight line by setting the number of vertices to two. The generation unit 106 may also set a parameter indicating whether the coordinates of the first and last vertices are closed (connected by a line segment). If the parameter is closed, the resulting shape is a polygon, and if the parameter is not closed, the resulting shape is a broken line.

[0053] Annotation 2 also indicates that the shape of the partial region is arbitrary by setting the first value 804 of "value=" to "5." In this case, the following four values ​​805 are parameters that describe the area in which the partial region fits, namely, the upper-left coordinate and the horizontal and vertical dimensions of the arbitrary region. The following value 806 is a value referenced when pixel-by-pixel information represented by a bit mask is integrated to generate a reduced image. Here, "2" is written as a mask that reduces the image by combining two adjacent pixels into one pixel, as shown in pixel integration example 820. Generating such mask data makes it possible to reduce the amount of data to approximately one-quarter. This pixel integration method may be set arbitrarily. In pixel integration example 820, "2" is written as a value applied to both the vertical and horizontal number of pixels to be integrated, but separate values ​​may be written for each direction. In this case, the value may be written as a single parameter, such as "n × m," where n corresponds to the vertical direction and m corresponds to the horizontal direction, or as two parameters, such as "n" and "m." Also, in the playlist 800, “mask_data” set as the representation ID 808 of the mask data is written in the last value 807 of the region information parameter of annotation 2, thereby associating the region information of annotation 2 with the mask data.

[0054] In MPEG-DASH, in order to enable content of the same content to be acquired by dynamically switching the bit rate or resolution, streams with different bit rates or resolutions can be prepared in advance, and URLs at which these can be acquired can be described in the MPD. With this configuration, it is possible to use content with an appropriate bit rate or resolution depending on the communication bandwidth or the processing power of the client. However, in the examples described in Figures 6 to 8, it was assumed that the position and size of the region information were set using units such as pixels, so even if the content is the same, if the resolution is different, the coordinate information will change, and the coordinates from the description may differ.

[0055] Taking such a case into consideration, a processing example that enables scaling of area information to accommodate changes in video resolution will be described with reference to Fig. 9. Fig. 9 illustrates an example of a playlist that basically has the same description as Fig. 6.

[0056] As shown in display example 910, playlist 900 is a description of data in which an annotation image is associated with one partial region of main image 901. In the example of Fig. 9, three images with different resolutions are described as main image 901, and region information for two patterns 902 and 906 corresponding to scaling is also described.

[0057] FIG. 9 shows region information for a partial region whose shape is a point. The description after "value=" in region information 902 begins with a leading value (903) of "1" followed by two values ​​(904) of "2400, 1800" representing the resolution of the main image. This is followed by value 905, "450, 400," representing the position of the partial region. Here, the generator 106 can generate a playlist so that, if the resolution of the main image changes, the position of the partial region changes to a corresponding position through proportional calculation. Note that this main image value 904 ("2400, 1800" in the example of FIG. 9) is assumed to be one of the resolutions of the main image, but it may be a different size from any of the stored main images. Even in this case, it is possible to determine the position of the partial region at a position corresponding to the main image being used through proportional calculation. Even if the shape of the partial region is not a point, the center coordinates of a circle, each length, etc., can be scaled through proportional calculation in a similar manner.

[0058] 9, the description after "value=" in region information 906 begins with a leading value (907) of "1", followed by a value (908) of "19,22" that indicates the position as a percentage from the top left position of the main image, rather than pixel coordinates. That is, as shown in display example 910, the description is generated so that the partial region exists at a position that is 19% of the total in terms of X coordinate and 22% of the total in terms of Y coordinate. This value 908 indicates the relative position when the top left corner is "0,0" and the bottom right corner is "100,100".

[0059] In this example, three main images with different resolutions are prepared, and therefore different representation IDs are prepared for each of them. In the example of Figure 9, the representation ID of the main image corresponding to the "annotationID" value 909 of annotation 1 (annotation1_1) is shown, and here the three representation IDs are written side by side with spaces between them.

[0060] Next, an example of associating one piece of annotation information with a plurality of different partial regions will be described with reference to Fig. 10. Fig. 10 discloses an example of a playlist that basically has the same description as Fig. 6.

[0061] Playlist 1000 is a description of data in which the same annotation information is associated with multiple partial areas within a main image, as shown by annotation 1 and annotation 2 in display example 1010. In this example, annotation 1 is associated with three partial areas, rectangles 1, 4, and 6, and annotation 2 is associated with rectangles 2 and 3 and circle 5.

[0062] In the playlist 1000, the value at the beginning of "value=" in the area information written as attribute information of the annotation information (representing the shape of the partial area) is followed by a value 1001 indicating the number of corresponding partial areas. In the example of FIG. 10, "3" is written as the value 1001, and the following value 1002 is written as parameters indicating the positions and sizes of the three partial areas. In the example of FIG. 10, the partial areas are rectangular, so each partial area is written with four parameters, the X and Y coordinates and size, for a total of 12 values. This is equivalent to attribute information 1006 which also lists the attribute information of the three partial areas, and may be written in any way.

[0063] In addition, the shapes of the subregions associated with the same annotation information may be different. In the example of Figure 10, the shape of circle 5 is different from the shapes of rectangles 2 and 3 associated with annotation 3, and this is shown without being summarized as a list of values ​​1003 and 1005.

[0064] Furthermore, an example of combining and displaying a plurality of image data as a main image will be described with reference to Fig. 11. In a playlist 1100 in Fig. 11, four images 1 to 4 are laid out in a tiled pattern as in a display example 1110, and annotation information is associated with partial regions in the same manner as in the example of Fig. 6.

[0065] Here, the generation unit 106 can describe the main image 1101 using SRD (Spatial Relationship Description), a technology defined in MPEG-DASH for spatially arranging images or videos. Here, representation IDs of image1 to image4 are defined for images 1 to 4, respectively. Each image constituting the main image is arranged within the main image using descriptions similar to those for the partial areas in FIGS. 6 to 10. In annotation information 1 described at the bottom of playlist 1100, the partial area can be represented by coordinates with the upper left corner of the main image as the origin. Furthermore, by describing the representation ID of an image that has an area overlapping with a partial area among the images constituting the main image in the value 1102 of "associationID," it becomes easy to identify the image related to the partial area to which annotation information is added.

[0066] The images constituting the main image do not need to be the same size, and they do not need to be arranged in a tiled pattern as in Figure 11. That is, the generation unit 106 may generate a playlist so that images of various sizes are overlaid at any coordinates and displayed in an overlay. In this case, the origin for setting the partial regions can be the point obtained by combining the leftmost point of the leftmost image and the topmost point of the topmost image among the images constituting the main image, but any other desired point may also be used as the origin. With this configuration, it is possible to use a composite image such as a panoramic image as the main image, and associate annotation information with partial regions set on the image and display them.

[0067] An example of adding tag information to annotation information in order to improve searchability and convenience of information control and management will be described with reference to Fig. 12. The example of Fig. 12 discloses an example of a playlist that basically has the same description as Fig. 6.

[0068] In the playlist 1200, there are six partial regions, regions 1 to 6, to which annotation information is added on the main image, and a common tag is added to annotation information with the same attribute. In the example of Fig. 12, tag 1 (1201) "car" is defined for annotation information 1 and 2 as annotation information related to automobiles, and tag 2 (1202) "human" is set for annotation information 3 to 5 as annotation information related to people. Here, of annotation information 1 to 5, the attribute of annotation information 1 to 3 is text, annotation information 4 is video, and annotation information 5 is audio. Furthermore, annotation information 3 is added to both regions 2 and 5, and annotation information 4 and 5 (video and audio) are both added to the same region 3.

[0069] A display example 1210 is an example of displaying information described in the playlist 1200. Taking into consideration the possibility that the display may become cumbersome if all annotation information is displayed superimposed on the main image, for example, the generation unit 106 may generate a playlist so that only annotation information having a specific tag is displayed or is displayed in a different color.

[0070] With this configuration, it is possible to generate and transmit a playlist that includes a network address for acquiring an image, area information that defines a partial area in the image, and annotation information that is information to be displayed in association with the partial area. Therefore, it is possible to generate a playlist for displaying annotation information in a partial area of ​​an input video and transmit the playlist to a user who requires the annotation information.

[0071] [Embodiment 2] The information processing device according to the first embodiment generates a playlist including region information defining partial regions and annotation information using the generation unit 106. On the other hand, in the present embodiment, the region information and annotation information are acquired from an external source. The information processing device according to the present embodiment has the same functional configuration as that shown in FIG. 2 and is used in the same system as that shown in FIG. 1, so a duplicated description will be omitted.

[0072] FIG. 13 is a diagram showing an example of a playlist generated by the generation unit 106 according to this embodiment. In this example, one main image and two types of region information and annotation information are defined in the playlist 1300. The generation unit 106 can generate a playlist including a URI for accessing each piece of region information and annotation information. For example, the generation unit 106 can describe the region information and annotation information in XMP (Extensible Metadata Platform). In this case, the generation unit 106 may set a codec type, such as "rgan (region annotation)," that is intended to include the region information and annotation information.

[0073] XMP1 and 2 in the playlist 1300 can be obtained by accessing the URLs described in the playlist 1300, and each contains area information that defines a partial area within the main image and annotation information associated with that area. The basic format of XMP is XML (Extensible Markup Language), and it is desirable that information for obtaining a schema for interpreting the description also be described.

[0074] Furthermore, the generation unit 106 may store information for image analysis in the XMP file, rather than storing the region information and annotation information directly. That is, information necessary for acquiring the region information and annotation information through image analysis may include, for example, the URI of an image analysis service, information identifying a function used by the service, or parameters to be passed to an API provided by the image analysis service. This process enables the region information and annotation information to be acquired through image analysis processing to be stored, rather than being directly stored in the playlist. In this case, the generation unit 106 may also store information indicating the method, type, or algorithm of image analysis. For example, the generation unit 106 may store information identifying the image analysis to be performed, such as context analysis, such as suspicious behavior analysis in a surveillance camera, or object analysis, such as identifying animals, people, or vehicles. The objects to be analyzed here may be any object recognizable by common analysis processing, such as a human face or pupil, a person, an animal, a motorcycle, a license plate, or a lesion (in medical image diagnosis, etc.). Furthermore, it is not necessary to store information for performing image analysis on both the region information and the annotation information, and it is possible to store either the region information or the annotation information as is.

[0075] Furthermore, although the above example basically describes the process of generating a playlist for still images, the generation unit 106 may generate a playlist including area information and annotation information for a moving image as the main image. The following describes the case where the main image is a moving image with reference to Fig. 14.

[0076] In the playlist 1400, one main image, which is a moving image, is defined, and two types of region information and annotation information for the main image are defined. Here, the region information and annotation information are timed metadata having information according to a time series, and can be acquired as an MP4 file, just like the main image (moving image). The format of the timed metadata may be an XMP / XML file, as in the case where the main image is a still image, but it is desirable for data to exist that is synchronized in time with the frames of the main image. Furthermore, even if the main image is a moving image, if the positions of the partial regions are fixed, the region information and annotation information may be described as in the first embodiment.

[0077] In MPEG-DASH and similar streaming technologies, it is also possible to assign different region information to each period, which is the time length of each segment. Therefore, region information may be set and updated for each period.

[0078] [Embodiment 3] In the first and second embodiments, the processing by the information processing device has been mainly described. In the present embodiment, the processing related to playlist analysis and playback performed by the receiving device 110 that has received the playlist output by the information processing device 100 will be described.

[0079] 15 is a flowchart showing an example of a process for analyzing a playlist, determining whether or not a video can be played, and playing the video when the receiving device 110 receives the playlist. The receiving device 110 can read each piece of information written in the playlist by the generating unit 106, as described in FIGS. 6 to 13 of the first embodiment.

[0080] In S1501, the receiving device 110 acquires a playlist from the information processing device 100. In S1502, the receiving device 110 determines, from the description in the playlist, whether or not annotation information exists in the media to be played. In the example of Fig. 15, it determines whether or not there is media in the MPD where the representation ID of the media to be played is described in "associationID" and whose "associationType" is "cdsc". If there is, the processing proceeds to S1503; if not, the processing ends.

[0081] In S1503, the receiving apparatus 110 determines whether a partial region is associated with the annotation information. Here, it is determined whether region information is assigned as an attribute of the annotation information. The region information is described as defined in a schema such as "urn:mpeg:dash:rgon:2021", for example, as in the first embodiment. If a partial region is associated with the annotation information, the process proceeds to S1504; if not, the process ends.

[0082] In S1504, the receiving device 110 defines a partial area in the main image based on the playlist. Here, the receiving device 110 acquires the size and area information of the media (main image) to be played from the description in the playlist, and specifies the shape and position of the partial area.

[0083] In S1505, the receiving device 110 acquires the encoded data of the media to be played back based on the network address listed in the playlist, and plays and displays it. In S1506, the receiving device 110 superimposes a frame surrounding the partial area on the display screen displayed in S1505. Next, in S1507, the receiving device 110 acquires annotation information and displays it on the display screen in association with the frame displayed in S1506.

[0084] According to this process, it is possible to acquire and play back the video to be played back based on the information in the playlist, and the annotation information to be displayed in association with a partial area of ​​the video.

[0085] [Other embodiments] Although the exemplary embodiments have been described above in detail, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a recording medium (storage medium), etc. Specifically, the present invention may be applied to a system made up of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus made up of a single device.

[0086] The present invention can also be achieved by supplying a software program that realizes the functions of the above-described embodiments to a system or device directly or remotely, and having the computer in the system or device read and execute the supplied program code. Note that the program in this case is computer-readable and corresponds to the flowcharts shown in the figures in the embodiments.

[0087] Therefore, the program code installed on a computer to realize the functional processing of the present invention also realizes the present invention, that is, the present invention includes the computer program itself for realizing the functional processing of the present invention.

[0088] In this case, as long as it has the functionality of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

[0089] The following media are available as recording media for supplying the program: floppy disks, hard disks, optical disks, magneto-optical disks, MOs, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, non-volatile memory cards, ROMs, DVDs (DVD-ROMs, DVD-Rs), etc.

[0090] The following methods are also possible for supplying the program. That is, the browser of a client computer is connected to an Internet homepage, and the computer program of the present invention itself (or a compressed file including an automatic installation function) is downloaded from there to a recording medium such as a hard disk. The program of the present invention can also be realized by dividing the program code constituting the program of the present invention into multiple files and downloading each file from a different homepage. In other words, the present invention also includes a WWW server that allows multiple users to download program files for implementing the functional processing of the present invention on a computer.

[0091] It is also possible to encrypt the program of the present invention, store it on a storage medium such as a CD-ROM, and distribute it to users, and allow users who meet certain conditions to download key information for decrypting the program from a website via the Internet. In other words, the user can use the key information to run the encrypted program and install it on their computer.

[0092] The functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, the OS running on the computer may perform some or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments may also be realized by this processing.

[0093] Furthermore, the functions of the above-described embodiments can also be realized by writing a program read from a recording medium into a memory provided in an expansion board inserted into a computer or in a function expansion unit connected to the computer, and then executing the program. That is, based on the instructions of the program, a CPU provided in the expansion board or function expansion unit can perform some or all of the actual processing.

[0094] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0095] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0096] 101: analysis unit, 102: extraction unit, 103: generation unit, 104: conversion unit, 105: storage unit, 106: generation unit, 107: communication unit

Claims

1. a generating means for generating a playlist including a network address referenced for acquiring an image, area information defining a spatial partial area in the image, and annotation information that can be displayed in association with the partial area; a transmitting means for transmitting the playlist generated by the generating means; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the region information includes information that defines the position and shape of the partial region in the image.

3. 3. The information processing apparatus according to claim 2, wherein the region information includes information that defines the shape of the partial region as a point, a rectangle, a circle, an ellipse, a polygon, or a pixel-designated region.

4. The information processing apparatus according to claim 3 , wherein the region information includes parameters indicating the position and size of the partial region, the number of parameters corresponding to the shape of the partial region.

5. the position and shape of the partial region in the image are specified by a description in a predetermined format; 5. The information processing apparatus according to claim 2, wherein a method for interpreting the description in the predetermined format is indicated by a schema.

6. 6. The information processing device according to claim 2, wherein, when there are multiple partial areas with the same shape, the generating means generates a playlist including area information in which the partial areas with the same shape are described together in one element.

7. 7. The information processing device according to claim 1, wherein the region information defines the partial region by reducing the amount of data by integrating adjacent pixels when the shape of the region is specified in pixel units.

8. 8. The information processing device according to claim 1, wherein the partial region is one of a region indicating an object detected from the image, a region in which a predetermined event is detected in the image by context analysis, and a region specified by a user.

9. The information processing apparatus according to claim 8 , wherein the annotation information includes information indicating an object detected from the image, or information indicating a means, a type, or an algorithm by which the region was identified.

10. 10. The information processing device according to claim 8, wherein the object detected from the image includes any one of a person, a face, an eye, an animal, a vehicle, a two-wheeled vehicle, a license plate, and a lesion.

11. The information processing device according to claim 1 , wherein the annotation information includes any one of text, image, video, and audio.

12. The information processing apparatus according to claim 1 , wherein the annotation information includes a tag indicating that the annotation information has common attribute information.

13. The information processing device according to claim 1 , wherein the playlist includes a network address of an analysis service that generates the region information or the annotation information by performing image analysis or context analysis on the image.

14. The information processing device according to claim 13 , wherein the playlist includes parameters to be provided to the analysis service for generating the region information or the annotation information.

15. The information processing device according to claim 1 , wherein the image is a composite image generated by combining a plurality of images.

16. 16. The information processing device according to claim 1, wherein the playlist is a Media Presentation Description defined in ISO / IEC 23009-1.

17. 17. The information processing device according to claim 1, wherein the image is a still image or an image constituting a moving image, and the partial region is a partial region in the still image or a partial region within an image of one or more frames in the moving image.

18. 18. The information processing device according to claim 1, wherein the playlist includes a network address referenced to acquire the area information or the annotation information.

19. a receiving means for receiving a playlist including a network address to be referenced for acquiring an image, area information defining a spatial partial area in the image, and annotation information that can be displayed in association with the partial area; an analysis means for analyzing the received playlist; an acquisition means for acquiring the image corresponding to the network address based on the result of the analysis; a display means for displaying the annotation information by superimposing it on the image; An information processing device comprising:

20. generating a playlist including a network address referenced for acquiring an image, area information defining a spatial partial area in the image, and annotation information that can be displayed in association with the partial area; transmitting the generated playlist; An information processing method comprising:

21. receiving a playlist including a network address to be referenced for acquiring an image, region information defining a spatial partial region in the image, and annotation information that can be displayed in association with the partial region; parsing the received playlist; acquiring the image corresponding to the network address based on the result of the analysis; displaying the annotation information superimposed on the image; An information processing method comprising:

22. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 19.

Images