Information processing apparatus and method

Abstract

To enable easier selection of a sub-picture stream.SOLUTION: Sub-picture-based image encoding data obtained by dividing an entire picture into a plurality of sub-pictures and encoding the sub-pictures, is managed. A control file to be used for controlling distribution of the image encoding data is created so as to include, separately from arrangement information for each picture region, information about regions, in the entire picture, corresponding to the sub-pictures. The present disclosure is applicable to an information processing device, an image processing device, an image encoding device, a file creating device, a file transmitting device, a distribution device, a file receiving device, an image decoding device, or a reproduction device, for example.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly to an information processing apparatus and method that can more easily select a sub-picture stream.

Background Art

[0002] Conventionally, as a standardization standard for adaptive content distribution technology based on the HTTP (Hypertext Transfer Protocol) protocol, there is MPEG-DASH (Moving Picture Experts Group - Dynamic Adaptive Streaming over HTTP) (see, for example, Non-Patent Document 1 and Non-Patent Document 2).

[0003] In addition, as the file format of this MPEG-DASH, there is ISOBMFF (International Organization for Standardization Base Media File Format), which is the file container specification of the international standard technology for video compression "MPEG-4 (Moving Picture Experts Group - 4)" (see, for example, Non-Patent Document 3).

[0004] By the way, it is considered to use MPEG-DASH for delivering an omnidirectional image (also referred to as a projection plane image) obtained by mapping a stereoscopic structure image, which is an image obtained by projecting an image of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction, like a so-called omnidirectional image, onto a planar image. For example, MPEG-DASH can be applied by mapping a stereoscopic structure image onto a single plane and delivering it as a projection plane image onto which the stereoscopic structure image is mapped. At that time, it has also been proposed to divide a projection plane image (also referred to as an entire picture) of one omnidirectional image into a plurality of sub-pictures and store them in a plurality of tracks. When identifying the display area of a sub-picture, first, it is necessary to construct an entire picture from the sub-picture based on sub-picture division information, and then rearrange the region-wise packed entire picture based on region-wise packing information (see, for example, Non-Patent Document 4).

[0005] [Non-Patent Document 1] "Information technology. Dynamic adaptive streaming over HTTP (DASH). Part 1: Media presentation description and segment formats", ISO / IEC23009-1, 2014 / 05 [Non-Patent Document 2] "Information technology. Dynamic adaptive streaming over HTTP (DASH). Part 1: Media presentation description and segment formats AMENDMENT2: Spatial relationship description, generalized URL parameters and other extensions", ISO / IEC 23009-1:2014 / Amd 2:2015, 2015 / 07

Non-Patent Document 3

Non-Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the currently proposed method, when sub-picture conversion is performed, the arrangement information (region-wise packing information of the entire picture before division) with the size and position changed for each picture area is signaled to the Region Wise Packing Box under the Sub Picture Composition Box. Therefore, when selecting and playing a sub-picture track, in order to identify the display area on the projected picture of the sub-picture track, it is necessary to parse the Sub Picture Composition Box and identify the region-wise packing information and the sub-picture division information. Compared with the case of selecting and playing a track that is not a sub-picture track, the processing load may increase.

[0007] The present disclosure has been made in view of such a situation, and enables easier selection of a sub-picture stream.

Means for Solving the Problems

[0008] An information processing apparatus according to one aspect of the present technology includes a file generation unit that generates a content file including image-encoded data in which an entire picture is encoded and a predetermined metadata box, and the metadata box Project the entire sky image onto a three-dimensional structure is an information processing apparatus that indicates whether a projection picture, which is a projection plane image mapped onto a plane, is the same as the entire picture composed of sub-pictures, and indicates whether region-wise packing processing has been performed on the entire picture.

[0009] An information processing method according to one aspect of the present technology generates a content file including image-encoded data in which an entire picture is encoded and a predetermined metadata box, and the metadata box Project the entire sky image onto a three-dimensional structure is an information processing method that indicates whether a projection picture, which is a projection plane image mapped onto a plane, is the same as the entire picture composed of sub-pictures, and indicates whether region-wise packing processing has been performed on the entire picture.

[0010] An information processing apparatus according to another aspect of the present technology includes a file acquisition unit that acquires a content file including image-encoded data in which an entire picture is encoded and a predetermined metadata box, and a file processing unit that processes the acquired content file, and the metadata box Project the entire sky image onto a three-dimensional structure is an information processing apparatus that indicates whether a projection picture, which is a projection plane image mapped onto a plane, is the same as the entire picture composed of sub-pictures, and indicates whether region-wise packing processing has been performed on the entire picture.

[0011] An information processing method according to another aspect of the present technology acquires a content file including image-encoded data in which an entire picture is encoded and a predetermined metadata box, processes the acquired content file, and the metadata box Project the entire sky image onto a three-dimensional structureAn information processing method that indicates whether a projection picture, which is a projection plane image mapped onto a plane, is the same as the overall picture composed of sub-pictures, and indicates whether region-wise packing processing has been performed on the overall picture.

[0012] In an information processing apparatus and method according to one aspect of the present technology, a content file including encoded image data of the overall picture and a predetermined metadata box is generated. By means of that metadata box, Project the entire sky image onto a three-dimensional structure it is indicated whether a projection picture, which is a projection plane image mapped onto a plane, is the same as the overall picture composed of sub-pictures, and it is indicated whether region-wise packing processing has been performed on that overall picture.

[0013] In an information processing apparatus and method according to another aspect of the present technology, a content file including encoded image data of the overall picture and a predetermined metadata box is acquired, and the acquired content file is processed. By means of that metadata box, Project the entire sky image onto a three-dimensional structure it is indicated whether a projection picture, which is a projection plane image mapped onto a plane, is the same as the overall picture composed of sub-pictures, and it is indicated whether region-wise packing processing has been performed on that overall picture.

Advantages of the Invention

[0014] According to the present disclosure, information can be processed. In particular, the selection of a sub-picture stream can be performed more easily.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 57

Figure 58

Figure 59

Figure 60

Figure 61

Figure 62

Figure 63

Figure 64

Figure 65

Figure 66

Figure 67

Figure 68

Figure 69

Figure 70

Figure 71

Figure 72

Figure 73

Figure 74

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order. 1. Signal of information related to sub-picture 2. First Embodiment (signal of the display area of sub-picture, extension of ISOBMFF) 3. Second Embodiment (signal of the display area of sub-picture, extension of MPD) 4. Third Embodiment (Signal of Stereo Information of Whole Picture, Extension of ISOBMFF) 5. Fourth Embodiment (Signal of Stereo Information of Whole Picture, Extension of MPD) 6. Supplementary Note

[0017] <Signal of Information Regarding 1 Sub-Picture> <Literature, etc. Supporting Technical Content and Technical Terms> The scope disclosed by this technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents that were publicly known at the time of filing the application.

[0018] Non-Patent Document 1: (Described above) Non-Patent Document 2: (Described above) Non-Patent Document 3: (Described above) Non-Patent Document 4: (Described above)

[0019] That is, the content described in the above non-patent documents also serves as a basis for judging the support requirements. For example, regarding technical terms such as Parsing, Syntax, and Semantics, even if there is no direct description in the embodiments, they are within the scope of the disclosure of this technology and are considered to meet the support requirements of the claims.

[0020] <mpeg-dash> Conventionally, for example, as described in Non-Patent Document 1 and Non-Patent Document 2, there is MPEG-DASH (Moving Picture Experts Group - Dynamic Adaptive Streaming over HTTP) as a standardization standard for adaptive content delivery technology based on the HTTP (Hypertext Transfer Protocol).

[0021] With this MPEG-DASH, for example, video playback at an optimal bitrate according to fluctuations in network bandwidth can be realized using HTTP, which is the same communication protocol as when downloading a web page on the Internet from a website.

[0022] This standard can make it easier to develop infrastructure for video distribution services and technologies for video playback clients. In particular, for operators of distribution services, in addition to improving the compatibility between video distribution services and video playback clients, there is an advantage that existing content assets can be easily utilized, and an effect of promoting market growth is expected.

[0023] MPEG-DASH mainly consists of two technical plans. It is a standard that defines a manifest file specification called MPD (Media Presentation Description) that describes metadata for managing video and audio files, and an operation standard for a file format called a segment format for actually transmitting video content.

[0024] As this file format, for example, as described in Non-Patent Document 3, there is ISOBMFF (International Organization for Standardization Base Media File Format), which is a file container specification of the international standard technology for video compression, "MPEG-4 (Moving Picture Experts Group - 4)". For ISOBMFF, functional extensions to meet the requirements of MPEG-DASH have been added as an extended specification of ISO / IEC (International Organization for Standardization / International Electrotechnical Commission) 14496-12.

[0025] <Delivery of Omnidirectional Video Using MPEG-DASH> By the way, there is a projected plane image in which a stereoscopic structure image obtained by projecting an image of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction, such as a so-called omnidirectional image, onto a stereoscopic structure is mapped onto a planar image. For example, by rendering the surrounding image (omnidirectional video) viewed from a viewpoint into a stereoscopic structure centered on that viewpoint to obtain a stereoscopic structure image, the surrounding image of the viewpoint can be expressed more naturally, or an image in a desired line-of-sight direction can be easily generated from the stereoscopic structure image.

[0026] In recent years, it has been considered to use MPEG-DASH for the delivery of this projected plane image (omnidirectional video, etc.). For example, as described in Non-Patent Document 4, MPEG-DASH can be applied by mapping a stereoscopic structure image onto a single plane and delivering it as a projected plane image onto which the stereoscopic structure image is mapped.

[0027] As a method of projecting onto a three-dimensional structure and mapping it onto a plane (also referred to as a projection format), for example, there are ERP (Equirectangular projection), CMP (Cubemap projection), etc. For example, in the case of ERP, a three-dimensional structure image in which an image of 360 degrees around the horizontal direction and 180 degrees around the vertical direction is projected onto a spherical three-dimensional structure is mapped onto a single plane such that the latitude direction and the longitude direction of the spherical three-dimensional structure are orthogonal. Also, for example, in the case of CMP, each face of a three-dimensional structure image in which an image of 360 degrees around the horizontal direction and 180 degrees around the vertical direction is projected onto each face of a cube is unfolded and mapped so as to be arranged in a predetermined order on a single plane.

[0028] The projected plane image onto which the omnidirectional image is projected and mapped in this way is also referred to as a projected picture. That is, a projected picture is a two-dimensional image (two-dimensional picture) representing an omnidirectional image, which is determined for each projection format.

[0029] In the MPEG-I Part2 Omnidirectional Media Format (ISO / IEC 23090-2) FDIS (Final Draft International Standards) (hereinafter also referred to as OMAF) described in Non-Patent Document 4, there is a discussion on a technique of dividing a projected plane image (also referred to as an overall picture) of one omnidirectional image into a plurality of sub-pictures and storing them in a plurality of tracks.

[0030] For example, there is a use case in which a sub-picture track corresponding to a visual field is configured for each specific visual field region, and the client selects and plays back the sub-picture track according to its own visual field region.

[0031] <Box Hierarchical Structure of ISOBMFF File> The Box hierarchy 11 in FIG. 1 shows an example of the Box hierarchy of an ISOBMFF file when converting a full-sphere video into sub-picture tracks.

[0032] As shown in the Box hierarchy 11, in this case, information about the entire picture is stored under the Track Group Box. For example, the Sub Picture Composition Box (spco) stores information used for grouping sub-picture tracks, such as whether the picture is sub-pictured. Below that, Boxes such as the Sub Picture Region Box (sprg), Region Wise Packing Box (rwpk), and Stereo Video Box (stvi) are formed.

[0033] The Sub Picture Region Box stores sub-picture division information indicating how the sub-picture is divided. The Region Wise Packing Box stores region-wise packing information of the entire picture before division. The Stereo Video Box stores information related to the stereo display (stereoscopic display) of the entire picture (stereo information). Stereo information is information indicating the type of image for stereoscopic display, such as side by side or top & bottom.

[0034] Also, below the Scheme Information Box (schi) under the Restricted Scheme Information Box (rinf) under the Restricted Sample Entry (resv) (a type of Sample Entry) under the Sample Description Box (stsd) under the Sample Table Box (stbl) under the Media Information Box (minf) under the Media Box (mdia), boxes such as the Projected Omnidirectional Video Box (povd) and StereoVideoBox (stvi) are formed.

[0035] The Projected Omnidirectional Video Box stores metadata related to the omnidirectional video. The StereoVideoBox stores stereo information about the sub-picture corresponding to that box.

[0036] The Box hierarchy structure 12 in Figure 2 shows an example of the Box hierarchy structure of an ISOBMFF file when the omnidirectional video is not sub-picture tracked.

[0037] As shown in the Box hierarchy structure 12, in this case, the Track Group Box is not formed, and the Region Wise Packing Box is formed under the Projected Omnidirectional Video Box.

[0038] That is, when sub-picture conversion is performed, the Region Wise Packing Box indicating the arrangement information with the size and position changed for each picture area is signaled only to the Sub Picture Composition Box and has the region-wise packing information of the entire picture before division. On the other hand, when sub-picture conversion is not performed, the Region Wise Packing Box is signaled to the Projected Omnidirectional Video Box and has the region-wise packing information of the pictures stored in the track. Hereinafter, a track having a Sub Picture Composition Box is referred to as a sub-picture track.

[0039] <Selection of sub-picture track> Therefore, depending on whether the track is a sub-picture track or a normal track without sub-picture conversion, the process for the client to identify the display area of the image of that track on the projected picture is different. For example, when selecting and playing a sub-picture track, in order to identify the display area on the projected picture of the sub-picture track, it is necessary to parse the Sub Picture Composition Box and identify the region-wise packing information and the sub-picture division information. On the other hand, when selecting and playing a track that is not a sub-picture track, this process is unnecessary.

[0040] Syntax 21 in FIG. 3 shows an example of the syntax of the Sub Picture Composition Box. As shown in Syntax 21, in this Sub Picture Composition Box, the Sub Picture Region Box and the Region Wise Packing Box are set.

[0041] The syntax 22 in FIG. 4 shows an example of the syntax of the Sub Picture Region Box. As shown in the syntax 22, in this Sub Picture Region Box, fields such as track_x, track_y, track_width, track_height, composition_width, and composition_height are defined.

[0042] The semantics 23 in FIG. 5 shows an example of the semantics of the fields defined within the Sub Picture Region Box. As shown in the semantics 23, track_x indicates the horizontal position on the overall picture of the sub-picture stored in the track. track_y indicates the vertical position on the overall picture of the sub-picture stored in the track. track_width indicates the width of the sub-picture stored in the track. track_height indicates the height of the sub-picture stored in the track. composition_width indicates the width of the overall picture. composition_height indicates the height of the overall picture.

[0043] The syntax 24 in FIG. 6 shows an example of the syntax of the Region Wise Packing Box. As shown in the syntax 24, in this Region Wise Packing Box, the Region Wise Packing Struct is set.

[0044] Syntax 25 in FIG. 7 shows an example of the syntax of Region Wise Packing Struct. As shown in Syntax 25, in this Region Wise Packing Struct, fields such as constituent_picture_matching_flag, num_regions, proj_picture_width, proj_picture_height, packed_picture_width, packed_picture_height, guard_band_flag[i], packing_type[i], GuardBand(i) are set.

[0045] Semantics 26 in FIG. 8 shows an example of the semantics of the fields defined within Region Wise Packing Struct. As shown in Semantics 26, constituent_picture_matching_flag is flag information indicating whether the same region-wise packing is applied to the left view and the right view when the picture is stereo. For example, when the value of this field is 0, it indicates that it is mono (mono viewpoint view) or that different packings are applied to the Left view and the Right view. Also, when the value of this field is 1, it indicates that the same packing is applied to the Left view and the Right view.

[0046] Also, num_regions indicates the number of packed regions. proj_picture_width indicates the width of the projected picture. proj_picture_height indicates the height of the projected picture. packed_picture_width indicates the width of the packed picture (the picture packed region-wise). packed_picture_height indicates the height of the packed picture.

[0047] Also, guard_band_flag[i] is flag information indicating whether a guard band exists. For example, when the value of this field is 0, it indicates that there is no guard band in the packed region, and when the value of this field is 1, it indicates that there is a guard band in the packed region. packing_type[i] indicates the shape of the packed region. For example, when the value of this field is 0, it indicates that the packed region is rectangular. GuardBand(i) is the guard band information around the region.

[0048] Also, as shown in Syntax 25, in the Region Wise Packing Struct, further, RectRegionPacking is set. Syntax 27 in FIG. 9 shows an example of the syntax of this RectRegionPacking. As shown in Syntax 27, in this RectRegionPacking, fields such as proj_reg_width[i], proj_reg_height[i], proj_reg_top[i], proj_reg_left[i], transform_type[i], packed_reg_width[i], packed_reg_height[i], packed_reg_top[i], packed_reg_left[i] are set.

[0049] The semantics 28 in FIG. 10 shows an example of the semantics of fields defined within RectRegionPacking. As shown in the semantics 28, proj_reg_width[i] indicates the width of the projected region from which region-wise packing is applied. proj_reg_height[i] indicates the height of the projected region from which region-wise packing is applied. proj_reg_top[i] indicates the vertical position of the projected region from which region-wise packing is applied. proj_reg_left[i] indicates the horizontal position of the projected region from which region-wise packing is applied. transform_type[i] indicates the rotation or mirroring of the packed region. packed_reg_width[i] indicates the width of the packed region rearranged by region-wise packing. packed_reg_height[i] indicates the height of the packed region rearranged by region-wise packing. packed_reg_top[i] indicates the vertical position of the packed region rearranged by region-wise packing. packed_reg_left[i] indicates the horizontal position of the packed region rearranged by region-wise packing.

[0050] That is, for example, when a client selects a sub-picture track according to the user's view, it is necessary to parse this information, and there is a risk that the processing load will increase compared to the case of selecting and playing a track that is not a sub-picture track.

[0051] <Identification of Stereo Information> Also, when the overall picture of the stereo omnidirectional video is sub-pictured, the Stereo Video Box indicating the stereo information of the overall picture (such as what kind of stereoscopic display image the overall picture is) is signaled to the Sub Picture Composition Box, and the Stereo Video Box indicating the stereo information of the sub-picture (such as what kind of stereoscopic display image the sub-picture is) is signaled under the Scheme Information Box of the Sample Entry of the track. On the other hand, when there is no sub-picturing, the Stereo Video Box is signaled only under the Scheme Information Box and has the stereo information of the picture stored in the track.

[0052] Therefore, depending on whether the track is a sub-picture track or a normal track without sub-picturing, the process for the client to identify the stereo information of the track is different. For example, when the overall picture is a stereo image (stereoscopic image), the divided sub-picture track includes the L view and the R view, but the frame packing arrangement may not be top & bottom or side by side.

[0053] Therefore, when identifying whether such a sub-picture can be displayed stereoscopically, it is necessary to parse the Sub Picture Composition Box and perform processes to identify the region-wise packing information, sub-picture division information, and stereo information. On the other hand, when selecting and playing a track that is not a sub-picture track, this process is not required.

[0054] That is, for example, when a client selects a sub-picture track according to its own stereo display ability, there is a possibility that the processing load increases compared to the case of selecting and playing a track that is not a sub-picture track.

[0055] In the above, the selection of the sub-picture track in the ISOBMFF file has been described. In the MPD file, the sub-picture is managed as an Adaptation Set. For the same reason, there is a possibility that the processing load increases for the selection of the Adaptation Set that refers to the sub-picture in this MPD file. That is, regardless of whether it is an ISOBMFF file or an MPD file, there is a possibility that the load of stream selection increases.

[0056] <Signal of the display area of the sub-picture> Therefore, when the entire picture is sub-pictured (sub-picture), information regarding the display area of the sub-picture is signaled (provided to the playback side of the content). The display area refers to the area in the entire picture. That is, the information regarding the display area of the sub-picture is information regarding the area in the entire picture corresponding to the sub-picture, that is, information indicating which part of the entire picture the sub-picture is an image of. With this information, for example, the position, size, shape, etc. of the area corresponding to the sub-picture are indicated. The method of expressing the area is arbitrary. For example, the range of the area may be indicated by coordinates or the like.

[0057] By doing so, the client that plays back the content can grasp where the sub-picture is displayed in the omnidirectional video based on this information.

[0058] At that time, information regarding the display area of this sub-picture is signaled as information for each sub-picture. By doing so, the client can easily obtain this information. Therefore, the client can easily select the stream of the desired sub-picture. For example, when selecting a stream according to the user's field of view, the client can easily select an appropriate stream corresponding to the direction and range of the field of view.

[0059] <Signal of stereo information of the entire picture to be sub-pictured> Also, stereo information, which is information regarding the stereo display of the entire picture to be sub-pictured, is signaled. By doing so, the client that plays the content can easily grasp whether the entire picture is a stereo image (image for stereoscopic viewing) based on this information, and if it is a stereo image, its type, etc. Thereby, the client can easily grasp what kind of image is included in the sub-picture (for example, which part of what type of stereo image (or monocular image (single-viewpoint image)) it corresponds to, etc.).

[0060] Therefore, the client can easily select the desired stream. For example, when selecting a stream according to its own capability, the client can easily select an appropriate stream according to its own capability.

[0061] <File generation device> Next, the configuration of the apparatus that processes signals related to sub-pictures will be described. FIG. 11 is a block diagram showing an example of the configuration of a file generation apparatus, which is an aspect of an information processing apparatus to which the present technology is applied. The file generation apparatus 100 shown in FIG. 11 is an apparatus that generates an ISOBMFF file (segment file) or an MPD file. For example, the file generation apparatus 100 implements the techniques described in Non-Patent Documents 1 to 4, and in a method compliant with MPEG-DASH, generates an ISOBMFF file including a stream and an MPD file, which is a control file used for stream distribution control, and uploads (transmits) those files to a server that distributes those files via a network.

[0062] Note that FIG. 11 shows the main components such as the processing unit and the data flow, and does not necessarily show all of them. That is, in the file generation apparatus 100, there may be a processing unit that is not shown as a block in FIG. 11, or a process or data flow that is not shown as an arrow or the like in FIG. 11.

[0063] As shown in FIG. 11, the file generation apparatus 100 includes a control unit 101, a memory 102, and a file generation unit 103.

[0064] The control unit 101 controls the overall operation of the file generation apparatus 100. For example, the control unit 101 controls the file generation unit 103 to generate an ISOBMFF file or an MPD file, or to upload the generated ISOBMFF file or MPD file. The control unit 101 performs the processes related to such control using the memory 102. For example, the control unit 101 loads a desired program or the like into the memory 102 and executes it to perform the processes related to the above-described control.

[0065] The file generation unit 103 performs processes related to the generation and upload (transmission) of an ISOBMFF file or an MPD file according to the control of the control unit 101. As shown in FIG. 11, the file generation unit 103 includes a data input unit 111, a data encoding / generation unit 112, an MPD file generation unit 113, a recording unit 114, and an upload unit 115.

[0066] The data input unit 111 performs processes related to receiving data input. For example, the data input unit 111 receives input of data such as images necessary for generating textures and meshes, and metadata necessary for generating an MPD file. Further, the data input unit 111 supplies the received data to the data encoding / generation unit 112 and the MPD file generation unit 113.

[0067] The data encoding / generation unit 112 performs processes related to data encoding and file generation. For example, the data encoding / generation unit 112 generates a stream such as a texture or a mesh based on the data such as an image supplied from the data input unit 111. Further, the data encoding / generation unit 112 generates an ISOBMFF file for storing the generated stream. Further, the data encoding / generation unit 112 supplies the generated ISOBMFF file to the recording unit 114.

[0068] As shown in FIG. 11, the data encoding / generation unit 112 includes a preprocessing unit 121, an encoding unit 122, and a segment file generation unit 123.

[0069] The preprocessing unit 121 performs processes on data such as an image before encoding. For example, the preprocessing unit 121 generates a stream such as a texture or a mesh based on the data such as an image supplied from the data input unit 111. Further, for example, the preprocessing unit 121 supplies the generated stream to the encoding unit 122.

[0070] The encoding unit 122 performs processing related to the encoding of the stream. For example, the encoding unit 122 encodes the stream supplied from the preprocessing unit 121. Also, for example, the encoding unit 122 supplies the encoded data obtained by the encoding to the segment file generation unit 123.

[0071] The segment file generation unit 123 performs processing related to the generation of the segment file. For example, the segment file generation unit 123 files the encoded data supplied from the encoding unit 122 in units of segments (generates a segment file) based on metadata and the like supplied from the data input unit 111. Also, for example, as processing related to the generation of the segment file, the segment file generation unit 123 supplies the ISOBMFF file generated as described above to the recording unit 114. For example, the segment file generation unit 123 generates an ISOBMFF file as the segment file and supplies the generated ISOBMFF file to the recording unit 114.

[0072] The MPD file generation unit 113 performs processing related to the generation of the MPD file. For example, the MPD file generation unit 113 generates an MPD file based on metadata and the like supplied from the data input unit 111. Also, for example, the MPD file generation unit 113 supplies the generated MPD file to the recording unit 114. Note that the MPD file generation unit 113 may obtain metadata and the like necessary for the generation of the MPD file from the segment file generation unit 123.

[0073] The recording unit 114 has an arbitrary recording medium such as a hard disk or a semiconductor memory, for example, and performs processing related to the recording of data. For example, the recording unit 114 records the MPD file supplied from the MPD file generation unit 113. Also, for example, the recording unit 114 records the segment file (for example, an ISOBMFF file) supplied from the segment file generation unit 123.

[0074] The upload section 115 performs processes related to file upload (transmission). For example, the upload section 115 reads the MPD file recorded in the recording section 114. Also, for example, the upload section 115 uploads (transmits) the read MPD file to a server (not shown) that distributes the MPD file to a client or the like via a network or the like.

[0075] Also, for example, the upload section 115 reads the segment files (for example, ISOBMFF files) recorded in the recording section 114. Also, for example, the upload section 115 uploads (transmits) the read segment files to a server (not shown) that distributes the segment files to a client or the like via a network or the like.

[0076] That is, the upload section 115 functions as a communication section that transmits MPD files and segment files (for example, ISOBMFF files) to the server. Note that the transmission destination of the MPD file and the transmission destination of the segment file (for example, ISOBMFF file) by the upload section 115 may be the same as each other or different from each other. Also, here, an example will be described in which the file generation device 100 functions as a device that uploads an MPD file and a segment file (for example, ISOBMFF file) to a server that distributes those files to a client, but the file generation device 100 may function as that server. In that case, the upload section 115 of the file generation device 100 may distribute the MPD file and the segment file (for example, ISOBMFF file) to the client via the network.

[0077] <Client device> FIG. 12 is a block diagram showing an example of the configuration of a client device, which is one aspect of an information processing apparatus to which the present technology is applied. The client device 200 shown in FIG. 12 is a device that acquires MPD files and segment files (for example, ISOBMFF files) and plays back content based on those files. For example, the client device 200 implements the technologies described in Non-Patent Documents 1 to 4, acquires segment files from a server (or the above-described file generation device 100) in a method compliant with MPEG-DASH, and plays back the stream (content) included in the segment files. At that time, the client device 200 may acquire an MPD file from a server (or the above-described file generation device 100), select a desired segment file using the MPD file, and acquire it from the server.

[0078] Note that in FIG. 12, main components such as processing units and data flows are shown, and what is shown in FIG. 12 is not necessarily all. That is, in the client device 200, there may be a processing unit that is not shown as a block in FIG. 12, or there may be a process or data flow that is not shown as an arrow or the like in FIG. 12.

[0079] As shown in FIG. 12, the client device 200 includes a control unit 201, a memory 202, and a playback processing unit 203.

[0080] The control unit 201 controls the operation of the entire client device 200. For example, the control unit 201 controls the playback processing unit 203 to acquire an MPD file or a segment file (for example, an ISOBMFF file) from a server, or to play back the stream (content) included in the segment file. The control unit 201 performs processing related to such control using the memory 202. For example, the control unit 201 loads a desired program or the like into the memory 202 and executes it to perform processing related to the above-described control.

[0081] The reproduction processing unit 203 performs processing related to the reproduction of the stream (content) included in the segment file according to the control of the control unit 201. As shown in FIG. 12, the reproduction processing unit 203 includes a measurement unit 211, an MPD file acquisition unit 212, an MPD file processing unit 213, a segment file acquisition unit 214, a display control unit 215, a data analysis and decoding unit 216, and a display unit 217.

[0082] The measurement unit 211 performs processing related to measurement. For example, the measurement unit 211 measures the transmission bandwidth of the network between the client device 200 and the server. Also, for example, the measurement unit 211 supplies the measurement result to the MPD file processing unit 213.

[0083] The MPD file acquisition unit 212 performs processing related to the acquisition of the MPD file. For example, the MPD file acquisition unit 212 acquires the MPD file corresponding to the desired content (content to be reproduced) from the server via the network. Also, for example, the MPD file acquisition unit 212 supplies the acquired MPD file to the MPD file processing unit 213.

[0084] The MPD file processing unit 213 performs processing based on the MPD file. For example, the MPD file processing unit 213 selects the stream to be acquired based on the MPD file supplied from the MPD file acquisition unit 212. Also, for example, the MPD file processing unit 213 supplies the selection result to the segment file acquisition unit 214. In selecting the stream to be acquired, the measurement result supplied from the measurement unit 211 and the information related to the user's viewpoint position and line-of-sight direction supplied from the display control unit 215 are also appropriately used.

[0085] The segment file acquisition unit 214 performs processes related to the acquisition of segment files (for example, ISOBMFF files). For example, the segment file acquisition unit 214 acquires a segment file storing a stream necessary for playing desired content from a server via a network. Also, for example, the segment file acquisition unit 214 supplies the acquired segment file to the data analysis and decoding unit 216.

[0086] Note that the server from which the segment file acquisition unit 214 acquires a segment file (for example, an ISOBMFF file) may be the same as or different from the server from which the MPD file acquisition unit 212 acquires the MPD file. Also, the segment file acquisition unit 214 may acquire a segment file based on the selection result of the stream supplied from the MPD file processing unit 213. That is, the segment file acquisition unit 214 may acquire from a server a segment file storing a stream selected based on an MPD file or the like.

[0087] The display control unit 215 performs processes related to the control of content playback (display). For example, the display control unit 215 acquires the detection results of the viewpoint position and the line-of-sight direction of the user who views the content. Also, for example, the display control unit 215 supplies the acquired detection results (information regarding the viewpoint position and the line-of-sight direction of the user) to the MPD file processing unit 213 and the data analysis and decoding unit 216.

[0088] The data analysis and decoding unit 216 performs processes related to data analysis, decoding, etc. For example, the data analysis and decoding unit 216 processes the ISOBMFF file supplied from the segment file acquisition unit 214 and generates a display image for the content. Also, the data analysis and decoding unit 216 supplies the data of the display image to the display unit 217.

[0089] As shown in FIG. 12, the data analysis and decoding unit 216 includes a segment file processing unit 221, a decoding unit 222, and a display information generation unit 223.

[0090] The segment file processing unit 221 performs processing on a segment file (for example, an ISOBMFF file). For example, the segment file processing unit 221 extracts encoded data of a desired stream from the ISOBMFF file supplied from the segment file acquisition unit 214. Also, for example, the segment file processing unit 221 supplies the extracted encoded data to the decoding unit 222.

[0091] Note that the segment file processing unit 221 may select a stream based on information regarding the user's viewpoint position and line-of-sight direction supplied from the display control unit 215, the transmission bandwidth measured by the measurement unit 211, etc., and extract the encoded data of the stream from the segment file.

[0092] The decoding unit 222 performs processing related to decoding. For example, the decoding unit 222 decodes the encoded data supplied from the segment file processing unit 221. Also, for example, the decoding unit 222 supplies the stream obtained by the decoding to the display information generation unit 223.

[0093] The display information generation unit 223 performs processing related to generating data of a display image. For example, the display information generation unit 223 generates data of a display image corresponding to the user's viewpoint position and line-of-sight direction based on information regarding the user's viewpoint position and line-of-sight direction supplied from the display control unit 215 and the stream supplied from the decoding unit 222. Also, for example, the display information generation unit 223 supplies the generated data of the display image to the display unit 217.

[0094] The display unit 217 has an arbitrary display device such as a display using a liquid crystal display panel or the like, or a projector, and performs processing related to image display using the display device. For example, the display unit 217 performs content reproduction such as image display based on the data supplied from the display information generation unit 223.

[0095] <2. First Embodiment> <Signal of sub-picture display area information by ISOBMFF> The signal of the information regarding the display area of the sub-picture described above may be carried out in an ISOBMFF file which is a segment file.

[0096] That is, information regarding the area in the entire picture corresponding to the sub-picture to be stored may be included as information different from the arrangement information for each picture area, and further, a file including the image coding data obtained by coding the sub-picture may be generated.

[0097] For example, in a file generation device 100 which is an information processing device, a segment file generation unit 123 may function as a file generation unit which includes information regarding the area in the entire picture corresponding to the sub-picture to be stored as information different from the arrangement information for each picture area, and further, generates a file including the image coding data obtained by coding the sub-picture. That is, the information processing device (for example, the file generation device 100) may be provided with a file generation unit (for example, the segment file generation unit 123).

[0098] By doing so, as described above, the client can more easily select a stream based on this information.

[0099] In the ISOBMFF file, a stream is managed as a track. That is, when using the ISOBMFF file, the selection of a stream is made by selecting a track.

[0100] In addition, the above-described picture (overall picture) may be all or part of an omnidirectional image (a projected plane image obtained by projecting and mapping an image of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction). The omnidirectional image is an omnidirectional image centered on the viewpoint (that is, an image of the surroundings seen from the viewpoint). This omnidirectional image can be rendered into a three-dimensional structure to obtain an image of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction. As described above, by mapping a three-dimensional structure image onto a single plane to obtain a projected plane image, stream delivery control applying MPEG-DASH becomes possible. That is, when the file generation device 100 uses all or part of such a projected plane image as the overall picture and sub-pictures it, the present technology can be applied as described above. Note that even when part of the projected plane image is used as the overall picture, information regarding the display area in the entire projected plane image of the sub-picture is signaled.

[0101] For example, as shown in FIG. 13, an image of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction is projected onto a three-dimensional structure (cube) by Cubemap projection to generate a three-dimensional structure image 301. Further, the three-dimensional structure image 301 is mapped onto a single plane by a predetermined method to generate a projected picture 302. The file generation device 100 sub-pictures such a projected plane image 302 to generate sub-pictures (sub-picture 303 to sub-picture 308), and generates an ISOBMFF file that stores each of them in a different track.

[0102] At that time, as indicated by arrow 311, the file generation device 100 signals in the ISOBMFF file information (display area information) indicating which sub-picture corresponds to which part of the overall picture (projected plane image 302).

[0103] By doing so, even when delivering a panoramic video, as described above, the client can more easily select a stream based on this information.

[0104] Note that the information regarding this area (display area information) may be included in the ISOBMFF file as information for each sub-picture. By doing so, the client can easily grasp which part of the entire picture the sub-picture corresponds to by simply referring to the information of the sub-picture track.

[0105] <Upload processing flow> An example of the upload processing flow executed by the file generation device 100 in FIG. 11 in that case will be described with reference to the flowchart of FIG. 14.

[0106] When the upload processing is started, the data input unit 111 of the file generation device 100 acquires an image and metadata in step S101.

[0107] In step S102, the segment file generation unit 123 generates an ISOBMFF file including display area information in the projected picture as information for each sub-picture.

[0108] In step S103, the recording unit 114 records the ISOBMFF file generated by the process of step S102.

[0109] In step S104, the upload unit 115 reads out the ISOBMFF file recorded in step S103 from the recording unit 114 and uploads it to the server.

[0110] When the process of step S104 ends, the upload processing ends.

[0111] By performing the upload process as described above, the file generation device 100 can generate an ISOBMFF file including the display area information in the projected picture as information for each sub-picture.

[0112] Therefore, based on that information, the client can more easily select and play an appropriate stream according to the user's field of view and the like.

[0113] <Use of the display area information of the sub-picture signaled in ISOBMFF> Also, the selection and playback of the stream may be performed using the information regarding the display area of the sub-picture signaled in the ISOBMFF file.

[0114] That is, information regarding the area in the entire picture corresponding to the sub-picture to be stored is included as information different from the arrangement information for each picture area, and further, a file including the image coding data obtained by coding the sub-picture is acquired, and based on the information regarding that area included in the acquired file, the selection of the stream of the image coding data may be performed.

[0115] For example, in the client device 200 which is an information processing device, the segment file acquisition unit 214 functions as a file acquisition unit that includes information regarding the area in the entire picture corresponding to the sub-picture to be stored as information different from the arrangement information for each picture area, and further acquires a file including the image coding data obtained by coding the sub-picture, and the data analysis / decoding unit 216 functions as an image processing unit that selects the stream of the image coding data based on the information regarding that area included in the file acquired by the file acquisition unit. That is, the information processing device (for example, the client device 200) may be provided with a file acquisition unit (for example, the segment file acquisition unit 214) and an image processing unit (for example, the data analysis / decoding unit 216).

[0116] By doing so, the client device 200 can more easily select a stream.

[0117] Note that the above-described picture (entire picture) may be all or part of an omnidirectional image (a projection plane image obtained by projecting and mapping an image of 360 degrees around the horizontal direction and 180 degrees around the vertical direction). That is, even when the client device 200 acquires and reproduces a stream in which all or part of the projection plane image is sub-pictured as an entire picture, the present technology can be applied as described above.

[0118] Also, information about this area (display area information) may be included in the ISOBMFF file as information for each sub-picture (sub-picture). By doing so, the client device 200 can easily grasp which part of the entire picture the sub-picture corresponds to by simply referring to the information of the sub-picture track.

[0119] <Flow of content reproduction processing> An example of the flow of content reproduction processing executed by the client device 200 in that case will be described with reference to the flowchart of FIG. 15.

[0120] When the content reproduction processing is started, the segment file acquisition unit 214 of the client device 200 acquires, in step S121, an ISOBMFF file including display area information in the projected picture as information for each sub-picture.

[0121] In step S122, the display control unit 215 acquires the measurement result of the user's viewpoint position (and line-of-sight direction).

[0122] In step S123, the measurement unit 211 measures the transmission bandwidth of the network between the server and the client device 200.

[0123] In step S124, the segment file processing unit 221 selects a sub-picture track corresponding to the user's field of view of the client device 200 based on the display area information in the projected picture of the sub-picture.

[0124] In step S125, the segment file processing unit 221 extracts the encoded data of the stream of the track selected in step S124 from the ISOBMFF file acquired in step S121.

[0125] In step S126, the decoding unit 222 decodes the encoded data of the stream extracted in step S125.

[0126] In step S127, the display information generation unit 223 plays back the stream (content) decoded in step S126. More specifically, the display information generation unit 223 generates data of a display image from the stream, supplies it to the display unit 217, and causes it to be displayed.

[0127] When the processing of step S127 ends, the content playback processing ends.

[0128] By performing the content playback processing as described above, the client device 200 can more easily select a stream using the information regarding the display area of the sub-picture included in the ISOBMFF file. For example, the client device 200 can easily select an appropriate stream according to the user's field of view based on the information.

[0129] <Definition by 2D Coverage Information Box> As described above, the segment file generation unit 123 of the file generation device 100 newly defines the display area information of the sub-picture indicating which part of the projected picture the sub-picture corresponds to in the OMAF's ISOBMFF file and signals it to the track. That is, the segment file generation unit 123 defines the display area information of the sub-picture as information for each sub-picture.

[0130] For example, the segment file generation unit 123 defines a 2D Coverage Information Box as the display area information of the sub-picture and signals it as a box different from the Region Wise Packing Box. For example, the segment file generation unit 123 defines the 2D Coverage Information Box in the Scheme Information Box. For example, the segment file generation unit 123 may define the 2D Coverage Information Box in the Projected Omnidirectional Video Box below the Scheme Information Box. Also, the segment file generation unit 123 may define the 2D Coverage Information Box in other boxes.

[0131] That is, the display area information of the sub-picture (information regarding the area in the entire picture corresponding to the sub-picture stored in the track) may be stored in a Scheme Information Box of the ISOBMFF file, which is different from the Region Wise Packing Box, or in a box in the lower layer of the Scheme Information Box.

[0132] By doing so, the client device 200 can easily select and play the sub-picture track without parsing the Sub Picture Composition Box.

[0133] Note that this 2D Coverage Information Box can also be used as a signal for display area information when the picture stored in the track is not a sub-picture or when there is no Region Wise Packing Box (when the picture is not Region Wise Packed).

[0134] The syntax 331 in FIG. 16 shows an example of the syntax of this 2D Coverage Information Box. As shown in the syntax 331, in the 2D Coverage Information Box, fields such as proj_picture_width, proj_picture_height, proj_reg_width, proj_reg_height, proj_reg_top, and proj_reg_left are set.

[0135] The semantics 332 in FIG. 17 shows an example of the semantics of the fields defined in this 2D Coverage Information Box. As shown in the semantics 332, proj_picture_width indicates the width of the projected picture. proj_picture_height indicates the height of the projected picture. proj_reg_width indicates the width of the region on the projected picture corresponding to the picture in the track. proj_reg_height indicates the height of the region on the projected picture corresponding to the picture in the track. proj_reg_top indicates the vertical coordinate of the region on the projected picture corresponding to the picture in the track. proj_reg_left indicates the horizontal coordinate of the region on the projected picture corresponding to the picture in the track.

[0136] That is, various types of information as shown in FIG. 18 are defined in the 2D Coverage Information Box.

[0137] Note that each of these fields may be indicated by the actual number of pixels, or proj_reg_width, proj_reg_height, proj_reg_top, and proj_reg_left may be indicated by relative values with respect to proj_picture_width and proj_picture_height. When indicated by the actual number of pixels, it is useful when selecting a track according to the resolution of the client's display.

[0138] By referring to the 2D Coverage Information Box of the sub-picture track with such a configuration, the client device 200 can easily identify the display area of the sub-picture track without parsing the Sub Picture Composition Box. Thereby, the client device 200 can, for example, more easily select a sub-picture track according to the user's field of view. Note that the client device 200 can also select a track that is not a sub-picture track by the same process.

[0139] Also, in the Sub Picture Composition Box shown in the syntax 21 of FIG. 3, as shown in the syntax 1001 of FIG. 73, an identical_to_proj_pic_flag field is additionally defined to indicate whether the entire picture is identical to the projected picture. When the entire picture is identical to the projected picture, the Sub Picture Region Box shown in the syntax 22 of FIG. 4 may indicate the display area information of the sub-picture track. The value of the identical_to_proj_pic_flag field indicates, for example, that 0 means the entire picture is different from the projected picture, and 1 means the entire picture is identical to the projected picture.

[0140] At this time, when the identical_to_proj_pic_flag field is 1, the entire picture has not undergone region-wise packing processing. The semantics of the track_x, track_y, track_width, track_height, composition_width, and composition_height fields of the Sub Picture Region Box shown in the semantics 23 of FIG. 5 are the same as the semantics of the proj_reg_left, proj_reg_top, proj_reg_width, proj_reg_height, proj_picture_width, and proj_picture_height fields of the 2D Coverage Information Box shown in the semantics 332 of FIG. 17, respectively.

[0141] Note that the identical_to_proj_pic_flag field may be additionally defined in the Sub Picture Region Box or in other Boxes. Also, the presence or absence of a specific Box may indicate whether the entire picture is identical to the projected picture.

[0142] Also, one bit of the 24-bit flags commonly held by the Sub Picture Composition Box and other boxes that extend the FullBox may be used to indicate whether the entire picture is the same as the projected picture.

[0143] <When the sub-picture contains discontinuous regions> Note that in the syntax 331 of FIG. 16, when the sub-picture contains discontinuous regions on the projected picture as shown in FIG. 19, it cannot be handled. In the example of FIG. 19, the projected picture 351 is sub-pictured, and sub-pictures 352 to 355 are formed. In this case, the sub-picture 352 contains the Left and Right faces in the three-dimensional structure image (the Left and Right faces are adjacent). These Left and Right faces are discontinuous in the projected picture 351. Also, the sub-picture 353 contains the Top and Bottom faces in the three-dimensional structure image (the Top and Bottom faces are adjacent). These Top and Bottom faces are discontinuous in the projected picture 351.

[0144] In the syntax 331 of FIG. 16, only one continuous region of the projected picture can be specified, so such discontinuous multiple regions cannot be specified.

[0145] Therefore, in the 2D Coverage Information Box, it may be possible to specify multiple regions so that multiple discontinuous regions in the projected picture can be specified.

[0146] The syntax 371 in FIG. 20 shows an example of the syntax of the 2D Coverage Information Box in this case. As shown in the syntax 371, in this case, the num_regions field is added to the defined fields. The semantics 372 in FIG. 21 shows an example of the semantics of the fields added in the 2D Coverage Information Box in this case. As shown in the semantics 372, num_regions indicates the number of regions on the projected picture included in its sub-picture.

[0147] That is, in this case, in the 2D Coverage Information Box, the num_regions field is used to define each field shown in FIG. 17 (independently of each other) for each region of the projected picture. Therefore, multiple regions of the projected picture can be specified. This enables signaling of discontinuous display regions of the projected picture.

[0148] Note that when the 2D Coverage Information Box is signaled to the Sub Picture Composition Box, it may be signaled to indicate the display region in the overall picture (projected picture).

[0149] Also, if the 2D Coverage Information Box does not exist in the Projected Omnidirectional Video Box of the track, it may be indicated that the track stores 360° omnidirectional video. Similarly, if the 2D Coverage Information Box does not exist in the Sub Picture Composition Box, it may be indicated that the overall picture composed of the sub-picture track is 360° omnidirectional video.

[0150] <Expansion of Region Wise Packing Box> The Region Wise Packing Struct within the Region Wise Packing Box defined by OMAF may be expanded to signal which part of the display area of the projected picture corresponds to the sub-picture of the track. The signaling location of the Region Wise Packing Box is under the Projected Omnidirectional Video Box of the Sample Entry of the sub-picture track. Note that this Region Wise Packing Box may be signaled at other locations.

[0151] For example, a flag indicating the signaling of new display area information of the sub-picture and a Rect Projected Region structure for signaling the display area information of the sub-picture are defined and signaled in the Region Wise Packing Struct. Note that this Region Wise Packing Struct can also be used for signaling this display area information even when the picture stored in the track is not a sub-picture.

[0152] The syntax 373 in FIG. 22 shows an example of the syntax of the Region Wise Packing Struct in that case. As shown in the syntax 373, in this case, the 2D_coverage_flag field is added to the fields defined in the Region Wise Packing Struct. The semantics 374 in FIG. 23 shows an example of the semantics of the fields additionally defined in the Region Wise Packing Struct in this case. As shown in the semantics 374, the 2D_coverage_flag is flag information indicating whether to signal only the display area on the projected picture. For example, when the value of this field is 0, it indicates signaling region-wise packing information. Also, when the value of this field is 1, it indicates signaling the display area on the projected picture.

[0153] Note that in the Region Wise Packing Struct in this case, further, RectProjetedRegion is defined. The syntax 375 in FIG. 24 shows an example of the syntax of the RectProjetedRegion. As shown in the syntax 375, in this RectProjetedRegion, fields such as proj_reg_width[i], proj_reg_height[i], proj_reg_top[i], proj_reg_left[i] are defined.

[0154] The semantics 376 in Figure 25 shows an example of the semantics of the fields defined in this RectProjetedRegion. As shown in the semantics 376, proj_reg_width indicates the width of the region on the projected picture corresponding to the picture of the track. proj_reg_height indicates the height of the region on the projected picture corresponding to the picture of the track. proj_reg_top indicates the vertical coordinate of the region on the projected picture corresponding to the picture of the track. proj_reg_left indicates the horizontal coordinate of the region on the projected picture corresponding to the picture of the track.

[0155] Note that each of the above fields may be indicated by the actual number of pixels, or may be indicated by relative values with respect to proj_picture_width and proj_picture_height signaled by the Region Wise Packing Struct for proj_reg_width, proj_reg_height, proj_reg_top, and proj_reg_left.

[0156] Also, the Rect Wise Packing Struct may be extended to signal only the display area information in the projected picture when 2D_coverage_flag == 1.

[0157] The syntax 377 in Figure 26 shows an example of the syntax of the Rect Wise Packing Struct in that case. The syntax 378 in Figure 27 is a diagram showing an example of the syntax of RectRegionPacking set in the Rect Wise Packing Struct in this case.

[0158] <Extension of Coverage Information Box> The Coverage Information Box indicating the display area on the spherical surface of the track defined by OMAF may be extended, and the display area on the projected picture may be signaled by the newly defined 2D Content Coverage Struct.

[0159] That is, the display area information of the sub-picture (information regarding the area in the entire picture corresponding to the sub-picture stored in the track) may be stored in the Coverage Information Box indicating the display area on the spherical surface of the track in the ISOBMFF file.

[0160] Syntax 379 in FIG. 28 shows an example of the syntax of the extended Coverage Information Box. As shown in Syntax 379, in the Coverage Information Box in this case, 2D_coverage_flag, ContentCoverageStruct(), and 2DContentCoverageStruct() are defined.

[0161] Semantics 380 in FIG. 29 shows an example of the semantics of these fields. As shown in Semantics 380, 2D_coverage_flag is flag information that signals the type of display area information. When this value is 0, it indicates signaling the display area information on the spherical surface, and when this value is 1, it indicates signaling the display area on the projected picture. ContentCoverageStruct() signals the display area on the spherical surface of the track. 2DContentCoverageStruct() signals the display area on the projected picture of the track. The fields in the 2D Content Coverage Struct are the same as those in the 2D Coverage Information Box in the case of FIG. 20.

[0162] Note that the Content Coverage Struct may be extended to signal the display area on the projected picture in addition to the display area on the spherical surface.

[0163] <Signaling when the splitting method of the sub-picture changes dynamically> In the above, the signaling when the splitting method of the sub-picture does not change dynamically in the stream has been described. In contrast, when the splitting method changes dynamically, the display area information of the sub-picture in the projected picture changes dynamically in the stream. In that case, it cannot be handled in the above example.

[0164] Therefore, an additional signaling example for signaling the dynamically changing display area information of the sub-picture will be described below. Note that the information to be signaled is the same as the information signaled in the above 2D Coverage Information Box (for example, Fig. 16, etc.).

[0165] <Supplemental Enhancement Information (SEI) message> In HEVC and AVC, a new 2D Coverage Information SEI message may be defined, and in it, the display area information of the sub-picture that changes dynamically in the stream may be signaled in units of access units.

[0166] That is, the display area information of the sub-picture (information regarding the area in the entire picture corresponding to the sub-picture stored in the track) may be stored in the Supplemental Enhancement information message of the ISOBMFF file.

[0167] The syntax 381 in FIG. 30 shows an example of the syntax of the 2D Coverage Information SEI message in that case. As shown in the syntax 381, in the 2D Coverage Information SEI message, 2D_coverage_information_cancel_flag, 2D_coverage_information_persistence_flag, 2D_coverage_information reserved_zero_6bits, proj_picture_width, proj_picture_height, num_regions, proj_reg_width[i], proj_reg_height[i], proj_reg_top[i], proj_reg_left[i], etc. are set.

[0168] The semantics 382 in FIG. 31 shows an example of the semantics of the fields defined in the 2D Coverage Information SEI message. As shown in the semantics 382, 2D_coverage_information_cancel_flag is flag information regarding the cancellation of 2D_coverage_information. When this value is 1, the persistent application of the SEI preceding in the output order is cancelled. Also, when this value is 0, 2D coverage information is signaled.

[0169] 2D_coverage_information_persitence_flag is flag information regarding the application range of the SEI. When this value is 0, the SEI information is applied only to the picture containing the SEI. Also, when this value is 1, the application of the SEI persists until a new coded video sequence is started or the end of the stream is reached.

[0170] The 2D_coverage_information_reserved_zero_6bits is filled with 0. proj_picture_width indicates the width of the projected picture. proj_picture_height indicates the height of the projected picture. num_regions indicates the number of regions on the projected picture. proj_reg_width indicates the width of the region on the projected picture corresponding to the stream. proj_reg_height indicates the height of the region on the projected picture corresponding to the stream. proj_reg_top indicates the vertical coordinate of the region on the projected picture corresponding to the stream. proj_reg_left indicates the horizontal coordinate of the region on the projected picture corresponding to the stream.

[0171] Note that each of the above fields may be indicated by the actual number of pixels, or proj_reg_width, proj_reg_height, proj_reg_top, and proj_reg_left may be indicated by relative values with respect to proj_picture_width and proj_picture_height.

[0172] <Timed metadata> Also, the mechanism of timed metadata, which is a stream for storing metadata that changes over time, may be used to newly define 2D Coverage Information timed metadata, and in this, the display area information of the sub-picture that dynamically changes within the referenced stream may be signaled. As the track reference type for the track associated with the 2D Coverage Information timed metadata, for example, '2dco' is used.

[0173] That is, the display area information of the sub-picture (information regarding the area in the entire picture corresponding to the sub-picture stored in the track) may be stored in the timed metadata of the ISOBMFF file.

[0174] By using the timed metadata, the client can identify in advance the dynamically changing display area without decoding the sub-picture stream, and use it as a criterion for selecting which stream to choose.

[0175] The syntax 383 in FIG. 32 shows an example of the syntax of the 2D Coverage Information Sample Entry. The syntax 384 in FIG. 33 shows an example of the syntax of the 2D Coverage Information Sample.

[0176] In the 2D Coverage Information Sample Entry, proj_picture_width and proj_picture_height, which are generally invariant in the stream, are signaled. If these change in the stream, they may be signaled within the 2D Coverage Information Sample.

[0177] Note that the semantics of each field in the 2D Coverage Information Sample Entry and the 2D Coverage Information Sample are the same as those in FIGS. 17 and 21.

[0178] <Sample Group> Using the tool Sample Group, which is a mechanism for associating meta-information in units of samples defined in ISOBMFF, the display area information of the sub-picture that dynamically changes within the stream may be signaled in units of samples.

[0179] As shown in Figure 34, the Sample Group in which meta-information is described is signaled as a Group Entry in the Sample Group Description Box of the Sample Table Box, and is associated with a sample via the Sample To Group Box.

[0180] As shown in Figure 34, the grouping_type of the Sample To Group Box indicates the grouping_type of the associated Sample Group Description Box. For each 1 entry, a sample_count and a group_description_index are signaled. The group_description_index indicates the index of the associated Group Entry, and the sample_count indicates the number of samples belonging to that Group Entry.

[0181] For example, a new 2D Coverage Information Sample Group Entry may be defined and the display area information of sub-pictures that dynamically change within the stream may be stored therein.

[0182] That is, the display area information of the sub-picture (information regarding the area in the entire picture corresponding to the sub-picture stored in the track) may be stored in the Sample Group Entry of the ISOBMFF file.

[0183] The syntax 391 in FIG. 35 shows an example of the syntax of its 2D Coverage Information Sample Group Entry. This Sample Group Entry is signaled in the Sample Group Description Box and associated with samples by the Sample To Group Box. The grouping_type is '2cgp'.

[0184] Note that the semantics of each field within this 2D Coverage Information Sample Group Entry are the same as those in FIGS. 16 and 21.

[0185] Note that the above three examples (Supplemental Enhancement Information (SEI) message, Timed metadata, Sample Group) can be used for signaling display area information that changes dynamically even when the picture stored in the track is not a sub-picture.

[0186] Also, when the display area on the projected picture of the sub-picture changes dynamically as described above, the information in the 2D Coverage Information Box signaled in the Projected Omnidirectional Video Box can be used as the initial value of the display area of the stream.

[0187] Alternatively, a flag indicating that the display area in the projected picture of the sub-picture changes dynamically in the stream may be signaled in the 2D Coverage Information Box or other boxes. With this information, the client can easily identify that the stream has a dynamically changing display area.

[0188] <3. Second Embodiment> <Signal of information on display area of sub-picture by MPD file> The signal of information on the display area of the sub-picture described above may be performed in the MPD file. That is, in order to enable a client to select and play an Adaptation Set referring to a sub-picture according to, for example, a user's visual field, in the MPD file, the display area information on the projected picture of the sub-picture may be newly defined and signaled to the Adaptation Set.

[0189] That is, it is also possible to manage the image encoding data for each sub-picture in which the entire picture is divided into a plurality of sub-pictures and encoded, and include information on the area in the entire picture corresponding to the sub-picture as information different from the arrangement information for each picture area, and generate a control file used for distribution control of the image encoding data.

[0190] For example, in the file generation device 100 which is an information processing device, the MPD file generation unit 113 may function as a file generation unit that manages the image encoding data for each sub-picture in which the entire picture is divided into a plurality of sub-pictures and encoded, and includes information on the area in the entire picture corresponding to the sub-picture as information different from the arrangement information for each picture area, and generates a control file used for distribution control of the image encoding data. That is, the information processing device (for example, the file generation device 100) may be provided with a file generation unit (for example, the MPD file generation unit 113).

[0191] By doing so, as described above, the client can more easily select a stream based on this information.

[0192] In the MPD file, metadata for each stream is managed as an Adaptation Set or a Representation. That is, when using the MPD file, the selection of a stream is performed by selecting an Adaptation Set or a Representation.

[0193] Also, the above-mentioned picture (entire picture) may be all or part of an omnidirectional video (a projection plane image obtained by projecting and mapping an image of 360 degrees horizontally and 180 degrees vertically around). That is, when the file generation device 100 uses all or part of such a projection plane image as the entire picture and sub-pictures it, the present technology can be applied as described above.

[0194] By doing so, even when distributing an omnidirectional video, the client can more easily select a stream based on this information as described above.

[0195] Note that the information regarding this area (display area information) may be included in the MPD file as information for each sub-picture. By doing so, the client can easily grasp which part of the entire picture the sub-picture corresponds to by simply referring to the information of the sub-picture referred to by the Adaptation Set.

[0196] <Flow of Upload Process> An example of the flow of the upload process executed by the file generation device 100 in FIG. 11 in that case will be described with reference to the flowchart of FIG. 36.

[0197] When the upload process is started, the data input unit 111 of the file generation device 100 acquires an image and metadata in step S201.

[0198] In step S202, the segment file generation unit 123 generates a segment file of the image.

[0199] In step S203, the MPD file generation unit 113 generates an MPD file including display area information in the projected picture as information for each sub-picture.

[0200] In step S204, the recording unit 114 records the segment file generated by the process of step S202. Also, the recording unit 114 records the MPD file generated by the process of step S203.

[0201] In step S205, the upload unit 115 reads out the segment file recorded in step S204 from the recording unit 114 and uploads it to the server. Also, the upload unit 115 reads out the MPD file recorded in step S204 from the recording unit 114 and uploads it to the server.

[0202] When the process of step S204 ends, the upload process ends.

[0203] By performing the upload process as described above, the file generation device 100 can generate an MPD file including display area information in the projected picture as information for each sub-picture.

[0204] Therefore, the client can more easily select and play an appropriate stream according to, for example, the user's visual field based on the display area information.

[0205] <Use of information regarding the display area of the sub-picture signaled in the MPD file> Also, the selection of the stream may be performed using information regarding the display area of the sub-picture signaled in the MPD file.

[0206] That is, manage the image encoding data for each sub-picture in which the entire picture is divided into a plurality of sub-pictures and encoded, and obtain a control file used for distribution control of the image encoding data, including information regarding the area in the entire picture corresponding to the sub-picture as information different from the arrangement information for each picture area. Then, the selection of the stream of the image encoding data may be performed based on the information regarding the area included in the obtained control file.

[0207] For example, in the client device 200 which is an information processing device, the MPD file acquisition unit 212 functions as a file acquisition unit that manages the image encoding data for each sub-picture in which the entire picture is divided into a plurality of sub-pictures and encoded, and obtains a control file used for distribution control of the image encoding data, including information regarding the area in the entire picture corresponding to the sub-picture as information different from the arrangement information for each picture area. The MPD file processing unit 213 functions as an image processing unit that selects the stream of the image encoding data based on the information regarding the area included in the control file obtained by the file acquisition unit. That is, the information processing device (for example, the client device 200) may be provided with a file acquisition unit (for example, the MPD file acquisition unit 212) and an image processing unit (for example, the MPD file processing unit 213).

[0208] By doing so, the client device 200 can more easily select the stream.

[0209] Note that the above picture (overall picture) may be all or part of the all-sky image (a projected plane image obtained by projecting and mapping an image with a 360-degree horizontal circumference and an 180-degree vertical circumference). That is, even when the client device 200 acquires and plays back a stream obtained by sub-pictureizing all or part of the projected plane image as the overall picture, the present technology can be applied as described above.

[0210] Also, information regarding this area (display area information) may be included in the MPD file as information for each sub-picture (sub-picture). By doing so, the client device 200 can easily grasp which part of the overall picture the sub-picture corresponds to by simply referring to the information of the sub-picture referred to by the Adaptation Set.

[0211] <Flow of content playback processing> An example of the flow of content playback processing executed by the client device 200 in that case will be described with reference to the flowchart of FIG. 37.

[0212] When the content playback processing is started, the MPD file acquisition unit 212 of the client device 200 acquires, in step S221, an MPD file including display area information in the projected picture as information for each sub-picture.

[0213] In step S222, the display control unit 215 acquires the measurement result of the user's viewpoint position (and line-of-sight direction).

[0214] In step S223, the measurement unit 211 measures the transmission bandwidth of the network between the server and the client device 200.

[0215] In step S224, the MPD file processing unit 213 selects an Adaptation Set that refers to a sub-picture corresponding to the view of the user of the client device 200 based on the display area information in the projected picture of the sub-picture.

[0216] In step S225, the MPD file processing unit 213 selects a representation according to the viewpoint position and line-of-sight direction of the user, the transmission bandwidth of the network between the client and the server, etc. from among the Adaptation Sets selected in step S224.

[0217] In step S226, the segment file acquisition unit 214 acquires a segment file corresponding to the representation selected in step S225.

[0218] In step S227, the segment file processing unit 221 extracts encoded data from the segment file acquired in step S226.

[0219] In step S228, the decoding unit 222 decodes the encoded data of the stream extracted in step S227.

[0220] In step S229, the display information generation unit 223 plays back the stream (content) obtained by decoding in step S228. More specifically, the display information generation unit 223 generates data of a display image from the stream, supplies it to the display unit 217, and causes it to be displayed.

[0221] When the process of step S229 ends, the content playback process ends.

[0222] By performing the content playback process as described above, the client device 200 can more easily select a stream by using the information regarding the display area of the sub-picture included in the MPD file. For example, the client device 200 can easily select an appropriate stream according to the user's visual field based on that information.

[0223] <Definition by 2D Coverage Information descriptor> As described above, the MPD file generation unit 113 of the file generation device 100 newly defines and signals the display area information of the sub-picture indicating which part of the display in the projected picture the sub-picture referred to by the Adaptation set corresponds to in the OMAF's MPD file. That is, the MPD file generation unit 113 defines the display area information of the sub-picture as information for each sub-picture.

[0224] For example, the MPD file generation unit 113 defines a 2D Coverage Information descriptor as the display area information of the sub-picture and signals it as a descriptor different from the Region wise packing descriptor. For example, the MPD file generation unit 113 defines the Supplemental Property with @schemeIdUri = "urn:mpeg:mpegI:omaf:2017:2dco" as the 2D coverage information descriptor. Note that the MPD file generation unit 113 may define the 2D coverage information descriptor using the Essential Property with the same schemeIdUri.

[0225] That is, the image coding data for each sub-picture is managed for each adaptation set, the placement information for each picture area is stored in the Region-wise packing descripitor, and the display area information of the sub-picture (information regarding the area in the entire picture corresponding to the sub-picture referred to by the adaptation set) may be defined in the Supplemental Property or Essential Property of the MPD file.

[0226] Note that DASH clients that do not correspond to the schemeIdUri of the EssentialProperty must ignore the Adaptation Set (which may also be a Representation, etc.) in which this Property is written. Also, DASH clients that do not correspond to the schemeIdUri of the SupplementalProperty may ignore this Property value and use the AdaptationSet (which may also be a Representation, etc.).

[0227] Note that this 2D Coverage Information descriptor may also exist in the MPD or Representation in addition to the Adaptation Set. Also, this 2D Coverage Information descriptor is applicable even if the picture referred to by the Adaptation Set is not a sub-picture or if the region-wise packing process is not performed.

[0228] The attribute value 411 in Figure 38 shows an example of the attribute value of this 2D coverage information descriptor. As shown in the attribute value 411, omaf:@proj_picture_width has a data type of xs:unsignedInt and indicates the width of the projected picture. omaf:@proj_picture_height has a data type of xs:unsignedInt and indicates the height of the projected picture. omaf:@proj_reg_width has a data type of xs:unsignedInt and indicates the width of the region on the projected picture corresponding to the picture referred to by the Adaptation Set. omaf:@proj_reg_height has a data type of xs:unsignedInt and indicates the height of the region on the projected picture corresponding to the picture referred to by the Adaptation Set. omaf:@proj_reg_top has a data type of xs:unsignedInt and indicates the vertical coordinate of the region on the projected picture corresponding to the picture referred to by the Adaptation Set. omaf:@proj_reg_left has a data type of xs:unsignedInt and indicates the horizontal coordinate of the region on the projected picture corresponding to the picture referred to by the Adaptation Set.

[0229] Each of the above attribute values may be shown in terms of actual pixel counts, or omaf:@proj_reg_width, omaf:@proj_reg_height, omaf:@proj_reg_top, and omaf:@proj_reg_left may be shown as relative values with respect to omaf:@proj_picture_width and omaf:@proj_picture_height.

[0230] Also, information indicating that the overall picture is identical to the projected picture may be defined in the Supplemental Property or Essential Property of the MPD file. For example, the MPD file generation unit 113 defines the Supplemental Property of @schemeIdUri = "urn:mpeg:mpegI:omaf:2017:prid" shown in FIG. 74 as the Projected picture identical descriptor. For example, when this descriptor exists in the AdaptationSet, it indicates that the overall picture composed of the sub-pictures referred to by the AdaptationSet has not been subjected to region-wise packing processing and is identical to the projected picture.

[0231] At this time, the display area of the sub-picture referred to by the AdaptationSet in which the Projected picture identical descriptor exists may be indicated, for example, by the MPEG-DASH SRD (Spatial Relationship Description) indicating the display area of each region when the overall picture is divided into two or more regions and independently encoded. Although not shown, in the SRD, similar to the Sub Picture Region Box shown in the syntax 22 of FIG. 4, sub-picture division information indicating how the sub-picture is divided is shown.

[0232] At this time, in the AdaptationSet where the Projected picture identical descriptor exists, although not shown in the figure, the semantics of the object_x, object_y, object_width, object_height, total_width, and total_height, which are the attribute values of the SRD, are the same as the semantics of the omaf:@proj_reg_left, omaf:@proj_reg_top, omaf:@proj_reg_width, omaf:@proj_reg_height, omaf:@proj_picture_width, and omaf:@proj_picture_height, which are the attribute values of the 2D coverage information descriptor shown in the attribute value 441 of FIG. 38, respectively.

[0233] Note that this Projected picture identical descriptor may exist not only in the Adaptation Set but also in the MPD or Representation, or information indicating that the entire picture is identical to the projected picture may be defined by other descriptors, elements, and attributes.

[0234] <When the <sub-picture> contains a discontinuous area> Note that in the above example, the display area information when the <sub-picture> contains a discontinuous area on the projected picture cannot be signaled. Therefore, the 2D Coverage Information descriptor may be made capable of handling cases where the <sub-picture> contains a discontinuous area on the projected picture.

[0235] The attribute value 412 in Figure 39 shows an example of the attribute value of the 2D Coverage Information descriptor in that case. As shown in the attribute value 412, twoDCoverage is a container element whose data type is omaf:twoDCoverageType. twoDCoverage@proj_picture_width has a data type of xs:unsignedInt and indicates the width of the projected picture. twoDCoverage@proj_picture_height has a data type of xs:unsignedInt and indicates the height of the projected picture.

[0236] twoDCoverage.twoDCoverageInfo indicates an element whose data type is omaf:twoDCoverageInfoType and shows the region information on the projected picture. This attribute value can be signaled multiple times. twoDCoverage.twoDCoverageInfo@proj_reg_width has a data type of xs:unsignedInt and indicates the width of the region on the projected picture corresponding to the picture referenced by the Adaptation Set. twoDCoverage.twoDCoverageInfo@proj_reg_height has a data type of xs:unsignedInt and indicates the height of the region on the projected picture corresponding to the picture referenced by the Adaptation Set.

[0237] twoDCoverage.twoDCoverageInfo@proj_reg_top indicates the vertical coordinate of the area on the projected picture corresponding to the picture referenced by the Adaptation Set, with a data type of xs:unsignedInt. twoDCoverage.twoDCoverageInfo@proj_reg_left indicates the horizontal coordinate of the area on the projected picture corresponding to the picture referenced by the Adaptation Set, with a data type of xs:unsignedInt.

[0238] The data type 413 in Figure 40 shows an example of the definition of the data type of this 2D Coverage Information descriptor.

[0239] As described above, by making it possible to signal multiple areas on the projected picture, it becomes possible to signal the discontinuous display areas on the projected picture.

[0240] <Extension of Region-wise packing descriptor> The Region-wise packing descriptor defined in OMAF may be extended to signal the display area information on the projected picture of the sub-picture referenced by the Adaptation Set.

[0241] The attribute value 414 in Figure 41 shows an example of the attribute value of the Region-wise packing descriptor extended based on the signal of the attribute value 411 in Figure 38. The data type is the same as in Figure 40.

[0242] As shown in the attribute value 414, omaf: @packing_type has a data type of omaf: OptionallistofUnsignedByte and indicates the packing type of region - wise packing. If this attribute value is 0, it indicates packing of a rectangular region.

[0243] omaf: @proj_picture_width has a data type of xs: unsignedInt and indicates the width of the projected picture. omaf: @proj_picture_height has a data type of xs: unsignedInt and indicates the height of the projected picture. omaf: @proj_reg_width has a data type of xs: unsignedInt and indicates the width of the region on the projected picture corresponding to the picture referred to by the Adaptation Set. omaf: @proj_reg_height has a data type of xs: unsignedInt and indicates the height of the region on the projected picture corresponding to the picture referred to by the Adaptation Set.

[0244] omaf: @proj_reg_top has a data type of xs: unsignedInt and indicates the vertical coordinate of the region on the projected picture corresponding to the picture referred to by the Adaptation Set. omaf: @proj_reg_left has a data type of xs: unsignedInt and is the horizontal coordinate of the region on the projected picture corresponding to the picture referred to by the Adaptation Set.

[0245] Each of the above attribute values may be indicated by the actual number of pixels, or omaf:@proj_reg_width, omaf:@proj_reg_height, omaf:@proj_reg_top, and omaf:@proj_reg_left may be indicated by relative values with respect to omaf:@proj_picture_width and omaf:@proj_picture_height.

[0246] The attribute value 415 in Figure 42 shows an example of the attribute value of the Region-wise packing descriptor extended based on the signal of the attribute value 412 in Figure 39, that is, the attribute value of the Region-wise packing descriptor corresponding to the case including discontinuous regions. The data type is the same as in Figure 40.

[0247] As shown in the attribute value 415, omaf: @packing_type indicates the packing type of region-wise packing where the data type is omaf:OptionallistofUnsignedByte. When this attribute value is 0, it indicates packing of a rectangular region.

[0248] twoDCoverage is a container element whose data type is omaf:twoDCoverageType. twoDCoverage@proj_picture_width has a data type of xs:unsignedInt and indicates the width of the projected picture. twoDCoverage@proj_picture_height has a data type of xs:unsignedInt and indicates the height of the projected picture. twoDCoverage.twoDCoverageInfo has a data type of omaf:twoDCoverageInfoType and indicates an element showing region information on the projected picture. This attribute value can be signaled multiple times.

[0249] twoDCoverage.twoDCoverageInfo@proj_reg_width has a data type of xs:unsignedInt and indicates the width of the area on the projected picture corresponding to the picture referenced by the Adaptation Set. twoDCoverage.twoDCoverageInfo@proj_reg_height has a data type of xs:unsignedInt and indicates the height of the area on the projected picture corresponding to the picture referenced by the Adaptation Set.

[0250] twoDCoverage.twoDCoverageInfo@proj_reg_top has a data type of xs:unsignedInt and indicates the vertical coordinate of the area on the projected picture corresponding to the picture referenced by the Adaptation Set. twoDCoverage.twoDCoverageInfo@proj_reg_left has a data type of xs:unsignedInt and indicates the horizontal coordinate of the area on the projected picture corresponding to the picture referenced by the Adaptation Set.

[0251] <Extension of the Content coverage descriptor> Alternatively, the Content coverage descriptor defined in OMAF that indicates the display area on the spherical surface of the Adaptation Set may be extended to signal the display area on the projected picture.

[0252] That is, the image coding data for each sub-picture is managed for each adaptation set, the arrangement information for each picture area is stored in the Region-wise packing descripitor, and the display area information of the sub-picture (information regarding the area in the entire picture corresponding to the sub-picture referred to in the adaptation set) may be defined in the Coverage Information descriptor indicating the display area on the sphere of the Adaptation Set in the MPD file.

[0253] The attribute value 416 in FIG. 43 and the attribute value 417 in FIG. 44 show examples of the attribute values of the extended Content coverage descriptor. When extending the Content coverage descriptor, similar to the case of extending the Content coverage Box in the above-described ISOBMFF file, the 2D_coverage_flag attribute is used to switch between signaling the area on the sphere and signaling the display area on the projected picture.

[0254] As shown in the attribute value 416, cc is a container element whose data type is omaf:CCType. cc@2D_coverage_flag is flag information whose data type is xs:boolean and indicates whether the display area is defined on the sphere or on the projected picture. When this attribute value is 0, it indicates that it is defined on the sphere, and when this value is 1, it indicates that it is defined on the projected picture.

[0255] cc.sphericalCoverage is a container element for spherical display area information with a data type of omaf:sphericalCoverageType. This element exists only when cc@2D_coverage_flag = 0. cc.sphericalCoverage @shape_type has a data type of xs:unsignedByte and indicates the shape of the area on the sphere. If the value of this attribute is 0, it indicates an area surrounded by 4 great circles. If the value of this attribute is 1, it indicates an area surrounded by 2 azimuth circles and 2 elevation angles.

[0256] cc.sphericalCoverage @view_idc_presence_flag has a data type of xs:boolean and is flag information indicating whether the view_idc attribute exists. If the value of this attribute is 0, it indicates that the view_idc attribute does not exist. If the value of this attribute is 1, it indicates that the view_idc attribute exists.

[0257] cc.sphericalCoverage @default_view_idc has a data type of omaf:ViewType and indicates a view that is common to all regions. For example, when this attribute value is 0, it indicates that the view type (view_idc) of all regions included in the sub-picture is a mono view. Also, when this attribute value is 1, it indicates that the view type (view_idc) of all regions included in the sub-picture is a left view. Also, when this attribute value is 2, it indicates that the view type (view_idc) of all regions included in the sub-picture is a right view. Also, when this attribute value is 3, it indicates that the view type (view_idc) of all regions included in the sub-picture is a stereo view. This attribute value must always exist when cc@view_idc_presence_flag = 0. Also, when cc@view_idc_presence_flag = 1, this attribute must not exist.

[0258] cc.sphericalCoverage.coverageInfo has a data type of omaf:coverageInfoType and is an element that indicates spherical region information. This element can signal multiple times.

[0259] cc.sphericalCoverage.coverageInfo@view_idc has a data type of omaf:ViewType and indicates the view for each region. For example, when this attribute value is 0, it indicates that the view type (view_idc) of the corresponding region is a mono view. When this attribute value is 1, it indicates that the view type (view_idc) of the corresponding region is a left view. When this attribute value is 2, it indicates that the view type (view_idc) of the corresponding region is a right view. When this attribute value is 3, it indicates that the view type (view_idc) of the corresponding region is a stereo view. This attribute value shall not exist when cc@view_idc_presence_flag = 0. Also, when cc@view_idc_presence_flag = 1, this attribute must exist.

[0260] cc.sphericalCoverage.coverageInfo@center_azimuth has a data type of omaf:Range1 and indicates the azimuth angle of the center of the display area on the spherical surface. cc.sphericalCoverage.coverageInfo@center_elevation has a data type of omaf:Range2 and indicates the elevation angle of the center of the display area on the spherical surface. cc.sphericalCoverage.coverageInfo@center_tilt has a data type of omaf:Range1 and indicates the tilt angle of the center of the display area on the spherical surface. cc.sphericalCoverage.coverageInfo@azimuth_range has a data type of omaf:HRange and indicates the azimuth range of the display area on the spherical surface. cc.sphericalCoverage.coverageInfo@elevation_range has a data type of omaf:VRange and indicates the elevation range of the display area on the spherical surface.

[0261] cc.twoDCoverage is a container element for display area information on a projected picture where the data type is omaf:twoDCoverageType. It exists only when cc@2D_coverage_flag = 1.

[0262] cc.twoDCoverage@proj_picture_width has a data type of xs:unsignedInt and indicates the width of the projected picture. cc.twoDCoverage@proj_picture_height has a data type of xs:unsignedInt and indicates the height of the projected picture. cc.twoDCoverage.twoDCoverageInfo has a data type of omaf:twoDCoverageInfoType and is an element indicating area information on the projected picture. This element can signal multiple times.

[0263] cc.twoDCoverage.twoDCoverageInfo@proj_reg_width has a data type of xs:unsignedInt and indicates the width of the area on the projected picture corresponding to the picture referenced by the Adaptation Set. cc.twoDCoverage.twoDCoverageInfo@proj_reg_height has a data type of xs:unsignedInt and indicates the height of the area on the projected picture corresponding to the picture referenced by the Adaptation Set.

[0264] cc.twoDCoverage.twoDCoverageInfo@proj_reg_top indicates the vertical coordinate of the area on the projected picture corresponding to the picture referenced by the Adaptation Set, with a data type of xs:unsignedInt. cc.twoDCoverage.twoDCoverageInfo@proj_reg_left indicates the horizontal coordinate of the area on the projected picture corresponding to the picture referenced by the Adaptation Set, with a data type of xs:unsignedInt.

[0265] The data types 418 in Figure 45, 419 in Figure 46, and 420 in Figure 47 show examples of the definitions of the data types of this extended Content coverage descriptor.

[0266] <Signaling when the splitting method of <sub-picture> changes dynamically> In addition, when the display area in the projected picture changes dynamically in the stream, in addition to the above signaling, flags may be signaled additionally in the 2D coverage information descriptor, region-wise packing descriptor, and Content coverage descriptor to indicate that the stream is one in which the display area in the projected picture changes dynamically.

[0267] <4. The Third Embodiment> <Signaling of Stereo Information> As described above in the <Identification of Stereo Information> of the <Signal of Information Regarding 1 Sub-Picture>, when the entire picture of the stereo omnidirectional video is sub-pictured, the stereo information of the entire picture is signaled by the Stereo Video Box signaled under the Sub Picture Composition Box, and the stereo information of the sub-picture is signaled by the Stereo Video Box under the Scheme Information Box of the Sample Entry.

[0268] When the entire picture is a stereo image, there are, for example, the following three patterns as variations of the sub-picture that can be generated.

[0269] FIG. 48 is a diagram showing an example of the state of the first pattern of sub-picturing. In this case, the projected picture (entire picture) 431 is sub-pictured to generate sub-pictures 432 to 437. The projected picture 431 is composed of side-by-side stereo images. Each of the sub-pictures 432 to 437 includes a Left view and a Right view that are the same display area on the projected picture 431, and has a frame packing arrangement that can be signaled by a Stereo Video Box such as top & bottom or side by side.

[0270] FIG. 49 is a diagram showing an example of the state of the second pattern of sub-picture conversion. In this case, the projected picture (entire picture) 441 is sub-picture converted to generate sub-pictures 442 to 446. The projected picture 441 is composed of side by side stereo images. Each of sub-pictures 442 to 446 includes a Left view and a Right view, but the display areas of each view on the projected picture 441 do not match, and it does not become a frame packing arrangement that can be signaled by a Stereo Video Box such as top & bottom or side by side.

[0271] FIG. 50 is a diagram showing an example of the state of the third pattern of sub-picture conversion. In this case, the projected picture (entire picture) 451 is sub-picture converted to generate sub-pictures 452 and 453. The projected picture 451 is composed of side by side stereo images. Sub-picture 452 is a monochrome picture including only the Left view. Sub-picture 453 is a monochrome picture including only the Right view.

[0272] In the case of the first pattern, a Stereo Video Box is signaled in the Sample Entry / rinf / schi of the sub-picture track, and appropriate frame packing arrangement information is signaled. For example, in the case of FIG. 48, it is side by side. Note that instead of the frame packing arrangement information of the sub-picture, the frame packing arrangement information of the entire picture may be signaled.

[0273] In the case of the second pattern, the Stereo Video Box is not signaled in Sample Entry / rinf / schi. Therefore, there was a possibility that the client could not identify whether the sub-picture was a monochrome picture or included Left view and Right view but no frame packing arrangement such as top & bottom or side by side was applied.

[0274] In the case of the third pattern, the Stereo Video Box is not signaled in Sample Entry / rinf / schi. Therefore, there was a possibility that the client could not identify whether the entire picture before splitting was a monochrome picture or a stereo image. Since whether upscaling is required during rendering depends on whether the entire picture before splitting is a monochrome picture or a stereo image, if the identification could not be made, there was a possibility that the client could not render appropriately.

[0275] For example, as shown in FIG. 50, for a sub-picture having only the Left view of a side by side stereo image as the entire picture, 2-fold upscaling in the horizontal direction is required during rendering. In contrast, this process is not necessary when the entire picture is a monochrome picture.

[0276] <Signal of stereo information of the entire picture sub-pictured by ISOBMFF> Therefore, stereo information, which is information regarding the stereo display of the entire picture to be sub-pictured, may be provided in the ISOBMFF file, which is a segment file.

[0277] That is, the entire picture may be divided into a plurality of sub-pictures, and the image data for each of the encoded sub-pictures may be stored in tracks with respect to each other, and a file including stereo information, which is information regarding the stereo display of the entire picture, may be generated.

[0278] For example, in a file generation device 100 which is an information processing device, a segment file generation unit 123 may function as a file generation unit that stores the image data for each of the sub-pictures obtained by dividing the entire picture into a plurality of sub-pictures and encoding them in tracks with respect to each other, and generates a file including stereo information, which is information regarding the stereo display of the entire picture. That is, the information processing device (for example, the file generation device 100) may be provided with a file generation unit (for example, the segment file generation unit 123).

[0279] By doing so, as described above, the client can more easily select a stream based on this information.

[0280] Also, the above-described picture (entire picture) may be all or part of an omnidirectional image (a projection plane image obtained by projecting and mapping an image of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction). That is, when the file generation device 100 uses all or part of the projection plane image as the entire picture and sub-pictures it, the present technology can be applied as described above.

[0281] By doing so, even when distributing an omnidirectional image, as described above, the client can more easily select a stream based on this information.

[0282] Note that the stereo information of this entire picture may be included in the ISOBMFF file as information for each sub-picture. By doing so, the client can easily grasp the stereo information of the entire picture (e.g., whether it is a stereo image of the entire picture and what type of stereo image it is, etc.) just by referring to the information of the sub-picture track.

[0283] <Flow of Upload Process> An example of the flow of the upload process executed by the file generation device 100 in FIG. 11 in that case will be described with reference to the flowchart of FIG. 51.

[0284] When the upload process is started, the data input unit 111 of the file generation device 100 acquires an image and metadata in step S301.

[0285] In step S302, the segment file generation unit 123 generates an ISOBMFF file including the stereo information of the entire picture (projected picture) as information for each sub-picture.

[0286] In step S303, the recording unit 114 records the ISOBMFF file generated by the process of step S302.

[0287] In step S304, the upload unit 115 reads the ISOBMFF file recorded in step S303 from the recording unit 114 and uploads it to the server.

[0288] When the process of step S304 ends, the upload process ends.

[0289] By performing the upload process as described above, the file generation device 100 can generate an ISOBMFF file including the stereo information of the projected picture as information for each sub-picture.

[0290] Therefore, based on that information, the client can more easily select and play an appropriate stream according to its own capabilities, etc.

[0291] <Use of stereo information of the entire picture to be sub-pictured signaled in ISOBMFF> Also, the selection and playback of the stream may be performed using the stereo information of the entire picture to be sub-pictured, which is signaled in the ISOBMFF file.

[0292] That is, the image data for each sub-picture in which the entire picture is divided into a plurality of sub-pictures and encoded is stored in tracks with each other, and a file including stereo information, which is information regarding the stereo display of the entire picture, is acquired, and based on the stereo information included in the acquired file, the selection of the stream of the image encoded data may be performed.

[0293] For example, in the client device 200 which is an information processing device, the segment file acquisition unit 214 functions as a file acquisition unit that stores the image data for each sub-picture in which the entire picture is divided into a plurality of sub-pictures and encoded in tracks with each other, and acquires a file including stereo information, which is information regarding the stereo display of the entire picture, and the data analysis / decoding unit 216 functions as an image processing unit that selects the stream of the image encoded data based on the stereo information included in the file acquired by the file acquisition unit. That is, the information processing device (for example, the client device 200) may be provided with a file acquisition unit (for example, the segment file acquisition unit 214) and an image processing unit (for example, the data analysis / decoding unit 216).

[0294] By doing so, the client device 200 can more easily select a stream.

[0295] Note that the above-described picture (overall picture) may be all or part of an omnidirectional image (a projection plane image obtained by projecting and mapping images of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction). That is, even when the client device 200 acquires and plays back a stream in which all or part of the projection plane image is sub-pictureized as an overall picture, the present technology can be applied as described above.

[0296] Also, the stereo information of this overall picture may be included in the ISOBMFF file as information for each sub-picture. By doing so, the client device 200 can easily grasp the stereo information of the overall picture (for example, whether it is an overall picture stereo image and what type of stereo image it is, etc.) just by referring to the information of the sub-picture track.

[0297] <Flow of content playback processing> An example of the flow of content playback processing executed by the client device 200 in that case will be described with reference to the flowchart of FIG. 52.

[0298] When the content playback processing is started, the segment file acquisition unit 214 of the client device 200 acquires, in step S321, an ISOBMFF file including the stereo information of the overall picture (projected picture) as information for each sub-picture.

[0299] In step S322, the display control unit 215 acquires the measurement result of the user's viewpoint position (and line-of-sight direction).

[0300] In step S323, the measurement unit 211 measures the transmission bandwidth of the network between the server and the client device 200.

[0301] In step S324, the segment file processing unit 221 determines whether the client device 200 performs stereo playback (or has a function (capability) of performing stereo playback). If it is determined that stereo playback is performed (or has a stereo playback function), the process proceeds to step S325.

[0302] In step S325, the segment file processing unit 221 sets a sub-picture track capable of stereo display as a selection candidate. At this time, the segment file processing unit 221 can also include, as selection candidates, sub-picture tracks capable of stereo display to which the second pattern of frame packing arrangement is not applied (for example, sub-picture 445 or sub-picture 446 in FIG. 49) by referring to the stereo information of the overall picture included in the ISOBMFF file acquired in step S321. When the process of step S325 ends, the process proceeds to step S327.

[0303] Also, in step S324, if it is determined that the client device 200 does not perform stereo playback (or does not have a stereo playback function), the process proceeds to step S326.

[0304] In step S326, based on the stereo information of the overall picture included in the ISOBMFF file acquired in step S321, the segment file processing unit 221 selects the sub-picture track of the monochrome picture as a selection candidate. At this time, by referring to the stereo information of the overall picture included in the ISOBMFF file acquired in step S321, the segment file processing unit 221 can grasp that for the sub-picture track of the third pattern (for example, sub-picture 452 or sub-picture 453 in FIG. 50), it is necessary to perform upscale by a factor of 2 in the horizontal direction during rendering. When the processing of step S326 is completed, the processing proceeds to step S327.

[0305] In step S327, the segment file processing unit 221 selects a sub-picture track corresponding to the view of the user of the client device 200 from among the candidates set in step S325 or step S326.

[0306] In step S328, the segment file processing unit 221 extracts the encoded data of the stream of the sub-picture track selected in step S327 from the ISOBMFF file acquired in step S321.

[0307] In step S329, the decoding unit 222 decodes the encoded data of the stream extracted in step S328.

[0308] In step S330, the display information generation unit 223 plays back the stream (content) decoded in step S329. More specifically, the display information generation unit 223 generates data of the display image from the stream, supplies it to the display unit 217, and causes it to be displayed.

[0309] When the processing of step S330 is completed, the content playback processing is completed.

[0310] By performing the content playback process as described above, the client device 200 can more easily select a stream by using the stereo information of the entire picture to be sub-pictured included in the ISOBMFF file. For example, the client device 200 can more easily select and play an appropriate stream according to its own capabilities based on the information.

[0311] <Signal the stereo information of the entire picture to the Sample Entry> For example, when the entire picture before segmentation is a stereo image, the segment file generation unit 123 of the file generation device 100 may signal the stereo information of the entire picture in the Sample Entry of the sub-picture track.

[0312] For example, the stereo information of the entire picture may be stored in the Scheme Information Box below the Sample Entry of the ISOBMFF file, or in a box in the lower hierarchy of the Scheme Information Box.

[0313] <Original Stereo Video Box> For example, in order to signal the stereo information of the entire picture before segmentation, the segment file generation unit 123 of the file generation device 100 may newly define the Original Stereo Video Box and signal the box below the Scheme Information Box (schi) of the Sample Entry of the sub-picture track. That is, the stereo information of the entire picture before segmentation may be stored in this Original Stereo Video Box.

[0314] Note that the location of this Original Stereo Video Box is arbitrary and is not limited to the above-mentioned Scheme Information Box. Also, the information signaled in the Original Stereo Video Box is the same as that in the Stereo Video Box.

[0315] The syntax 461 in FIG. 53 shows an example of the syntax of this Original Stereo Video Box. As shown in syntax 461, in the Original Stereo Video Box, fields such as single_view_allowed, stereo_scheme, length, and stereo_indication_type are defined.

[0316] The semantics 462 in FIG. 54 shows an example of the semantics of the fields defined in this Original Stereo Video Box. As shown in semantics 462, single_view_allowed is information indicating the type of view allowed. For example, when the value of this field is 0, it indicates that the content is intended to be displayed only on a stereoscopic-compatible display. Also, when the value of this field is 1, it indicates that the content is allowed to be displayed as the right view on a monoscopic display. Further, when the value of this field is 2, it indicates that the content is allowed to be displayed as the left view on a monoscopic display.

[0317] The stereo_scheme is information regarding the frame packing method. For example, if the value of this field is 1, it indicates that the frame packing method follows the Frame packing arrangement SEI in ISO / IEC 14496-10. Also, if the value of this field is 2, it indicates that the frame packing method follows Annex.L in ISO / IEC 13818-2. Further, if the value of this field is 3, it indicates that the frame packing method follows frame / service compatible and 2D / 3D Mixed service in ISO / IEC 23000-11.

[0318] length indicates the byte length of the stereo_indication_type. Also, the stereo_indication_type indicates the frame packing method according to the stereo_shceme.

[0319] The segment file processing unit 221 of the client device 200 can obtain the stereo information of the entire picture by referring to the Original Stereo Video Box and the 2D Coverage Information Box. Then, based on this information, when the Stereo Video Box is not signaled in the Sample Entry of the sub-picture track, the segment file processing unit 221 can easily identify whether the sub-picture is mono or, although it includes the Left view and the Right view, the frame packing arrangement signaled by the Stereo Video Box is not applied, without parsing the Sub Picture Composition Box. That is, similar to the case of a track that is not a sub-picture track, the stereo information can be identified only from the information stored in the Sample Entry.

[0320] That is, the client device 200 can select and play a sub-picture track alone without parsing the Sub Picture Composition Box.

[0321] <Signal of display size> Furthermore, the width and height of the Track Header of the sub-picture track that stores the mono sub-picture (sub-picture that is a mono picture) generated by dividing the stereo whole picture may signal the display size upscaled based on the frame packing arrangement of the whole picture.

[0322] That is, the ISOBMFF file may include information regarding the display size of the sub-picture.

[0323] An example of this is shown in FIG. 55. As shown in FIG. 55, in the sub-pictures 471 and 472 generated from the whole picture of the stereo image, the images in each area are downscaled horizontally. Therefore, at the time of display (rendering), it is necessary to upscale horizontally. Thus, the display size of the image 473 at the time of this display is signaled as the width and height of the Track Header. Thereby, the client device 200 can appropriately render the mono sub-picture.

[0324] Note that instead of signaling in the width and height of this Track Header, a Pixel Aspect Ratio Box (pasp) that signals the pixel aspect ratio information at the time of display, defined in ISOBMFF, may be signaled to the Visual Sample Entry. Also in this case, the same effect as when signaling in the width and height of the above-mentioned Track Header can be obtained.

[0325] The syntax 481 in FIG. 56 shows an example of the syntax of the Pixel Aspect Ratio Box. As shown in the syntax 481, in the Pixel Aspect Ratio Box, fields such as hSpacing and vSpacing are defined, for example.

[0326] The semantics 482 in FIG. 57 shows an example of the semantics of the fields defined in this Pixel Aspect Ratio Box. As shown in the semantics 482, hSpacing and vSpacing are information indicating the relative pixel height and width. During rendering, based on this information, the pixel width is multiplied by vSpace / hSpace and displayed.

[0327] An example of this is shown in FIG. 58. The sub-pictures 491 and 492 shown in FIG. 58 are sub-pictures generated from the entire picture of the stereo image, and the images in each area are downscaled horizontally. Therefore, by signaling hSpace = 1 and vSpase = 2, when the client device 200 displays (renders) the sub-picture 491, for example, based on this information, it can double the pixel width and render it to display the image with an appropriate aspect ratio like the image 493. That is, the client device 200 can properly render a mono sub-picture (a sub-picture consisting of a mono image).

[0328] <Original Scheme Information Box> Also, an Original Scheme Information Box may be newly defined under the Restricted Scheme Information Box (rinf) of the Sample Entry of the sub-picture track, and the stereo information of the entire picture before splitting may be signaled within that box. Note that the location where the Original Scheme Information Box is defined is arbitrary and is not limited to the above-mentioned rinf.

[0329] Syntax 501 in FIG. 59 shows an example of the syntax of the Original Scheme Information Box in that case. As shown in Syntax 501, in the Original Scheme Information Box, for example, scheme_specific_data is defined.

[0330] This scheme_specific_data signals information about the entire picture before it is split into sub-pictures. For example, if the entire picture is stereo, a Stereo Video Box with the stereo information of the entire picture may be signaled. By doing so, the client device 200 can select and play only the sub-picture track without parsing the Sub Picture Composition Box.

[0331] Note that this scheme_specific_data may signal not only the Stereo Video Box but also post-processing information about the entire picture (for example, Region Wise Packing Box, etc.).

[0332] <Additional information to facilitate sub-picture track selection> Furthermore, the 2D Coverage Information Box may be extended to signal stereo-related information that facilitates track selection.

[0333] For example, a stereo_presentation_suitable_flag may be added to the 2D Coverage Information Box to signal whether the sub-picture track is capable of stereo display. By referring to this information, the client device 200 can determine whether stereo display is possible without performing a process of identifying whether stereo display is possible based on the stereo information of the overall picture described above in the third embodiment and the region information on the projected picture signaled in the 2D Coverage Information Box defined in the first embodiment.

[0334] That is, the ISOBMFF file may further include sub-stereo information, which is information regarding stereo display for each sub-picture.

[0335] The syntax 502 in FIG. 60 shows an example of the 2D Coverage Information Box in this case. As shown in the syntax 502, in this case, stereo_presentation_suitable is further defined (stereo_presentation_suitable is added to the defined field).

[0336] The semantics 503 in FIG. 61 shows an example of the semantics of this added field. As shown in the semantics 503, stereo_presentation_suitable is information regarding stereo display of the picture of the track. When the value of this field is 0, it indicates that the picture of the track is mono or includes an L view and an R view but is not capable of stereo display. When the value of this field is 1, it indicates that the picture of the track is capable of stereo display in some regions. When the value of this field is 2, it indicates that all regions of the picture of the track are capable of stereo display.

[0337] Figure 62 is a diagram showing signal examples of stereo_presentation_suitable. For example, as shown in the upper part of Figure 62, the overall picture (projected picture) 511, which is a side by side stereo image, is sub-pictured to generate sub-pictures 512 to 517, which include the L view and the R view and can be stereoscopically displayed. Therefore, stereo_presentation_suitable = 2 is set for these sub-pictures.

[0338] On the other hand, as shown in the lower part of Figure 62, the overall picture (projected picture) 521, which is a side by side stereo image, is sub-pictured to generate sub-pictures 522 to 526, which include the L view and the R view. However, sub-pictures 522 to 524 cannot be stereoscopically displayed. Therefore, stereo_presentation_suitable = 0 is set for these sub-pictures.

[0339] Also, sub-pictures 525 and 526 can stereoscopically display some areas. Therefore, stereo_presentation_suitable = 1 is set for these sub-pictures.

[0340] Note that information on whether or not it can be stereoscopically displayed may be newly defined in an individual Box (a dedicated Box for storing that information), for example, a Track Stereo Video Box, and signaled below the schi of the sub-picture track.

[0341] The syntax 531 in Figure 63 shows an example of the syntax of the Track Stereo Video Box in that case.

[0342] Also, a Region Wise Packing Box signaled under the Projected Omnidirectional Video Box of the Sample Entry of the sub-picture track, or an extension of the RectProjected Region structure described above in the first embodiment may be used to signal the stereo_presentation_suitable_flag. Also, the stereo_presentation_suitable_flag may be signaled in other Boxes.

[0343] Furthermore, in the case where stereo display is not possible, the track_not_intended_for_presentation_alone flag of the Track Header Box may be used to signal that sub-picture alone playback is not desirable.

[0344] Note that the above various types of information are also applicable when the picture stored in the track is not a sub-picture.

[0345] <Signal of <view> information> Also, the 2D Coverage Information Box may be extended to additionally signal <view> information for the display area on the projected picture of the sub-picture. By referring to this information, the client device 20 can easily identify whether the sub-picture is a monochrome image or includes the L view and the R view without performing the process of identifying the <view> information of each area from the stereo information of the overall picture described above in the third embodiment and the area information on the projected picture signaled in the 2D Coverage Information Box defined in the third embodiment.

[0346] That is, the ISOBMFF file may further include view information indicating the view type of the subpicture.

[0347] The syntax 532 in FIG. 64 shows an example of the syntax of the 2D Coverage Information Box in this case. As shown in the syntax 532, in this case, fields such as view_idc_presence_flag, default_view_idc, and view_idc are additionally defined in the 2D Coverage Information Box.

[0348] The semantics 533 in FIG. 65 shows an example of the semantics of the fields additionally defined in this 2D Coverage Information Box. As shown in the semantics 533, the view_idc_presense_flag indicates whether there is an individual view_idc for each region. For example, when the value of this field is 0, it indicates that there is no individual view_idc for each region. Also, when the value of this field is 1, it indicates that there is an individual view_idc for each region.

[0349] That is, the ISOBMFF file may further include information indicating whether view information exists for each region.

[0350] The default_view_idc indicates a view common to all regions. For example, when the value of this field is 0, it indicates that all regions in this subpicture are mono views. Also, when the value of this field is 1, it indicates that all regions in this subpicture are left views. Also, when the value of this field is 2, it indicates that all regions in this subpicture are right views. Also, when the value of this field is 3, it indicates that all regions in this subpicture are stereo views.

[0351] view_idc indicates the view for each region. For example, if the value of this field is 0, it indicates that the region is a mono view. If the value of this field is 1, it indicates that the region is a left view. If the value of this field is 2, it indicates that the region is a right view. If the value of this field is 3, it indicates that the region is a stereo view. If this field does not exist, it indicates that default_view_idc indicates the view for each region.

[0352] That is, the view information may be information for each region included in the sub-picture.

[0353] Figure 66 is a diagram showing an example of signaling this view_idc. As shown in Figure 66, when the entire side by side picture 541 is sub-pictured like sub-picture 542 and sub-picture 543, view_idc = 3 is set for each of these sub-pictures.

[0354] On the other hand, when the entire picture 541 is sub-pictured like sub-picture 544 and sub-picture 545, view_idc = 1 is set for sub-picture 544 and view_idc = 2 is set for sub-picture 545.

[0355] Note that even when the picture stored in the track is not a sub-picture, these additional information can be applied.

[0356] Similarly, the Region Wise Packing Box and the Rect Projected Region structure defined in the first embodiment may be extended to signal view information.

[0357] <5. Fourth Embodiment> <Signal of stereo information of the entire picture to be sub-pictured by the MPD file> The signal of the stereo information of the entire picture to be sub-pictured as described above may be performed in the MPD file. That is, in order to enable a client to select and play an Adaptation Set that refers to a sub-picture according to, for example, the capability of the client, in the MPD file, the stereo information of the entire picture to be sub-pictured may be newly defined and signaled to the Adaptation Set.

[0358] That is, for each adaptation set, manage the image encoding data for each sub-picture in which the entire picture is divided into a plurality of sub-pictures and encoded, and generate a control file used for distribution control of the image encoding data, including stereo information which is information regarding stereo display of the adaptation set.

[0359] For example, in a file generation device 100 which is an information processing device, the MPD file generation unit 113 may function as a file generation unit that manages, for each adaptation set, the image encoding data for each sub-picture in which the entire picture is divided into a plurality of sub-pictures and encoded, and generates a control file used for distribution control of the image encoding data, including stereo information which is information regarding stereo display of the adaptation set. That is, the information processing device (for example, the file generation device 100) may be provided with a file generation unit (for example, the MPD file generation unit 113).

[0360] By doing so, as described above, the client can more easily select a stream based on this information.

[0361] Further, the above-mentioned picture (overall picture) may be all or part of the omnidirectional image (a projected plane image obtained by projecting and mapping an image of 360 degrees around the horizontal direction and 180 degrees around the vertical direction). That is, when the file generation device 100 uses all or part of such a projected plane image as the overall picture and sub-pictures it, the present technology can be applied as described above.

[0362] By doing so, even when distributing an omnidirectional image, as described above, the client can more easily select a stream based on this information.

[0363] Note that the information regarding this area (display area information) may be included in the MPD file as information for each sub-picture (sub-picture). By doing so, the client can easily grasp which part of the overall picture the sub-picture corresponds to by simply referring to the information of the sub-picture referred to by the Adaptation Set.

[0364] <Flow of Upload Process> An example of the flow of the upload process executed by the file generation device 100 in FIG. 11 in that case will be described with reference to the flowchart of FIG. 67.

[0365] When the upload process is started, the data input unit 111 of the file generation device 100 acquires an image and metadata in step S401.

[0366] In step S402, the segment file generation unit 123 generates a segment file of the image.

[0367] In step S403, the MPD file generation unit 113 generates an MPD file including stereo information of the overall picture (projected picture) as information for each sub-picture.

[0368] In step S404, the recording unit 114 records the segment file generated by the process of step S402. Further, the recording unit 114 records the MPD file generated by the process of step S403.

[0369] In step S405, the upload unit 115 reads out the segment file recorded in step S404 from the recording unit 114 and uploads it to the server. Further, the upload unit 115 reads out the MPD file recorded in step S404 from the recording unit 114 and uploads it to the server.

[0370] When the process of step S404 ends, the upload process ends.

[0371] By performing the upload process as described above, the file generation device 100 can generate an MPD file including the stereo information of the entire picture as information for each sub-picture.

[0372] Therefore, the client can more easily select and play an appropriate stream according to, for example, the function (capability) of the client device 200 based on the display area information.

[0373] <Use of stereo information of the entire picture to be sub-pictured signaled in the MPD file> Also, the selection of the stream may be performed using the stereo information of the entire picture to be sub-pictured, which is signaled in the MPD file.

[0374] That is, the entire picture is divided into a plurality of sub-pictures, and the image encoding data for each sub-picture thus encoded is managed for each adaptation set. A control file used for distribution control of the image encoding data is obtained, which includes stereo information that is information regarding stereo display of the adaptation set. Based on the stereo information included in the obtained control file, selection of a stream of the image encoding data may be performed.

[0375] For example, in a client device 200 which is an information processing device, an MPD file acquisition unit 212 may function as a file acquisition unit that obtains a control file used for distribution control of the image encoding data, which manages the image encoding data for each sub-picture where the entire picture is divided into a plurality of sub-pictures and encoded, for each adaptation set, and includes stereo information that is information regarding stereo display of the adaptation set. An MPD file processing unit 213 may function as an image processing unit that selects a stream of the image encoding data based on the stereo information included in the obtained control file. That is, the information processing device (for example, the client device 200) may be provided with a file acquisition unit (for example, the MPD file acquisition unit 212) and an image processing unit (for example, the MPD file processing unit 213).

[0376] By doing so, the client device 200 can more easily select a stream.

[0377] Note that the above-mentioned picture (entire picture) may be all or a part of an omnidirectional image (a projection plane image obtained by projecting and mapping an image of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction). That is, even when the client device 200 obtains and reproduces a stream in which all or a part of the projection plane image is sub-pictured as an entire picture, the present technology can be applied as described above.

[0378] Also, information regarding this area (display area information) may be included in the MPD file as information for each sub-picture. By doing so, the client device 200 can easily grasp which part of the entire picture the sub-picture corresponds to by simply referring to the information of the sub-picture referred to by the Adaptation Set.

[0379] <Flow of content playback processing> An example of the flow of content playback processing executed by the client device 200 in that case will be described with reference to the flowchart of FIG. 68.

[0380] When the content playback processing is started, the MPD file acquisition unit 212 of the client device 200 acquires, in step S421, an MPD file including stereo information of the projected picture as information for each sub-picture.

[0381] In step S422, the display control unit 215 acquires the measurement result of the user's viewpoint position (and line-of-sight direction).

[0382] In step S423, the measurement unit 211 measures the transmission bandwidth of the network between the server and the client device 200.

[0383] In step S424, the MPD file processing unit 213 determines whether the client device 200 performs stereo playback (or has the capability to perform stereo playback). If it is determined that stereo playback is performed (or has the function of stereo playback), the process proceeds to step S425.

[0384] In step S425, the MPD file processing unit 213 selects, as candidate selections, Adaptation Sets that reference sub-pictures capable of stereo display. At this time, the MPD file processing unit 213 can also include, as candidate selections, Adaptation Sets that reference stereo-displayable sub-pictures (for example, sub-picture 445 and sub-picture 446 in FIG. 49) to which the second pattern of frame packing arrangement is not applied, by referring to the stereo information of the overall pictures included in the MPD file acquired in step S421. When the processing of step S425 ends, the processing proceeds to step S427.

[0385] Also, in step S424, when it is determined that the client device 200 does not perform stereo playback (or does not have a stereo playback function), the processing proceeds to step S426.

[0386] In step S426, the MPD file processing unit 213 selects, as candidate selections, Adaptation Sets that reference mono-picture sub-pictures based on the stereo information and the like of the overall pictures included in the MPD file acquired in step S421. At this time, the MPD file processing unit 213 can grasp that, for the third pattern of sub-pictures (for example, sub-picture 452 and sub-picture 453 in FIG. 50), upscaling by a factor of two in the horizontal direction is required during rendering, by referring to the stereo information of the overall pictures included in the MPD file acquired in step S421. When the processing of step S426 ends, the processing proceeds to step S427.

[0387] In step S427, the MPD file processing unit 213 selects, from among the candidates set in step S425 or step S426, an Adaptation Set that references a sub-picture corresponding to the field of view of the user of the client device 200.

[0388] In step S428, the MPD file processing unit 213 selects a representation according to the user's viewpoint position, line-of-sight direction, the transmission bandwidth of the network between the client and the server, etc. from among the Adaptation Sets selected in step S424.

[0389] In step S429, the segment file acquisition unit 214 acquires a segment file corresponding to the representation selected in step S428.

[0390] In step S430, the segment file processing unit 221 extracts encoded data from the segment file acquired in step S429.

[0391] In step S431, the decoding unit 222 decodes the encoded data of the stream extracted in step S430.

[0392] In step S432, the display information generation unit 223 plays back the stream (content) obtained by decoding in step S431. More specifically, the display information generation unit 223 generates data of a display image from the stream, supplies it to the display unit 217, and causes it to be displayed.

[0393] When the processing of step S432 ends, the content playback processing ends.

[0394] By performing the content playback processing as described above, the client device 200 can more easily select a stream using the information regarding the display area of the sub-picture included in the MPD file. For example, the client device 200 can more easily select and play back an appropriate stream according to its own capabilities etc. based on that information.

[0395] <Signal Details of Stereo Information in the MPD File> For example, the 2D coverage information descriptor may be extended to signal the stereo_presentation_suitable field and view information in the same manner as described in the third embodiment.

[0396] That is, the MPD file may further include view information indicating the view type of the subpicture.

[0397] Also, the view information may be information for each region included in the subpicture.

[0398] Furthermore, the MPD file may further include information indicating whether the view information exists for each region.

[0399] The attribute value 551 in FIG. 69 and the attribute value 552 in FIG. 70 show examples of the attribute values of the extended 2D coverage information descriptor. As shown in the attribute value 551 and the attribute value 552, twoDCoverage is a container element whose data type is omaf:twoDCoverageType. twoDCoverage@stereo_presentation_suitable has a data type of omaf:StereoPresentationType and indicates whether the Adaptation Set is capable of stereo presentation. For example, when this attribute value is 0, it indicates that the picture referred to by the Adaptation Set is monochrome or not capable of stereo presentation. Also, when this attribute value is 1, it indicates that the picture referred to by the Adaptation Set is capable of stereo presentation in some regions. Also, when this attribute value is 2, it indicates that the picture referred to by the Adaptation Set is capable of stereo presentation in all regions.

[0400] The twoDCoverage@view_idc_presence_flag has a data type of xs:boolean and indicates whether there is an individual view_idc for each region. For example, if the attribute value is 0, it indicates that there is no individual view_idc for each region. Also, if the attribute value is 1, it indicates that there is an individual view_idc for each region. The twoDCoverage@default_view_idc has a data type of omaf:ViewType and indicates the view common to all regions. For example, if the attribute value is 0, it indicates a mono view; if the attribute value is 1, it indicates a left view; if the attribute value is 2, it indicates a right view; if the attribute value is 3, it indicates a stereo view. Note that when twoDCoverage@view_idc_presence_flag = 0, this attribute must exist. Also, when twoDCoverage@view_idc_presence_flag = 1, this attribute must not exist.

[0401] The twoDCoverage@proj_picture_width has a data type of xs:unsignedInt and indicates the width of the projected picture. The twoDCoverage@proj_picture_height has a data type of xs:unsignedInt and indicates the height of the projected picture. The twoDCoverage.twoDCoverageInfo has a data type of omaf:twoDCoverageInfoType and is an element indicating the region information on the projected picture. This element can be signaled multiple times.

[0402] twoDCoverage.twoDCoverageInfo@view_idc has a data type of omaf:ViewType and indicates the view for each region. For example, when this attribute value is 0, it indicates a mono view; when this attribute value is 1, it indicates a left view; when this attribute value is 2, it indicates a right view; when this attribute value is 3, it indicates a stereo view. Note that when twoDCoverage@view_idc_presence_flag = 0, this attribute must not exist. Also, when twoDCoverage@view_idc_presence_flag = 1, this attribute must exist.

[0403] twoDCoverage.twoDCoverageInfo@proj_reg_width has a data type of xs:unsignedInt and indicates the width of the region on the projected picture corresponding to the picture referenced by the Adaptation Set. twoDCoverage.twoDCcoverageInfo@proj_reg_height has a data type of xs:unsignedInt and indicates the height of the region on the projected picture corresponding to the picture referenced by the Adaptation Set.

[0404] twoDCoverage.twoDCoverageInfo@proj_reg_top has a data type of xs:unsignedInt and indicates the vertical coordinate of the region on the projected picture corresponding to the picture referenced by the Adaptation Set. twoDCoverage.twoDCoverageInfo@proj_reg_left has a data type of xs:unsignedInt and indicates the horizontal coordinate of the region on the projected picture corresponding to the picture referenced by the Adaptation Set.

[0405] The data type 553 in FIG. 71 shows an example of the definition of the data type in this case.

[0406] In addition, the extended Region-wise packing descriptor and Content coverage descriptor described in the second embodiment may be further extended to signal stereo_presentation_suitable and view information. Alternatively, these information may be signaled using other descriptors.

[0407] <6. Supplementary Note> <Computer> The above-described series of processes can be executed by hardware or by software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer incorporated in dedicated hardware, and a general-purpose personal computer or the like that can execute various functions by installing various programs.

[0408] FIG. 72 is a block diagram showing a configuration example of the hardware of a computer that executes the above-described series of processes by a program.

[0409] In the computer 900 shown in FIG. 72, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0410] An input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

[0411] The input unit 911 is composed of, for example, a keyboard, a mouse, a microphone, a touch panel, input terminals, etc. The output unit 912 is composed of, for example, a display, a speaker, output terminals, etc. The storage unit 913 is composed of, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 is composed of, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0412] In the computer configured as described above, the CPU 901 loads and executes, for example, a program stored in the storage unit 913 via the input / output interface 910 and the bus 904 into the RAM 903, thereby performing the series of processes described above. The RAM 903 also appropriately stores data and the like necessary for the CPU 901 to execute various processes.

[0413] The program executed by the computer (CPU 901) can be recorded and applied to, for example, a removable medium 921 as a package medium or the like. In that case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable medium 921 on the drive 915.

[0414] Also, this program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0415] In addition, this program can be installed in advance in the ROM 902 or the storage unit 913.

[0416] <Application Target of the Present Technology> In the above, the case of applying the present technology to an ISOBMFF file or an MPD file has been described. However, the present technology is not limited to these examples and can be applied to files of any standard that deliver a stream of projection plane images obtained by mapping a three-dimensional structure image onto a single plane. That is, as long as it does not conflict with the present technology described above, the specifications of various processes such as distribution control, file format, encoding / decoding method, etc. are arbitrary. Also, as long as it does not conflict with the present technology, some of the processes and specifications described above may be omitted.

[0417] Also, in the above, the file generation device 100 and the client device 200 have been described as application examples of the present technology. However, the present technology can be applied to any configuration.

[0418] For example, the present technology can be applied to various electronic devices such as transmitters and receivers (e.g., television receivers and mobile phones) in satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to terminals via cellular communication, or devices that record images on media such as optical disks, magnetic disks, and flash memories, or reproduce images from these storage media (e.g., hard disk recorders and cameras).

[0419] Also, for example, the present technology can be implemented as a part of the configuration of a device such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using a plurality of processors, etc., a unit (e.g., a video unit) using a plurality of modules, etc., or a set (e.g., a video set) obtained by adding other functions to the unit.

[0420] In addition, for example, the present technology can also be applied to a network system composed of a plurality of devices. For example, the present technology may be implemented as cloud computing in which a plurality of devices share and jointly process via a network. For example, the present technology may be implemented in a cloud service that provides services related to images (moving images) to any terminal such as a computer, an AV (Audio Visual) device, a portable information processing terminal, an IoT (Internet of Things) device, etc.

[0421] In this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules are housed in one housing are both systems.

[0422] <Fields and Applications Applicable to the Present Technology> The system, device, processing unit, etc. to which the present technology is applied can be used in any field such as, for example, transportation, medical care, crime prevention, agriculture, livestock industry, mining, beauty, factories, home appliances, meteorology, natural monitoring, etc. Also, its application is arbitrary.

[0423] For example, the present technology can be applied to a system or device used for providing ornamental content or the like. Also, for example, the present technology can also be applied to a system or device used for transportation such as traffic situation monitoring and automatic driving control. Further, for example, the present technology can also be applied to a system or device used for security. Also, for example, the present technology can be applied to a system or device used for automatic control of machines or the like. Further, for example, the present technology can also be applied to a system or device used for agriculture and livestock industry. Also, the present technology can be applied to a system or device for monitoring the state of nature such as volcanoes, forests, oceans, etc. and wild animals. Further, for example, the present technology can also be applied to a system or device used for sports.

[0424] <Others> In addition, in this specification, a "flag" is information for identifying a plurality of states, and includes not only information used for identifying two states of true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values that this "flag" can take may be, for example, two values of 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary and may be 1 bit or a plurality of bits. Further, identification information (including flags) is assumed to be included in the bit stream not only in the form of including the identification information itself, but also in the form of including the difference information of the identification information with respect to a certain reference information. Therefore, in this specification, "flags" and "identification information" include not only the information itself, but also the difference information with respect to the reference information.

[0425] Also, various information (such as metadata) regarding the encoded data (bit stream) may be transmitted or recorded in any form as long as it is associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) the other data when processing one data. That is, the data associated with each other may be grouped as one data or may be individual data. For example, the information associated with the encoded data (image) may be transmitted on a transmission path different from that of the encoded data (image). Also, for example, the information associated with the encoded data (image) may be recorded on a recording medium different from that of the encoded data (image) (or another recording area of the same recording medium). Note that this "association" may be a part of the data instead of the entire data. For example, an image and the information corresponding to the image may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part within a frame.

[0426] In addition, in this specification, terms such as "synthesize", "multiplex", "add", "integrate", "include", "store", "insert", "plug in", "insert" mean, for example, combining multiple things into one, such as combining encoded data and metadata into one piece of data, and mean one way of the above-mentioned "associate".

[0427] Moreover, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0428] For example, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, the configurations described as a plurality of devices (or processing units) above may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, if the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or another processing unit).

[0429] Also, for example, the above-described program may be executed on any device. In that case, it suffices if the device has necessary functions (such as function blocks) and can obtain necessary information.

[0430] Also, for example, each step of one flowchart may be executed by one device, or may be executed by a plurality of devices in cooperation. Furthermore, when a plurality of processes are included in one step, the plurality of processes may be executed by one device, or may be executed by a plurality of devices in cooperation. In other words, the plurality of processes included in one step can also be executed as processes of a plurality of steps. Conversely, the processes described as a plurality of steps can also be executed as one step.

[0431] In addition, for example, the program executed by a computer may be such that the processing of the steps for describing the program is executed in time series in accordance with the order described in this specification, or may be executed in parallel or individually at a necessary timing such as when a call is made. That is, as long as there is no contradiction, the processing of each step may be executed in an order different from the order described above. Further, the processing of the steps for describing this program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.

[0432] In addition, for example, a plurality of techniques related to the present technology can be implemented independently as a single unit as long as there is no contradiction. Of course, any plurality of the present technologies can also be implemented in combination. For example, a part or all of the present technology described in any one of the embodiments can be implemented in combination with a part or all of the present technology described in other embodiments. Also, a part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0433] Note that the present technology can also adopt the following configuration. (1) An information processing apparatus including a file generation unit that generates a file including information regarding a region in the entire picture corresponding to a stored sub-picture as information different from the arrangement information for each picture region, and further including the image encoding data obtained by encoding the sub-picture. An information processing apparatus comprising the same. (2) The picture is an omnidirectional image. The information processing apparatus according to (1). (3) The information regarding the region is included in the file as information for each sub-picture. The information processing apparatus according to (1) or (2). (4) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file. The arrangement information for each picture area is information signaled to the Region Wise Packing Box, The information regarding the region is stored in a Scheme Information Box of the ISOBMFF file, which is different from the Region Wise Packing Box, or in a box at a lower level of the Scheme Information Box. The information processing apparatus according to (3). (5) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information regarding the region is stored in a Coverage Information Box indicating a display area on the spherical surface of the track. The information processing apparatus according to (3). (6) The information regarding the region changes dynamically in the stream The information processing apparatus according to any one of (1) to (5). (7) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information regarding the region is stored in a Supplemental Enhancement information message. The information processing apparatus according to (6). (8) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information regarding the region is stored in timed metadata. The information processing apparatus according to (6). (9) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information about the region is stored in the Sample Group Entry The information processing apparatus according to (6). (10) Include information about the region in the entire picture corresponding to the stored sub-picture as information different from the arrangement information for each picture region, and further generate a file including the image coding data in which the sub-picture is coded Information processing method.

[0434] (11) Include information about the region in the entire picture corresponding to the stored sub-picture as information different from the arrangement information for each picture region, and further, a file acquisition unit that acquires a file including the image coding data in which the sub-picture is coded, An image processing unit that selects a stream of the image coding data based on the information about the region included in the file acquired by the file acquisition unit An information processing apparatus comprising the same. (12) The picture is an omnidirectional video The information processing apparatus according to (11). (13) The information about the region is included in the file as information for each sub-picture The information processing apparatus according to (11) or (12). (14) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information about the region is stored in the Scheme Information Box of the ISOBMFF file or a box in the lower layer of the Scheme Information Box The information processing apparatus according to (13). (15) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information regarding the region is stored in a Coverage Information Box indicating a display region on the spherical surface of the track The information processing apparatus according to (13). (16) The information regarding the region changes dynamically in the stream The information processing apparatus according to any one of (11) to (15). (17) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information regarding the region is stored in a Supplemental Enhancement information message The information processing apparatus according to (16). (18) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information regarding the region is stored in timed metadata The information processing apparatus according to (16). (19) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The information regarding the region is stored in a Sample Group Entry The information processing apparatus according to (16). (20) Include information regarding the region in the entire picture corresponding to the stored sub-picture as information different from the arrangement information for each picture region, and further obtain a file including the image coding data in which the sub-picture is coded, Select a stream of the image encoding data based on information regarding the region included in the obtained file. An information processing method.

[0435] (21) A file generation unit that stores, in a track, image data for each sub-picture obtained by dividing an entire picture into a plurality of sub-pictures and encoding them, and generates a file including stereo information, which is information regarding stereo display of the entire picture. An information processing apparatus including the same. (22) The picture is an omnidirectional video. The information processing apparatus according to (21). (23) The stereo information is included in the file as information for each sub-picture. The information processing apparatus according to (21) or (22). (24) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, and the stereo information is stored in a Scheme Information Box of the ISOBMFF file or a box at a lower layer of the Scheme Information Box. The information processing apparatus according to (23). (25) The file further includes information regarding a display size of the sub-picture. The information processing apparatus according to any one of (21) to (24). (26) The file further includes sub-stereo information, which is information regarding stereo display for each sub-picture. The information processing apparatus according to any one of (21) to (25). (27) The file further includes view information indicating a view type of the sub-picture. The information processing apparatus according to any one of (21) to (26). (28) The view information is information for each region included in the sub-picture. The information processing apparatus described in (27). (29) The file further includes information indicating whether the view information exists for each of the regions. (28) The information processing apparatus described in (28). (30) Store the image data for each sub-picture obtained by dividing the entire picture into a plurality of sub-pictures and encoding them in tracks with each other, and generate a file including stereo information which is information regarding the stereo display of the entire picture. Information processing method.

[0436] (31) A file acquisition unit that stores the image data for each sub-picture obtained by dividing the entire picture into a plurality of sub-pictures and encoding them in tracks with each other, and acquires a file including stereo information which is information regarding the stereo display of the entire picture; An image processing unit that selects a stream of the image encoded data based on the stereo information included in the file acquired by the file acquisition unit An information processing apparatus comprising: (32) The picture is an omnidirectional video. (31) The information processing apparatus described in (31). (33) The stereo information is included in the file as information for each sub-picture. (31) or (32) The information processing apparatus described in (31) or (32). (34) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The stereo information is stored in a Scheme Information Box of the ISOBMFF file or a box in a lower layer of the Scheme Information Box. (33) The information processing apparatus described in (33). (35) The file further includes information regarding the display size of the sub-picture. The information processing apparatus according to any one of (31) to (34). (36) The file further includes sub-stereo information which is information regarding stereo display for each of the sub-pictures. The information processing apparatus according to any one of (31) to (35). (37) The file further includes view information indicating a view type of the sub-picture. The information processing apparatus according to any one of (31) to (36). (38) The view information is information for each area included in the sub-picture. The information processing apparatus according to (37). (39) The file further includes information indicating whether the view information exists for each of the areas. The information processing apparatus according to (38). (40) Image data for each of the sub-pictures obtained by dividing and encoding an entire picture into a plurality of sub-pictures are stored in tracks with each other, and a file including stereo information which is information regarding stereo display of the entire picture is acquired. Based on the stereo information included in the acquired file, selection of a stream of the image encoded data is performed. An information processing method.

[0437] (41) A file generation unit that manages image encoded data for each of the sub-pictures obtained by dividing and encoding an entire picture into a plurality of sub-pictures, and generates a control file used for distribution control of the image encoded data, the control file including information regarding an area in the entire picture corresponding to the sub-picture as information different from arrangement information for each picture area. An information processing apparatus including the same. (42) The picture is an omnidirectional video. The information processing apparatus according to (41). (43) The information regarding the area is included in the control file as information for each of the sub-pictures. The information processing apparatus according to (41) or (42). (44) The control file is an MPD (Media Presentation Description) file. The image encoding data for each sub-picture is managed for each adaptation set, The arrangement information for each picture area is stored in a Region-wise packing descripitor, The information regarding the area is defined in the Supplemental Property or Essential Property of the MPD file The information processing apparatus according to (43). (45) The control file is an MPD (Media Presentation Description) file, The image encoding data for each sub-picture is managed for each adaptation set, The arrangement information for each picture area is stored in a Region-wise packing descripitor, The information regarding the area is defined in the Content coverage descriptor of the MPD file The information processing apparatus according to (43). (46) Manage the image encoding data for each sub-picture obtained by dividing and encoding the entire picture into a plurality of sub-pictures, and include information regarding the area in the entire picture corresponding to the sub-picture as information different from the arrangement information for each picture area, and generate a control file used for distribution control of the image encoding data Information processing method.

[0438] (51) Manage the image encoding data for each sub-picture obtained by dividing and encoding the entire picture into a plurality of sub-pictures, and include information regarding the area in the entire picture corresponding to the sub-picture as information different from the arrangement information for each picture area, and a file acquisition unit that acquires a control file used for distribution control of the image encoding data An image processing unit that selects a stream of the image encoding data based on the information regarding the area included in the control file acquired by the file acquisition unit An information processing apparatus comprising the same. (52) The picture is an all-sky image The information processing apparatus according to (51). (53) Information about the region is included in the control file as information for each sub-picture. The information processing apparatus according to (51) or (52). (54) The control file is an MPD (Media Presentation Description) file, The image encoding data for each sub-picture is managed for each adaptation set, The arrangement information for each picture region is stored in a Region-wise packing descripitor, The information about the region is defined in the Supplemental Property or Essential Property of the MPD file. The information processing apparatus according to (53). (55) The control file is an MPD (Media Presentation Description) file, The image encoding data for each sub-picture is managed for each adaptation set, The arrangement information for each picture region is stored in a Region-wise packing descripitor, The information about the region is defined in the Content coverage descriptor of the MPD file. The information processing apparatus according to (53). (56) Manage the image encoding data for each sub-picture obtained by dividing and encoding the entire picture into a plurality of sub-pictures, and include information about the region in the entire picture corresponding to the sub-picture as information different from the arrangement information for each picture region, and obtain a control file used for distribution control of the image encoding data, Select a stream of the image encoding data based on the information about the region included in the obtained control file. Information processing method.

[0439] (61) A file generation unit that manages, for each adaptation set, the image encoding data for each sub-picture obtained by dividing the entire picture into a plurality of sub-pictures and encoded, and generates a control file used for distribution control of the image encoding data, the control file including stereo information which is information related to stereo display of the adaptation set. An information processing apparatus comprising the same. (62) The picture is an omnidirectional video. The information processing apparatus according to (61). (63) The control file further includes view information indicating a view type of the sub-picture. The information processing apparatus according to (61) or (62). (64) The view information is information for each area included in the sub-picture. The information processing apparatus according to (63). (65) The control file further includes information indicating whether the view information exists for each area. The information processing apparatus according to (63) or (64). (66) The control file further includes information indicating whether an adaptation set is stereo-displayable. The information processing apparatus according to any one of (63) to (65). (67) Manage the image encoding data for each sub-picture obtained by dividing the entire picture into a plurality of sub-pictures and encoded, for each adaptation set, and generate a control file used for distribution control of the image encoding data, the control file including stereo information which is information related to stereo display of the adaptation set. An information processing method.

[0440] (71) A file acquisition unit that manages, for each adaptation set, the image encoding data for each sub-picture obtained by dividing the entire picture into a plurality of sub-pictures and encoded, and acquires a control file used for distribution control of the image encoding data, the control file including stereo information which is information related to stereo display of the adaptation set. An image processing unit that selects a stream of the image encoding data based on the stereo information included in the control file acquired by the file acquisition unit An information processing apparatus comprising the same. (72) The picture is an omnidirectional image The information processing apparatus according to (71). (73) The control file further includes view information indicating a view type of the sub-picture The information processing apparatus according to (71) or (72). (74) The view information is information for each region included in the sub-picture The information processing apparatus according to (73). (75) The control file further includes information indicating whether the view information exists for each region The information processing apparatus according to (73) or (74). (76) The control file further includes information indicating whether an adaptation set is stereo-displayable The information processing apparatus according to any one of (73) to (75). (77) Manage the image encoding data for each sub-picture obtained by dividing and encoding the entire picture into a plurality of sub-pictures for each adaptation set, and acquire a control file used for distribution control of the image encoding data, the control file including stereo information which is information regarding stereo display of the adaptation set Select a stream of the image encoding data based on the stereo information included in the acquired control file An information processing method.

Explanation of Signs

[0441] 100 File generation device, 101 Control unit, 102 Memory, 103 File generation unit, 111 Data input unit, 112 Data encoding / generation unit, 113 MPD file generation unit, 114 Recording unit, 115 Upload unit, 121 Preprocessing unit, 122 Encoding unit, 123 Segment file generation unit, 200 Client device, 201 Control unit, 202 Memory, 203 Reproduction processing unit, 211 Measurement unit, 212 MPD file acquisition unit, 213 MPD file processing unit, 214 Segment file acquisition unit, 215 Display control unit, 216 Data analysis / decoding unit, 217 Display unit, 221 Segment file processing unit, 222 Decoding unit, 223 Display information generation unit, 900 Computer

Claims

1. A file generation unit that generates a content file including image encoding data in which an entire picture is encoded and a predetermined metadata box, The metadata box indicates whether a projection picture, which is a projection plane image obtained by projecting an omnidirectional video onto a three-dimensional structure and mapping it onto a plane, is the same as the entire picture composed of sub-pictures, and indicates whether region-wise packing processing has been performed on the entire picture An information processing apparatus.

2. The metadata box includes a numerical value indicating whether the region-wise packing processing has been performed on the entire picture The information processing apparatus according to claim 1.

3. The metadata box includes a flag field indicating whether the region-wise packing processing has been performed on the entire picture The information processing apparatus according to claim 1.

4. A value of 1 in the flag field indicates that the region-wise packing processing has not been performed on the entire picture The information processing apparatus according to claim 3.

5. The metadata box includes a flag field indicating whether the entire picture is the same as the projection picture The information processing apparatus according to claim 1.

6. A value of 1 in the flag field indicates that the entire picture is the same as the projection picture The information processing apparatus according to claim 5.

7. Generate a content file including image encoding data in which an entire picture is encoded and a predetermined metadata box, The metadata box indicates whether a projection picture, which is a projection plane image obtained by projecting an omnidirectional video onto a three-dimensional structure and mapping it onto a plane, is the same as the entire picture composed of sub-pictures, and indicates whether region-wise packing processing has been performed on the entire picture An information processing method.

8. The metadata box includes a numerical value indicating whether the region-wise packing processing has been performed on the entire picture The information processing method according to claim 7.

9. The metadata box includes a flag field indicating whether the region-wise packing processing has been performed on the entire picture The information processing method according to claim 7.

10. That the value of the flag field is 1 indicates that the region-wise packing process has not been performed on the entire picture. The information processing method according to claim 9.

11. The metadata box includes a flag field indicating whether the entire picture is the same as the projection picture. The information processing method according to claim 7.

12. That the value of the flag field is 1 indicates that the entire picture is the same as the projection picture. The information processing method according to claim 11.

13. A file acquisition unit that acquires a content file including encoded image encoding data of the entire picture and a predetermined metadata box, A file processing unit that processes the acquired content file and The metadata box indicates whether the projection picture, which is a projection plane image obtained by projecting the omnidirectional video onto a three-dimensional structure and mapping it onto a plane, is the same as the entire picture composed of sub-pictures, and also indicates whether the region-wise packing process has been performed on the entire picture. An information processing apparatus.

14. The metadata box includes a numerical value indicating whether the region-wise packing process has been performed on the entire picture. The information processing apparatus according to claim 13.

15. The metadata box includes a flag field indicating whether the region-wise packing process has been performed on the entire picture. The information processing apparatus according to claim 13.

16. That the value of the flag field is 1 indicates that the region-wise packing process has not been performed on the entire picture. The information processing apparatus according to claim 15.

17. The metadata box includes a flag field indicating whether the entire picture is the same as the projection picture. The information processing apparatus according to claim 13.

18. That the value of the flag field is 1 indicates that the entire picture is the same as the projection picture. The information processing apparatus according to claim 17.

19. Acquire a content file including encoded image encoding data of the entire picture and a predetermined metadata box, Process the acquired content file, The metadata box indicates whether the projection picture, which is a projection plane image obtained by projecting a full-sphere video onto a three-dimensional structure and mapping it onto a plane, is the same as the entire picture composed of sub-pictures, and also indicates whether region-wise packing processing has been performed on the entire picture. An information processing method.

20. The metadata box includes a numerical value indicating whether region-wise packing processing has been performed on the entire picture. The information processing method according to claim 19.

21. The metadata box includes a flag field indicating whether region-wise packing processing has been performed on the entire picture. The information processing method according to claim 19.

22. A value of 1 in the flag field indicates that region-wise packing processing has not been performed on the entire picture. The information processing method according to claim 21.

23. The metadata box includes a flag field indicating whether the entire picture is the same as the projection picture. The information processing method according to claim 19.

24. A value of 1 in the flag field indicates that the entire picture is the same as the projection picture. The information processing method according to claim 23.

Images