Information processing device and method

Abstract

The present disclosure relates to an information processing device and method with which an increase in playback processing load can be suppressed. Media data relating to haptics is encoded, a media bit stream is generated, a content file for storing content configured from a plurality of samples continuously reproduced in the time direction is generated, the media bit stream is divided into sample units and stored in the content file, information relating to a stream type indicating whether media data is present in the sample is generated, and said information is stored in the content file as management information. The present disclosure can be applied to, e.g., an information processing device or an information processing method.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that are capable of suppressing an increase in the load of playback processing.

[0002] Conventionally, the International Organization for Standardization Base Media File Format (ISOBMFF) has been used as a container format for storing media such as images and audio (see, for example, Non-Patent Document 1). However, in addition to images and audio, media also includes, for example, haptic media that expresses tactile sensations through vibrations, and a method for handling such haptic media in ISOBMFF has been proposed (see, for example, Non-Patent Document 2).

[0003] Compared to audio and video, haptic media is less likely to be continuous data and is often played back intermittently. Therefore, device control that prevents the output device from being driven when not playing back is useful for reducing power consumption. However, with haptic media, there are a variety of output devices, and their output characteristics are also diverse. Therefore, playback must take into account the startup time in advance, depending on the type of haptic and the characteristics of the device.

[0004] In the haptics file format, intermittent haptic signals are stored in ISOMFF by defining them as MIHS samples, regardless of whether or not a haptic signal is present (see, for example, Non-Patent Document 3). The presence or absence of a haptic signal is signaled as media information for the MIHS sample.

[0005] Furthermore, files that store media may be distributed, and methods for controlling this distribution have been devised (see, for example, Non-Patent Document 4).

[0006] "14496-12_Ed8_DIS_potential_improvements", ISO / IEC JTC 1 / SC 29 / WG 03 N22316, WG3_N000792, 2023-02-11"Information technology - Coded representation of immersive media - Part 32: Carriage of haptics data", ISO / IEC 23090-32:2024(E), ISO / IEC JTC 1 / SC 29 / WG 3, 2024-02-DD"Text of ISO / IEC FDIS 23090-31 MPEG Haptics Coding", ISO / IEC JTC 1 / SC 29 / WG 7 N832, 2024-02-27"Information technology . Dynamic adaptive streaming over HTTP (DASH). Part 1: Media presentation description and segment formats", ISO 23009-1:2021(X), ISO / IEC JTC 1 / SC 29 / WG 3, 2021-06-24

[0007] However, with this method, in order to determine the presence or absence of a haptic signal in the time axis direction, it is necessary to acquire and analyze all samples, which may increase the load of the playback process.

[0008] The present disclosure has been made in consideration of such circumstances, and aims to make it possible to suppress an increase in the load of the regeneration process.

[0009] An information processing device according to one aspect of the present technology is an information processing device that includes an encoding unit that encodes media data related to haptics and generates a media bitstream, and a content file generation unit that generates a content file that stores content consisting of a plurality of samples that are played continuously in the time direction, divides the media bitstream into sample units and stores them in the content file, generates information regarding a stream type that indicates whether or not the media data is present in the sample, and stores the information in the content file as management information.

[0010] An information processing method according to one aspect of the present technology includes encoding media data related to haptics to generate a media bitstream, generating a content file that stores content consisting of a plurality of samples that are played continuously in the time direction, dividing the media bitstream into sample units and storing them in the content file, generating information regarding a stream type that indicates whether or not the media data is present in the samples, and storing the information in the content file as management information.

[0011] Another aspect of the information processing device of the present technology is an information processing device that includes: a control unit that controls an output device that outputs the media data based on information regarding a stream type that indicates whether or not media data related to haptics is present in a content file that stores content consisting of a plurality of samples that are played continuously in the time direction, and that is stored as management information in the content file; and a decoding unit that decodes the media bitstream stored in the content file and generates the media data.

[0012] Another aspect of the information processing method of the present technology is an information processing method that includes controlling an output device that outputs the media data based on information regarding a stream type that indicates whether or not media data related to haptics is present in a content file that stores content consisting of a plurality of samples that are played continuously in the time direction, and decoding the media bitstream stored in the content file to generate the media data.

[0013] In an information processing device and method according to one aspect of the present technology, media data relating to haptics is encoded, a media bitstream is generated, a content file is generated that stores content consisting of multiple samples that are played continuously in the time direction, the media bitstream is divided into sample units and stored in the content file, and information regarding the stream type indicating whether or not media data is present in the sample is generated and stored in the content file as management information.

[0014] In another aspect of the information processing device and method of the present technology, an output device that outputs media data is controlled based on information regarding the stream type that indicates the presence or absence of haptic media data in the sample, which information is stored as management information in a content file that stores content consisting of multiple samples that are played continuously in the time direction, and the media bitstream stored in the content file is decoded to generate media data.

[0015] 1 is a diagram illustrating an example of a box structure of an ISOBMFF file. FIG. 1 is a diagram illustrating a fragment movie. FIG. 2 is a diagram illustrating an example of a structure in a sample table box. FIG. 3 is a diagram illustrating a user data box. FIG. 4 is a diagram illustrating an MIHS sample. FIG. 5 is a diagram illustrating an example of a structure of a bit stream of a haptics signal. FIG. 6 is a diagram illustrating an example of a structure of an MIHS unit. FIG. 7 is a diagram illustrating an example of an MIHS unit type. FIG. 8 is a diagram illustrating an example of a structure of a haptics general data model. FIG. 9 is a diagram illustrating haptic experience properties. FIG. 10 is a diagram illustrating haptic perception properties. FIG. 11 is a diagram illustrating haptic device properties. FIG. 12 is a diagram illustrating haptic channel properties. FIG. 13 is a diagram illustrating supplemental properties. FIG. 14 is a diagram illustrating an example of a method for transmitting information about a stream type. FIG. 15 is a diagram illustrating an example of a method for transmitting information about a stream type. FIG. 16 is a diagram illustrating an example of the structure of an ISOBMFF. FIG. 17 is a diagram illustrating an example of a definition of a haptics silent stream sample group. FIG. 18 is a diagram illustrating an example of a definition of a haptics packet stream sample group. FIG. 19 is a diagram illustrating an example of a definition of a haptics stream type sample group. FIG. 19 is a diagram illustrating an example of a definition of a haptics stream type sample group. FIG. 19 is a diagram illustrating an example of a definition of a haptics stream type description sample entry. FIG. 1 is a diagram illustrating an example of the structure of an ISOBMFF. FIG. 2 is a diagram illustrating an example of a definition of a haptics contiguous stream type sample group. FIG. 3 is a diagram illustrating an example of a definition of a haptics contiguous stream type sample group. FIG. 4 is a diagram illustrating an example of an application of a haptics contiguous stream type sample group. FIG. 5 is a diagram illustrating an example of an application of a haptics contiguous stream type sample group.10A and 10B are diagrams illustrating an example of a definition of a haptic contiguous stream type sample group; a diagram illustrating an example of a definition of a haptic contiguous stream type sample group; a diagram illustrating an example of an application of a haptic contiguous stream type sample group; a diagram illustrating an example of a definition of a haptic contiguous stream type box; a diagram illustrating an example of a definition of a haptic contiguous stream type box; a diagram illustrating an example of a definition of a haptic contiguous stream type box; a diagram illustrating an example of a definition of a haptic packet stream sample group; a diagram illustrating an example of a definition of a haptic packet stream sample group; a diagram illustrating an example of a definition of a haptic stream type sample group; a diagram illustrating an example of a definition of a haptic stream type sample group; a diagram illustrating an example of a definition of a haptic contiguous stream type sample group; a diagram illustrating an example of a definition of a haptic contiguous stream type sample group; a diagram illustrating an example of a definition of a haptic contiguous stream type box; a diagram illustrating an example of a definition of a haptic contiguous stream type box. FIG. 1 is a diagram showing an example of a definition of a haptics contiguous stream type box. FIG. 2 is a diagram showing an example of a definition of a haptics contiguous stream type box. FIG. 3 is a diagram showing an example of a description of an MPD. FIG. 4 is a diagram showing an example of the configuration of a Matryoshka media container. FIG. 5 is a block diagram showing an example of the main configuration of a file generation device. FIG. 6 is a flowchart showing an example of the flow of a file generation process. FIG. 7 is a flowchart showing an example of the flow of a file generation process. FIG. 8 is a flowchart showing an example of the flow of a file generation process. FIG. 9 is a flowchart showing an example of the flow of a file generation process.1 is a flowchart showing an example of the flow of a file generation process. 2 is a flowchart showing an example of the flow of a file generation process. 3 is a flowchart showing an example of the flow of a file generation process. 4 is a flowchart showing an example of the flow of a file generation process. 5 is a block diagram showing an example of the main configuration of a playback device. 6 is a flowchart showing an example of the flow of a playback process. 7 is a flowchart showing an example of the flow of a playback process. 8 is a flowchart showing an example of the flow of a device media process. 9 is a flowchart showing an example of the flow of a playback process. 10 is a flowchart showing an example of the flow of a device media process. 11 is a flowchart showing an example of the flow of a playback process. 12 is a block diagram showing an example of the main configuration of a computer.

[0016] Below, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature etc. supporting technical content and technical terminology 2. Playback of media data 3. Transmission of information related to stream type 4. First embodiment (file generation device) 5. Second embodiment (playback device) 6. Supplementary notes

[0017] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0018] Non-patent document 1: (described above) Non-patent document 2: (described above) Non-patent document 3: (described above) Non-patent document 4: (described above)

[0019] In other words, the contents of any of the above non-patent documents, as well as the contents of other documents referenced in the above non-patent documents, are also used as the basis for determining the support requirements. For example, even if a syntax or term described in any of the above non-patent documents is not directly defined in this disclosure, it is considered to be within the scope of this disclosure and meet the support requirements of the claims. Similarly, for technical terms such as parsing, syntax, and semantics, even if they are not directly defined in this disclosure, they are considered to be within the scope of this disclosure and meet the support requirements of the claims.

[0020] <2. Playback of media data> <ISOBMFF>

[0021] Conventionally, as described in Non-Patent Document 1, for example, ISOBMFF (International Organization for Standardization Base Media File Format) has been used as a container format for storing media such as images and audio as content.

[0022] ISOBMFF stores content media data (also referred to as media data) and management information for that media data. The management information may include, for example, time information, access information, and codec information for the media data. ISOBMFF is a general-purpose file format that is independent of the type of media or codec. ISOBMFF is widely used, for example, as a recording format for video data and a content distribution format. In this specification, files that conform to this ISOBMFF are also referred to as ISOBMFF files.

[0023] As shown in Figure 1, four boxes are defined at the file level in ISOBMFF: the file type box (FileTypeBox('ftyp')), the meta box (MetaBox('meta')), the movie box (MovieBox('moov')), and the media data box (MediaDataBox('mdat')).

[0024] The file type box (FileTypeBox('ftyp')) is a box that stores brand information that indicates the type of boxes used in the file. The meta box (MetaBox('meta')) is a box that stores metadata. The meta box is optional. The meta box can also be defined under a movie box or under a track box (TrackBox) under a movie box. The movie box (MoovBox('moov')) is a box that stores management information such as time information, access information, and codec information for media data. Codec information is defined by a different standard for each codec. The media data box (MediaDataBox('mdat')) is a box that stores media data. Access information for media data (e.g., offset, size, etc.) is defined under the movie box.

[0025] In ISOBMFF, content information is divided into samples and managed. For media that are played continuously for a predetermined period of time, such as video or audio, ISOBMFF defines a method called FragmentMovie, in which the samples are divided into fragments of any length (e.g., 10-second intervals) and managed accordingly. That is, in the case of a fragment movie, as shown in FIG. 2 , a Movie Fragment Box (MovieFragmentBox('moof')) and a Media Data Box (MediaDataBox('mdat')) are defined for each fragment. The Media Data Box of each fragment stores the media of the samples belonging to that fragment, and the Movie Fragment Box stores management information for that sample. In this case, codec information is stored as a sample entry in the Sample Description Box of the Movie Box. Sample access information equivalent to the Time to Sample Box, Sample Size Box, Sample to Chunk Box, and Sync Sample Box may be stored on a sample-by-sample basis in the Track Fragment Run Box of the Movie Fragment Box.

[0026] The Sample Table Box defines boxes for managing codec information, access information, and time information of a sample. Typically, one sample is associated with one piece of codec information required to decode that sample. For example, to play back from an intra-coded sample, information identifying the intra-coded sample must be managed. As shown in FIG. 3 , the Sample Table Box may define a Sample Group Description Box (SampleGroupDescriptionBox('sgpd')) and a Sample To Group Box (SampleToGroupBox('sbgp')). The Sample Group Description Box is a general-purpose box that can store sample information based on a grouping type (grouping_type), separate from codec information and intra-coding information. This information can be associated with any sample using the information stored in the Sample To Group Box.

[0027] In other words, in the Sample Table Box, samples can be grouped to define a playback method different from that of the Sample Entry. This group of samples is called a Sample Group. The playback method is defined by the information stored in the Sample Group Description Box. The playback method is linked to the sample by the information stored in the Sample To Group Box. In other words, the information stored in the Sample Group Description Box and the Sample To Group Box allows for playback processing of samples that differs from normal playback. For example, a Temporal Level Entry can select a sample with an appropriate frame rate. Multiple Sample Group Description Boxes can store information for multiple samples.

[0028] By defining a sample group description box and further defining a grouping type parameter (GroupingTypeParameter), it is possible to specify samples according to multiple subparameters of the same grouping type (GroupingType). Multiple sample group description boxes can be used to store information about multiple samples.

[0029] An ISOBMFF file also provides a User Data box, as shown in Figure 4. This User Data box is a container box for storing useful user data. This user data is formatted as a collection of boxes with more specific box types, which declare more precise content. This User Data box contains objects that declare user information about the box and its data (presentation or track). Boxes contained in this User Data box can be predefined, registered, or regular boxes using UUID extension box types.

[0030] Incidentally, in addition to images and sounds, there is also haptic media that expresses tactile sensations through vibrations, and Non-Patent Document 2 proposes a method for handling such haptic media in ISOBMFF.

[0031] Haptic media (vibration and touch) is less likely to be continuous data than other media such as audio and video, and is often data that is played intermittently. In other words, haptic media generally has longer periods of non-playback than images, audio, etc. Furthermore, output devices that output media can generally reduce their power consumption by shutting down or going into hibernation mode, rather than in an active state where media can be output. Therefore, if a haptic media output device is kept in an active state continuously while processing content, the time it remains in the active state even when no output is occurring increases, which could unnecessarily increase power consumption. Therefore, device control that prevents the output device from being driven during periods when media is not being output is useful for reducing power consumption.

[0032] However, such switching of the output device state had to be performed in a way that did not affect the media output. For example, if the switch from a shutdown or hibernation state to an active state was not in time for the output timing, the output device might not be able to output the media correctly (the playback quality (i.e., the quality of the media being played) might be reduced). In the case of haptic media, there are a variety of output devices, and their output characteristics also vary. For example, the startup time for an output device to switch from a shutdown or hibernation state to an active state varied depending on the output device. Therefore, it was necessary to appropriately control the switching of the output device state depending on the media being played and the output device.

[0033] As shown in FIG. 5, Non-Patent Document 2 defines a codec-dependent MIHS sample entry (MIHSSampleEntry) that stores a haptics coding stream in ISOBMFF, and an MIHS sample (MIHSsample), which is an elementary stream. A silent flag (silent_flag) is signaled for an MIHS sample that does not contain a haptics signal. This silent flag (silent_flag) is flag information that indicates whether or not haptics data is included in the data packet. In this case, the data packet payload size (data_packet_payload_size) is zero.

[0034] For example, Non-Patent Document 3 discloses a haptics file format that defines an intermittent haptic signal as an MIHS sample, regardless of whether or not a haptic signal is present, in order to store the signal in ISOMFF. The presence or absence of a haptic signal is signaled as media information for the MIHS sample. As shown in FIG. 6 , a haptic bitstream that encodes a haptic signal is composed of MIHS units, and metadata (e.g., an MIHS unit header) and an elementary stream (e.g., an MIHS packet (MIHS Packet 1 to MIHS Packet n)) are formed for each MIHS unit.

[0035] As shown in Fig. 7, the MIHS unit header stores information such as type, sync, layer, duration, and length. Also, as shown in Fig. 8, types such as unit type initialization (UNITTYPE_INITIALIZATION), unit type temporal (UNITTYPE_TEMPORAL), unit type spatial (UNITTYPE_SPATIAL), and unit type silent (UNITTYPE_SILENT) are available as types. In other words, setting unit type silent (UNITTYPE_SILENT) here (MIHSUnitType = 3) indicates that the MIHS unit does not include a haptic signal.

[0036] By referencing this information (such as the silent flag) during playback, it is possible to determine which samples contain haptic media. This means that the output timing of the haptic media can be accurately determined. Therefore, during playback, it is possible to appropriately control the switching of the output device state so that the output timing is met, thereby suppressing increases in power consumption.

[0037] Non-Patent Document 3 defines a structure as shown in FIG. 9 as a haptics general data model. Specifically, the highest-level concept of haptic content is experience, and its components are defined as follows: For example, an experience has multiple perceptions. Specifically, as shown in FIG. 10, a list of perceptions describing haptic signals is defined in haptic experience properties. Furthermore, one perception includes multiple device lists and multiple channel lists. Specifically, as shown in FIG. 11, an ID, perception modality, reference devices, and channels are defined in haptic perception properties. The ID is identification information for the perception in the experience. The perception modality indicates the type of perception modality of the haptic signal. The reference device indicates a list of haptic reference devices or actuators to be used for this haptic perception. The channel indicates a list of haptic channels that make up this perception. Furthermore, one channel references a device. That is, as shown in FIG. 12, an ID is defined in the haptic device properties. This ID indicates the ID of the device in this experience. This value is used to reference the device within the channel. Furthermore, as shown in FIG. 13, an ID and a reference device ID are defined in the haptic channel properties. The ID is unique identification information for this channel. This identifier is unique among all channels in the perception that includes this channel.The reference device ID is the identification information of the reference device that is the target of reference from the list defined in the perception.

[0038] As described in Non-Patent Document 4, in recent years, there has been an expectation that a method for controlling the distribution of ISOBMFF files will be established using MPEG-DASH (Moving Picture Experts Group Dynamic Adaptive Streaming over HTTP (Hypertext Transfer Protocol)). In MPEG-DASH, the distribution of ISOBMFF files is controlled using a distribution control file called MPD (Media Presentation Description), which stores distribution control information for controlling the distribution. For example, in this MPD, a supplemental property descriptor is defined as shown in FIG. 14. That is, with respect to the supplemental property element, the creator of the media presentation indicates that the descriptor contains supplemental information that a DASH client can use for optimized processing. Note that if the scheme or value of this descriptor is not recognized, the DASH client is expected to ignore the descriptor. In addition, there may be multiple supplemental property elements.

[0039] However, in the above-mentioned method, whether a sample contains media data or not is indicated within the media bitstream. Therefore, in order to determine whether each sample contains media data, it is necessary to acquire and analyze all samples. This may increase the load of the playback process.

[0040] In this specification, media data such as images, sounds, and vibrations that constitute content are also referred to as media data. Also, encoded data obtained by encoding this media data is also referred to as a media bitstream. Also, a file that stores this media bitstream is also referred to as a content file. A content file may be composed of a single track (single-track configuration) or multiple tracks (multi-track configuration). Also, a content file may have any specifications, but the following description will use an ISOBMFF file as an example of this content file.

[0041] Although the above description uses the case of haptic media as an example, other media, such as images and audio, may also experience periods when no media is output. Therefore, switching the state of the output device to reduce power consumption may occur, just as with haptic media. Regardless of the media type, whether a sample contains media data is indicated within the media bitstream. In other words, regardless of the type of media, determining whether each sample contains media data requires acquiring and analyzing all samples. This can increase the load on the playback process.

[0042] <3. Transmission of Information Related to Stream Type> <Regarding Method 1> Therefore, as shown in the top row of the table in Fig. 15, information related to the stream type of a sample is stored as management information in a content file ("Method 1" in the table). Here, the stream type indicates status information indicating the presence or absence of target media data. The target media data indicates the media data to which this stream type corresponds (i.e., media data indicating the presence or absence of the stream type in the sample). The information related to the stream type may include, for example, information defining the stream type of the sample.

[0043] The media data (media stream) stored in a content file may be any type of media as long as it includes haptic (vibration or tactile) media. For example, it may include images (moving images), audio, 3D data, or other media. This media data (media stream) may be singular or multiple. If there are multiple media data (media streams), the media types may all be the same or different. For example, a media stream of haptic media and a media stream of other media may be stored in a content file. The target media data is haptic media data. The target media data may be singular or multiple. If there are multiple target media data, other media such as images and audio may be included in addition to haptic media. In other words, the target media data may be composed of multiple types of media data. In this specification, an example will be described in which the target media data is only haptic media (i.e., vibration or tactile) data.

[0044] For example, a first information processing device (file generating device) includes an encoding unit that encodes media data related to haptics and generates a media bitstream, and a content file generating unit that generates a content file that stores content consisting of multiple samples that are played continuously in the time direction, divides the media bitstream into sample units and stores them in the content file, generates information regarding the stream type that indicates the presence or absence of media data in the sample, and stores it in the content file as management information.

[0045] In addition, the first information processing method (file generation method) includes encoding media data related to haptics and generating a media bitstream, generating a content file that stores content consisting of multiple samples that are played continuously in the time direction, dividing the media bitstream into sample units and storing them in the content file, generating information regarding the stream type that indicates whether or not media data is present in the sample, and storing it in the content file as management information.

[0046] In addition, the first program is a program for causing a computer to execute processes including encoding media data related to haptics and generating a media bitstream, generating a content file for storing content consisting of multiple samples played continuously in the time direction, dividing the media bitstream into sample units and storing them in the content file, generating information regarding the stream type indicating the presence or absence of media data in the sample, and storing it in the content file as management information.

[0047] In a playback process in which a media bitstream is decoded and media data is played back, the timing for starting up the output device can be determined based on the information about the stream type (information defining the stream type of the sample), and therefore switching of the output device state can be easily and appropriately controlled. In other words, the first information processing device can suppress an increase in power consumption during the playback process while suppressing a decrease in playback quality (the quality of the played media).

[0048] Furthermore, since the information on the stream type includes information defining the stream type of the sample, the stream type of the sample (i.e., whether or not target media data is present) can be easily determined in the playback process based on the information on the stream type (information defining the stream type of the sample) without analyzing the media bitstream. Therefore, the first information processing device can suppress an increase in power consumption due to analysis of the media bitstream. Furthermore, the first information processing device can suppress an increase in processing time due to analysis of the media bitstream. In other words, the first information processing device can suppress an increase in the load of the playback process and an increase in the cost of the playback process.

[0049] For example, the second information processing device (playback device) includes a control unit that controls an output device to output media data based on information regarding the stream type indicating whether or not there is haptic media data in the sample, which information is stored as management information in a content file that stores content consisting of multiple samples that are played continuously in the time direction, and a decoding unit that decodes the media bitstream stored in the content file and generates media data.

[0050] In addition, the second information processing method (playback method) includes controlling an output device that outputs the media data based on information regarding the stream type indicating the presence or absence of media data related to haptics in the sample, which information is stored as management information in a content file that stores content consisting of a plurality of samples that are played continuously in the time direction, and decoding the media bitstream stored in the content file to generate the media data.

[0051] The second program is also a program for causing a computer to execute processing including controlling an output device that outputs media data based on information regarding a stream type that indicates whether or not media data related to haptics is present in a sample, the information being stored as management information in a content file that stores content consisting of multiple samples that are played continuously in the time direction, and decoding the media bitstream stored in the content file to generate media data.

[0052] The second information processing device that decodes the media bitstream and plays back the media data can determine the timing to start up the output device based on the information about the stream type (information defining the stream type of the sample), and can easily and appropriately control switching of the state of the output device. In other words, the second information processing device can suppress an increase in power consumption during playback processing while suppressing a decrease in playback quality (the quality of the played media).

[0053] Furthermore, since the information on the stream type includes information defining the stream type of the sample, the second information processing device can easily determine the stream type of the sample (i.e., whether or not target media data is present) based on the information on the stream type (information defining the stream type of the sample) without analyzing the media bitstream. Therefore, the second information processing device can suppress an increase in power consumption due to analysis of the media bitstream. Furthermore, the second information processing device can suppress an increase in processing time due to analysis of the media bitstream. In other words, the second information processing device can suppress an increase in the load of the playback process and an increase in the cost of the playback process.

[0054] Furthermore, the information about the stream type may further include information linking the stream type to a sample. For example, if the content file is an ISOBMFF file, a sample group may be defined as information defining the stream type, and a sample to group may be defined as information linking the stream type to a sample.

[0055] For example, in the case of an ISOBMFF file, as shown in Fig. 17, a Sample Group Description Box ('sgpd') is provided in a Sample Table Box ('stbl') belonging to a Track Box ('TrackBox('trak')) in a Movie Box ('Moov') and a Sample Group Description Box ('sgpd') is provided in which the sample groups are defined. Furthermore, a Sample To Group Description Box ('sbgp') is provided in the Sample Table Box ('stbl') and a Sample To Group Description Box ('sbgp') is provided in which the sample to groups are defined.

[0056] When a content file has such a structure, the sample group description box may define a sample group that defines a stream type, and the sample to group description box may define a sample to group that links the stream type to a sample.

[0057] <Regarding Method 1-1> For example, when "Method 1" in the table of FIG. 15 is applied, a sample group (mhss) with a stream type of silent may be defined as shown in the second row from the top of the table ("Method 1-1" in the table). The stream type "silent" indicates that no haptic data is included in the samples. In other words, a sample group with a stream type of silent is a sample group composed of samples that do not include haptic data.

[0058] For example, a haptics silent stream sample group may be defined as shown in A of FIG. 18. This haptics silent stream sample group is a sample group whose stream type is silent, and whose group type is set to 'mhss'. This haptics silent stream sample group is also stored in a sample group description box. This haptics silent stream sample group is not a required sample group, and may be absent or multiple groups may be present.

[0059] As shown in B of Figure 18, this haptics silent stream sample group is a sample group in an MIHS track that is composed of samples that do not contain haptics data. In other words, the value of the silent flag of the MIHS samples that belong to this sample group is true (silent_flag = 1). The silent flag is flag information that indicates whether haptics data is present in a sample. If the value of this flag information is true, it indicates that haptics data is not contained in the sample.

[0060] Note that even if this haptics silent stream sample group does not exist, samples that do not contain haptics data may exist in the MIHS track. Similarly, samples that contain haptics data may also exist in the MIHS track. Note that the grouping type parameter (grouping_type_parameter) is not defined for the SampleToGroupBox (SampleToGroupBox) with the grouping type "mhss."

[0061] An example of the syntax of this haptics silent stream sample group is shown in C of Fig. 18. "HapticsSilentStreamGroupEntry extends HapticSampleGroupEntry('mhss')" may be defined.

[0062] For example, in the first information processing device, a content file generation unit may generate, as information on the stream type, information defining a group of samples for which no media data exists, and in the second information processing device, a control unit may control an output device based on the information on the stream type that defines a group of samples for which no media data exists.

[0063] In the playback process, by referencing the information defining the sample group with the silent stream type (e.g., the haptic silent stream sample group (mhss)), samples that do not contain haptic data (i.e., target media data) can be easily identified. In other words, it is possible to easily identify periods during which haptic data is not output and to appropriately switch the state of the output device depending on the periods. In other words, it is possible to suppress an increase in power consumption during the playback process while suppressing a decrease in playback quality. Furthermore, in the playback process, the information defining the sample group with the silent stream type can be easily referenced without analyzing the media bitstream. Therefore, it is possible to suppress an increase in power consumption due to analysis of the media bitstream. Furthermore, it is possible to suppress an increase in processing time due to analysis of the media bitstream. In other words, it is possible to suppress an increase in the load of the playback process and an increase in the cost of the playback process.

[0064] <Regarding Method 1-2> Furthermore, when "Method 1" in the table of FIG. 15 is applied, a sample group (mhps) with an active stream type may be defined as shown in the third row from the top of the table ("Method 1-2" in the table). The "active" stream type indicates that haptic data is included in the sample. In other words, a sample group with an active stream type is a sample group composed of samples that include haptic data.

[0065] For example, a haptics packet stream sample group may be defined as shown in A of FIG. 19. This haptics packet stream sample group is a sample group with an active stream type, and the group type is set to 'mhps'. This haptics packet stream sample group is also stored in a sample group description box. This haptics silent stream sample group is not a required sample group, and may be absent or multiple samples may be present.

[0066] 19B, this haptics packet stream sample group is a sample group consisting of samples in an MIHS track that contain haptics data. In other words, the silent flag value of the MIHS samples that belong to this sample group is false (silent_flag = 0). A false value for this silent flag indicates that the sample contains haptics data.

[0067] Note that even if this haptics packet stream sample group does not exist, samples that do not contain haptic data may exist in the MIHS track. Similarly, samples that contain haptic data may also exist in the MIHS track. Note that the grouping type parameter (grouping_type_parameter) is not defined for the SampleToGroupBox (SampleToGroupBox) with the grouping type "mhps."

[0068] An example of the syntax of this haptics packet stream sample group is shown in C of Fig. 19. "HapticsPacketStreamSampleGroupEntry extends HapticSampleGroupEntry('mhps')" may be defined.

[0069] For example, in the first information processing device, a content file generation unit may generate, as information on the stream type, information defining a group of samples in which media data exists, and in the second information processing device, a control unit may control an output device based on the information defining a group of samples in which media data exists, which information is included in the information on the stream type.

[0070] In the playback process, by referencing information defining a sample group with an active stream type (e.g., a haptic packet stream sample group (mhps)), samples containing haptic data (i.e., target media data) can be easily identified. In other words, the period during which haptic data is output can be easily identified and the state of the output device can be appropriately switched depending on that period. In other words, an increase in power consumption during the playback process can be suppressed while suppressing a decrease in playback quality. Furthermore, in the playback process, the information defining the sample group with an active stream type can be easily referenced without analyzing the media bitstream. Therefore, an increase in power consumption due to analysis of the media bitstream can be suppressed. Furthermore, an increase in processing time due to analysis of the media bitstream can be suppressed. In other words, an increase in the load of the playback process can be suppressed, and an increase in the cost of the playback process can be suppressed.

[0071] <Regarding Methods 1-3> Furthermore, when "Method 1" in the table of FIG. 15 is applied, a sample group (mhst) that specifies the stream type may be defined, as shown in the fourth row from the top of the table ("Method 1-3" in the table). In other words, this sample group can be composed of only samples that include haptic data, or only samples that do not include haptic data. Which samples the sample group is composed of is indicated by specifying whether the stream type is silent or active. For example, if the stream type is specified as silent, none of the samples belonging to this sample group include haptic data. Also, if the stream type is specified as active, all of the samples belonging to this sample group include haptic data.

[0072] For example, in the first information processing device, the content file generation unit may generate, as the information on the stream type, information defining a group of samples that indicates whether or not media data is present in the samples, and in the second information processing device, the control unit may control the output device based on the information on the stream type that defines a group of samples that indicates whether or not media data is present in the samples.

[0073] In the playback process, by referencing the information defining the sample group specifying the stream type as described above, it is possible to easily identify samples that include haptic data (i.e., target media data) or samples that do not include haptic data (i.e., target media data). In other words, it is possible to easily identify periods during which haptic data is not output or periods during which haptic data is output, and to appropriately switch the state of the output device depending on the periods. In other words, it is possible to suppress an increase in power consumption during the playback process while suppressing a decrease in playback quality. Furthermore, in the playback process, the information defining the sample group specifying the stream type can be easily referenced without analyzing the media bitstream. Therefore, it is possible to suppress an increase in power consumption due to analysis of the media bitstream. Furthermore, it is possible to suppress an increase in processing time due to analysis of the media bitstream. In other words, it is possible to suppress an increase in the load of the playback process and an increase in the cost of the playback process.

[0074] <Regarding Method 1-3-1> The stream type may be specified in any manner. For example, the stream type may be specified using flag information. That is, when "Method 1-3" in the table of FIG. 15 is applied, the stream type may be specified using flag information, as shown in the fifth row from the top of the table ("Method 1-3-1" in the table).

[0075] For example, a haptics stream type sample group may be defined as shown in A of FIG. 20. This haptics stream type sample group is a sample group that specifies the stream type, and the group type is set to 'mhst'. This haptics stream type sample group is also stored in a sample group description box. This haptics stream type sample group is not a required sample group, and may be absent or multiple groups may be present.

[0076] As shown in B of FIG. 20 , this haptics stream type sample group is a sample group in an MIHS track that does not contain haptic data or is composed of samples that do not contain haptic data. Whether or not the samples that make up this haptics stream type sample group contain haptic data is identified by the silent flag (silent_flag). For example, if the value of this silent flag is true (silent_flag = 1), this indicates that the MIHS samples belonging to this sample group do not contain haptic data. Conversely, if the value of this silent flag is false (silent_flag = 0), this indicates that the MIHS samples belonging to this sample group contain haptic data. Even if this haptics stream type sample group does not exist, samples that do not contain haptic data may exist in an MIHS track. Similarly, samples that contain haptic data may also exist in an MIHS track. Note that the grouping type parameter (grouping_type_parameter) is not defined for the sample to group box (SampleToGroupBox) of the grouping type "mhss."

[0077] An example of the syntax of this haptics stream type sample group is shown in A of Fig. 21. "HapticsStreamTypeGroupEntry extends HapticSampleGroupEntry('mhst')" is defined. A silent flag (silent_flag) may also be defined.

[0078] An example of the semantics of this haptics stream type sample group is shown in B of Figure 21. The silent flag (silent_flag), when true, indicates that the data packet does not contain haptics data. In this case, the data packet payload size (data_packet_payload_size) is zero.

[0079] For example, in the first information processing device, a content file generating unit may generate information defining a sample group that indicates the presence or absence of media data in a sample by flag information, and in the second information processing device, the information on the stream type may include flag information that indicates the presence or absence of media data in a sample.

[0080] By doing so, it is possible to obtain the same effect as that described above in <Regarding Method 1-3>.

[0081] <Regarding Method 1-3-2> The stream type may also be specified using a grouping type parameter defined in sample-to-group. That is, when "Method 1-3" in the table of Fig. 15 is applied, the stream type may be specified using a grouping type parameter, as shown in the sixth row from the top of the table ("Method 1-3-2" in the table).

[0082] For example, a haptics stream type sample group may be defined as shown in A of FIG. 22. This haptics stream type sample group is a sample group that specifies the stream type, and the group type is set to 'mhst'. This haptics stream type sample group is also stored in a sample group description box. This haptics stream type sample group is not a required sample group, and may be absent or multiple groups may be present.

[0083] As shown in B of FIG. 22, this haptics stream type sample group is a sample group consisting of MIHS samples that do not contain haptic data or MIHS samples that do not contain haptic data within an MIHS track. Whether or not this MIHS sample contains haptic data is identified by a silent flag (silent_flag). Note that even if this haptics stream type sample group does not exist, samples that do not contain haptic data may exist in an MIHS track. Similarly, samples that include haptic data may also exist in an MIHS track.

[0084] In this case, a grouping type parameter (grouping_type_parameter) is defined in the SampleToGroupBox (SampleToGroupBox) of grouping type "mhst." For example, if this grouping type parameter is "mhss," it indicates a sample group consisting of MIHS samples that do not contain haptic data (i.e., samples with silent_flag = 1). If this grouping type parameter is "mhps," it indicates a sample group consisting of MIHS samples that include haptic data (i.e., samples with silent_flag = 0).

[0085] An example of the syntax of this haptics stream type sample group is shown in C of Fig. 22. "HapticsStreamTypeGroupEntry extends HapticSampleGroupEntry('mhst')" may be defined.

[0086] For example, in the first information processing device, a content file generation unit may generate information defining a sample group that indicates the presence or absence of media data in a sample by a grouping type parameter that indicates the stream type of the sample group, and in the second information processing device, the information regarding the stream type may include a grouping type parameter that indicates the stream type of the sample group.

[0087] By doing so, it is possible to obtain the same effect as that described above in <Regarding Method 1-3>.

[0088] <Regarding Method 1-4> Furthermore, when "Method 1" in the table of Figure 15 is applied, a sample entry (mih1) that explicitly indicates the stream type may be defined, as shown in the seventh row from the top of the table ("Method 1-4" in the table).

[0089] For example, as shown in A of Fig. 23, a stream type description box may be provided in a sample entry (MIHS sample entry ('mih1')), and a haptics stream type description box that clearly indicates the stream type may be defined within the stream type description box. Note that this haptics stream type description box is not essential and may be absent or multiple boxes may be present.

[0090] As shown in B of FIG. 23 , this haptics stream type description indicates a sample in an MIHS track, where whether or not it contains haptics data is indicated by a silent flag. Whether or not the sample indicated by this haptics stream type description contains haptics data is identified by the silent flag (silent_flag). For example, if the value of this silent flag is true (silent_flag = 1), this indicates that the MIHS sample indicated by this sample entry does not contain haptics data. Also, if the value of this silent flag is false (silent_flag = 0), this indicates that the MIHS sample indicated by this sample entry contains haptics data.

[0091] An example of the syntax of this haptics stream type description box is shown in A of Fig. 24. "HapticsStreamTypeDescriptionBox()" may be defined. Also, a silent flag (silent_flag) may be set.

[0092] An example of the semantics of this haptics stream type description box is shown in B of Fig. 24. If the silent flag (silent_flag) is true, it indicates that the data packet does not contain haptics data. In this case, the data packet payload size (data_packet_payload_size) is zero.

[0093] For example, in the first information processing device, a content file generation unit may generate, as information on the stream type, information defining a sample that indicates whether or not media data is included in the sample, and in the second information processing device, a control unit may control an output device based on the information defining a sample that indicates whether or not media data is included in the sample, which is included in the information on the stream type.

[0094] In the playback process, by referencing the information defining the sample entry specifying the stream type as described above, it is possible to easily identify samples that include haptic data (i.e., target media data) or samples that do not include haptic data (i.e., target media data). In other words, it is possible to easily identify periods during which haptic data is not output or periods during which haptic data is output, and to appropriately switch the state of the output device depending on the periods. In other words, it is possible to suppress an increase in power consumption during the playback process while suppressing a decrease in playback quality. Furthermore, in the playback process, the information defining the sample entry specifying the stream type can be easily referenced without analyzing the media bitstream. Therefore, it is possible to suppress an increase in power consumption due to analysis of the media bitstream. Furthermore, it is possible to suppress an increase in processing time due to analysis of the media bitstream. In other words, it is possible to suppress an increase in the load of the playback process and an increase in the cost of the playback process.

[0095] <Regarding Methods 1-5> For example, as described with reference to Fig. 2, in the case of an ISOBMFF file, a media bitstream can be divided into a plurality of fragments and managed (fragment movie). This format is used, for example, in the case of streaming distribution. The present technology may also be applied to such fragment movies.

[0096] However, in the case of a fragment movie, it is difficult to manage the information about samples in multiple fragments within a movie box (moov) alone. Therefore, the management information for each fragment's sample may be stored in the management information for that fragment. In other words, the sample management information may be stored in multiple movie fragment boxes (moof).

[0097] In the case of a fragment movie, the structure of an ISOBMFF file is as shown in FIG. 25. That is, in this case, the ISOBMFF file has one movie box, and a movie fragment box (moof) and a movie data box (mdat) for each fragment. In such a case, a sample group description box ('sgpd') may be provided in the track fragment box ('traf') within the movie fragment box (moof), and information defining the stream type may be stored therein. Furthermore, a sample to group description box ('sbgp') may be provided in the track fragment box ('traf') and information linking the stream type to samples may be stored therein. In other words, information related to the stream type (information defining the stream type and information linking the stream type to samples) may be defined for each fragment and stored in a distributed manner in the management information for multiple fragments.

[0098] For example, in the first information processing device, a content file may be configured to manage a plurality of samples constituting the content by dividing them into a plurality of fragments, and a content file generation unit may generate information on the stream type for each fragment and store the information in the content file as management information for each fragment.

[0099] By doing so, the present technology can also be applied to fragment movies.

[0100] In this type of structure, information about the stream type is stored in a distributed manner in the management information for each fragment, which can make it difficult to obtain information about the stream type of fragments other than the one being processed. In other words, since sample information for the next or subsequent fragments to be processed cannot be obtained, it can be difficult to control switching of the output device state across fragments.

[0101] 15 is applied, information about the stream types of multiple fragments may be obtained and analyzed prior to normal media processing, as shown in the eighth row from the top of the table ("Method 1-5" in the table). In other words, in normal media processing, each fragment is processed sequentially in chronological order, but prior to that processing, movie fragment boxes (moof) for multiple fragments may be obtained, and information about the stream types of the multiple fragments may be obtained and analyzed collectively.

[0102] For example, in the second information processing device, a content file may manage a plurality of samples constituting the content by dividing them into a plurality of fragments, and a control unit may control an output device based on information about the stream type of each fragment, which information is stored in the content file as management information for each fragment.

[0103] This makes it possible to easily obtain sample information for the fragments to be processed next and thereafter, and to appropriately control switching of the output device state across fragments. In other words, this technology can also be applied to fragment movies.

[0104] <About Method 1-6> If the ISOBMFF file is a fragment movie, the movie fragment box (moof) of each fragment must be referenced to collect information about the stream type and determine whether each sample contains haptic data. This means that the stream type information must be acquired and analyzed repeatedly, potentially increasing the load on the playback process. In particular, when analyzing the movie fragment boxes (moof) of multiple fragments prior to normal media processing, as in "Method 1-5" in the table in Figure 15, the movie fragment boxes (moof) of the fragments being processed are also analyzed during normal media processing, which increases redundant processing and potentially increases the load on the playback process.

[0105] In order to appropriately control the switching of the output device state, it is necessary to know the timing at which the target media data is output. In other words, it is necessary to know the timing (sample) at which the stream type changes. Therefore, "information about consecutive samples without a change in stream type" may be transmitted. For example, when "Method 1" in the table of FIG. 15 is applied, information about consecutive samples without a change in stream type may be stored as management information in the content file, as shown in the ninth row from the top of the table ("Method 1-6" in the table).

[0106] For example, "information indicating a consecutive period of the same type" may be transmitted as this "information regarding consecutive samples without a change in stream type." A "consecutive period of the same type" indicates a period during which consecutive samples have the same stream type as a reference sample. The reference sample may be any sample within the fragment to be processed. For example, a sample at a predetermined position within the fragment, such as the first or last sample of the fragment, may be used as the reference sample. It may also be possible to specify which sample is used as the reference sample. In other words, a "consecutive period of the same type" can start from any position (any sample) within the fragment. Furthermore, the length (number of samples) of a "consecutive period of the same type" is also arbitrary (it may be any length).

[0107] For example, in the first information processing device, the content file generation unit may generate, as the information on the stream type, information on a same-type consecutive period in which samples of the same stream type occur in succession, and in the second information processing device, the control unit may control the output device based on the information on the stream type, which information on the same-type consecutive period in which samples of the same stream type occur in succession.

[0108] By doing so, in the playback process, it is possible to easily grasp the timing at which the target media data will be output based on the information about the same type consecutive period. In this case, it is sufficient to analyze the information about the same type consecutive period, and there is no need to check and tally the stream type for each sample, so an increase in the load of the playback process can be suppressed.

[0109] Furthermore, this same-type consecutive period (a group of consecutive samples with the same stream type) can be set regardless of the fragment. That is, a group of consecutive samples within a fragment can be set as the same-type consecutive period, or a group of consecutive samples that spans (spans) a fragment can be set as the same-type consecutive period. Therefore, by referencing information about this same-type consecutive period, information about not only the fragment to be processed but also the next and subsequent fragments can be obtained. That is, similar to "Method 1-5" in the table of FIG. 15 , information about the stream type of the sample in the previous fragment can be obtained prior to normal media processing. This allows appropriate control of output device state switching across fragments. Generally, fragment lengths are often measured in seconds. Therefore, considering the startup characteristics of the device, knowing the stream type of the next fragment allows control to be performed in response to device control delays of several hundred milliseconds. In other words, this technology can also be applied to fragment movies.

[0110] The length of the "same type continuous period" may be expressed in terms of the number of samples, or may be expressed in terms of the length of time (sample offset). For example, if the sample time length (Haptics sample duration) is not a fixed length, it may be difficult to determine the appropriate startup timing for the output device based on the number of samples in the "same type continuous period." In such cases, by expressing the length of the "same type continuous period" in terms of the length of time, the appropriate startup timing for the output device can be determined more accurately.

[0111] <Regarding Method 1-6-1> When "Method 1-6" in the table of Figure 15 is applied, as shown in the 10th row from the top of the table, a sample group (mhcs) indicating a period of consecutive samples of the same type may be defined as "information regarding consecutive samples whose stream type does not change" ("Method 1-6-1" in the table).

[0112] For example, a haptics contiguous stream type sample group may be defined as shown in A of FIG. 26. This haptics contiguous stream type sample group is a sample group that indicates a continuous period of the same type, and the group type is set to 'mhcs'. Furthermore, this haptics contiguous stream type sample group is stored in a sample group description box. Furthermore, this haptics contiguous stream type sample group is not a required sample group, and may be absent or multiple samples may be present.

[0113] As shown in B of FIG. 26, this haptics contiguous stream type sample group indicates the number of consecutive samples (number of samples in the same type consecutive period) that have the same stream type as the marked sample (reference sample). In this case, the stream type of the next sample in the same type consecutive period is unknown. In other words, the stream type of the next sample in the same type consecutive period may be the same as or different from the stream type of the sample in the same type consecutive period. Also, if this sample group does not exist, the length (number of samples) of the same type consecutive period is unknown. Also, the grouping type parameter (grouping_type_parameter) is not defined in the SampleToGroupBox of grouping type "mhcs."

[0114] An example of the syntax of this haptics contiguous stream type sample group is shown in A of Fig. 27. "HapticsContiguousStreamTypeGroupEntry extends HapticSampleGroupEntry('mhcs')" may be defined. Also, a contiguous sample count (contiguous_sample_count) may be set.

[0115] An example of the semantics of this haptics contiguous stream type sample group is shown in B of Fig. 27. The contiguous sample count is information indicating the number of consecutive samples that have the same stream type as the marked sample (reference sample), and is set as an unsigned integer.

[0116] For example, in the first information processing device, the content file generation unit may generate, as information regarding a consecutive period of the same type, information defining a group of samples included in the consecutive period of the same type. Also, in the second information processing device, the information regarding a consecutive period of the same type may include information defining a group of samples included in the consecutive period of the same type.

[0117] By doing so, it is possible to suppress an increase in the load of the reproduction process.

[0118] FIG. 28 shows an example where "Method 1-6-1" in the table of FIG. 15 is applied. Consider two consecutive fragments (fragment 1 and fragment 2). Assume that the duration of fragment 1 and fragment 2 is 128 ms each, and the number of samples is 24. Assume also that the stream type of all 24 samples belonging to fragment 1 is silent (silent_flag = 1). Assume also that the stream type of all 24 samples belonging to fragment 2 is silent (silent_flag = 1).

[0119] In this case, for fragment 1, the Sample Group Description Box ('sgpd') and the Sample To Group Description Box ('sbgp') can be defined as shown in FIG. 28.

[0120] As shown in FIG. 28, in this case, two sample group entries are defined in the Sample Group Description Box (Sample Group Description Box('sgpd')) on the left side of the figure. The upper entry (SampleGroupEntry('mhss')) defines a haptics silent stream sample group with a grouping type of "mhss." The lower entry (SampleGroupEntry('mhcs')) defines a haptics contiguous stream type sample group with a grouping type of "mhcs." In other words, this entry defines a continuous period of the same type. This entry also defines that the number of samples is 24 (Sample_count=24). This indicates that the length of the continuous period of the same type is 24 samples.

[0121] Additionally, the Sample To Group Description Box ('sbgp') in the center of the figure defines that a sample group with a grouping type of "mhss" (i.e., a haptics silent stream sample group) is linked to the 24 samples of Fragment 1 (Sample_count=24, Group_description_index=0x10001). Therefore, the stream type of the 24 samples of Fragment 1 is silent (i.e., silent_flag = 1). Note that if this group description index is a value other than "0" (or "(0x10000)"), it indicates that the sample is included in this sample group (i.e., linked to the sample group).

[0122] Additionally, the Sample To Group Description Box ('sbgp') on the right side of the figure defines that the sample group with a grouping type of "mhcs" (i.e., the haptics contiguous stream type sample group) is not linked to the first 23 samples of Fragment 1 (Sample_count=23, Group_description_index=0). Note that if this group description index is "0" (or "(0x10000)"), this indicates that the sample is not included in this sample group (i.e., is not linked to the sample group). Furthermore, it defines that the sample group with a grouping type of "mhcs" (i.e., the haptics contiguous stream type sample group) is linked to the last sample of Fragment 1 (Sample_count=1, Group_description_index=0x10010).

[0123] That is, this sample-to-group entry indicates that the reference sample (marked sample) of the 24-sample "same type consecutive period" defined in the sample group description box is the 24th sample of fragment 1. In other words, that "same type consecutive period" is composed of samples of fragment 2. Therefore, the stream type of the 24 samples of fragment 2 is silent (i.e., slinet_flag = 1).

[0124] FIG. 29 shows another example of applying "Method 1-6-1" in the table of FIG. 15. Consider two consecutive fragments (fragment 1 and fragment 2). Assume that the duration of fragment 1 and fragment 2 is 128 ms each, and that the number of samples is 24. Furthermore, assume that of the 24 samples belonging to fragment 1, the stream type of the first 12 samples is active (silent_flag = 0), and the stream type of the remaining 12 samples is silent (silent_flag = 1). Furthermore, assume that the stream type of all 24 samples belonging to fragment 2 is silent (silent_flag = 1).

[0125] In this case, for fragment 1, the Sample Group Description Box ('sgpd') and the Sample To Group Description Box ('sbgp') can be defined as shown in Figure 29.

[0126] As shown in FIG. 29, in this case, two sample group entries are defined in the Sample Group Description Box ('sgpd') on the left side of the figure. The upper entry (SampleGroupEntry('mhss')) defines a haptics silent stream sample group with a grouping type of "mhss." The lower entry (SampleGroupEntry('mhcs')) defines a haptics contiguous stream type sample group with a grouping type of "mhcs." In other words, this entry defines a continuous period of the same type. This entry also defines that the number of samples is 35 (Sample_count=35). This indicates that the length of the continuous period of the same type is 35 samples.

[0127] Furthermore, the Sample To Group Description Box ('sbgp') in the center of the figure defines that the sample group with a grouping type of "mhss" (i.e., the haptics silent stream sample group) is not linked to the first 12 samples of Fragment 1 (Sample_count=12, Group_description_index=0). It also defines that the sample group with a grouping type of "mhss" (i.e., the haptics silent stream sample group) is linked to the last 12 samples of Fragment 1 (Sample_count=12, Group_description_index=0x10001). Therefore, the stream type of the last 12 samples of Fragment 1 is silent (i.e., silent_flag = 1).

[0128] Additionally, the Sample To Group Description Box ('sbgp') on the right side of the figure defines that a sample group with a grouping type of "mhcs" (i.e., a haptics contiguous stream type sample group) is not linked to the first 12 samples of Fragment 1 (Sample_count=12, Group_description_index=0). It also defines that a sample group with a grouping type of "mhcs" (i.e., a haptics contiguous stream type sample group) is linked to the 13th sample (1 sample) of Fragment 1 (Sample_count=1, Group_description_index=0x10010). It also defines that a sample group with a grouping type of "mhcs" (i.e., a haptics contiguous stream type sample group) is not linked to the last 11 samples of Fragment 1 (Sample_count=11, Group_description_index=0). Note that this third entry may be omitted (undescribed samples are considered to have Group_description_index=0).

[0129] That is, in this sample to group, the reference sample (marked sample) of the "same type consecutive period" of 35 samples in length defined in the sample group description box is the 13th sample of fragment 1. In other words, the "same type consecutive period" is made up of the last 11 samples of fragment 1 and the 24 samples of fragment 2. Therefore, the stream type of the last 11 samples of fragment 1 and the 24 samples of fragment 2 is silent (i.e., slinet_flag = 1).

[0130] The reference sample for the "same type consecutive period" may be fixed to the last sample of the fragment. This eliminates the need for a definition for specifying the reference sample. For example, as shown in A of FIG. 30, the semantics of a haptics contiguous stream type sample group may be defined, and this sample group may indicate the number of consecutive samples that have the same stream type as the last sample in the fragment.

[0131] B in Figure 30 shows an example of this case. We will explain two consecutive fragments (fragment 1 and fragment 2). The time length of fragment 1 and fragment 2 is each 128 ms, and the number of samples is 24. Also, assume that the stream type of all 24 samples belonging to fragment 1 is silent (silent_flag = 1). Also, assume that the stream type of all 24 samples belonging to fragment 2 is silent (silent_flag = 1).

[0132] In this case, for fragment 1, the Sample Group Description Box ('sgpd') and the Sample To Group Description Box ('sbgp') can be defined as shown in B of Figure 30.

[0133] As shown in B of FIG. 30, in this case, two sample group entries are defined in the Sample Group Description Box (Sample Group Description Box('sgpd')). The upper entry (SampleGroupEntry('mhss')) defines a haptics silent stream sample group with a grouping type of "mhss". The lower entry (SampleGroupEntry('mhcs')) defines a haptics contiguous stream type sample group with a grouping type of "mhcs". In other words, this entry defines a consecutive period of the same type. This entry also defines that the number of samples is 24 (Sample_count=24). This indicates that the length of the consecutive period of the same type is 24 samples.

[0134] Additionally, the Sample To Group Description Box ('sbgp') defines that a sample group with a grouping type of "mhss" (i.e., a haptics silent stream sample group) is linked to the 24 samples of Fragment 1 (Sample_count=24, Group_description_index=0x10001). Therefore, the stream type of the 24 samples of Fragment 1 is silent (i.e., silent_flag = 1).

[0135] In this example, since the reference sample is known to be the last sample of fragment 1, there is no need for a sample-to-group description box (an entry linking a sample group with grouping type "mhcs" (i.e., a haptics contiguous stream type sample group) to specify the reference sample)

[0136] That is, in this sample to group, the marked sample of the 24-sample "same type consecutive period" defined in the sample group description box is the last sample of fragment 1. In other words, the "same type consecutive period" is composed of the 24 samples of fragment 2. Therefore, the stream type of the 24 samples of fragment 2 is silent (i.e., slinet_flag = 1).

[0137] <Regarding Method 1-6-1-1> Note that in the case of "Method 1-6-1" in the table of FIG. 15, the stream type of the sample following the consecutive period of the same type was unknown, but the stream type of this sample may be explicitly indicated. For example, when "Method 1-6-1" in the table of FIG. 15 is applied, information regarding a change in the stream type of a sample following the consecutive period of the same type may be explicitly indicated, as shown in the eleventh row from the top of the table ("Method 1-6-1-1" in the table). In other words, information regarding consecutive samples whose stream type does not change may further include information regarding a change in the stream type of a sample following the consecutive period of the same type. This "information regarding a change in stream type" may be, for example, information indicating whether the stream type of a sample following the consecutive period of the same type changes from the stream type of the sample during the consecutive period of the same type, information indicating whether it does not change, or information indicating whether there is a change.

[0138] For example, a haptics contiguous stream type sample group may be defined as shown in A of FIG. 31. This haptics contiguous stream type sample group is a sample group that indicates a continuous period of the same type, and the group type is set to 'mhcs'. Furthermore, this haptics contiguous stream type sample group is stored in a sample group description box. Furthermore, this haptics contiguous stream type sample group is not a required sample group, and may be absent or multiple samples may be present.

[0139] As shown in FIG. 31B, this haptics contiguous stream type sample group indicates the number of consecutive samples (number of samples in the same type consecutive period) that have the same stream type as the marked sample (reference sample). It also indicates whether the stream type of the next sample in the same type consecutive period is the same as the stream type of the sample in the same type consecutive period. If this sample group does not exist, the length (number of samples) of the same type consecutive period is unknown. Furthermore, the grouping type parameter (grouping_type_parameter) is not defined in the SampleToGroupBox of grouping type "mhcs."

[0140] An example of the syntax of this haptics contiguous stream type sample group is shown in A of Fig. 32. "HapticsContiguousStreamTypeGroupEntry extends HapticSampleGroupEntry('mhcs')" may be defined. In addition, an exact flag (exact_flag), a contiguous sample count (contiguous_sample_count), etc. may be set.

[0141] An example of the semantics of this haptic contiguous stream type sample group is shown in FIG. 32B. The exact flag (exact_flag) is flag information indicating whether the stream type of the next sample in the same-type consecutive period indicated by the contiguous sample count will be changed from the stream type of the sample in that same-type consecutive period. For example, if the value of this exact flag is true (exact_flag = 1), this indicates that the stream type of the next sample in the same-type consecutive period will be changed from the stream type of the sample in that same-type consecutive period. On the other hand, if the value of this exact flag is false (exact_flag = 0), this indicates that the stream type of the next sample in the same-type consecutive period will not be changed from the stream type of the sample in that same-type consecutive period. The contiguous sample count is information indicating the number of consecutive samples having the same stream type as the marked sample (reference sample) and is set as an unsigned integer.

[0142] For example, in the first information processing device, the content file generation unit may generate, as the information regarding the same type consecutive period, information indicating whether the stream type will change in the sample following the same type consecutive period. Also, in the second information processing device, the information regarding the same type consecutive period may further include information indicating whether the stream type will change in the sample following the same type consecutive period.

[0143] By doing so, it is possible to suppress an increase in the load of the reproduction process.

[0144] FIG. 33 shows an example where "Method 1-6-1-1" in the table of FIG. 15 is applied. Consider two consecutive fragments (fragment 1 and fragment 2). Assume that the duration of fragment 1 and fragment 2 is 128 ms each, and the number of samples is 24. Assume also that the stream type of all 24 samples belonging to fragment 1 is silent (silent_flag = 1). Assume also that the stream type of the first 12 samples belonging to fragment 2 is silent (silent_flag = 1), and the stream type of the last 12 samples is active (silent_flag = 0).

[0145] In this case, for fragment 1, the Sample Group Description Box ('sgpd') and the Sample To Group Description Box ('sbgp') can be defined as shown in Figure 33.

[0146] As shown in FIG. 33 , in this case, two sample group entries are defined in the Sample Group Description Box (Sample Group Description Box('sgpd')) on the left side of the figure. The upper entry (SampleGroupEntry('mhss')) defines a haptics silent stream sample group with a grouping type of "mhss." The lower entry (SampleGroupEntry('mhcs')) defines a haptics contiguous stream type sample group with a grouping type of "mhcs." That is, this entry defines a same-type consecutive period. This entry also defines that the number of samples is 12 (Sample_count=12). This indicates that the length of the same-type consecutive period is 12 samples. Furthermore, this entry sets the exact flag value to "1" (Exact_flag=1). This indicates that the stream type of the sample following this same-type consecutive period will change from the stream type of the same-type consecutive period.

[0147] Additionally, the Sample To Group Description Box ('sbgp') in the center of the figure defines that a sample group with a grouping type of "mhss" (i.e., a haptics silent stream sample group) is linked to the 24 samples of Fragment 1 (Sample_count=24, Group_description_index=0x10001). Therefore, the stream type of the 24 samples of Fragment 1 is silent (i.e., silent_flag = 1).

[0148] Additionally, the Sample To Group Description Box ('sbgp') on the right side of the figure defines that the sample group with the grouping type "mhcs" (i.e., the haptics contiguous stream type sample group) is not linked to the first 23 samples of Fragment 1 (Sample_count=23, Group_description_index=0). Also, it defines that the sample group with the grouping type "mhcs" (i.e., the haptics contiguous stream type sample group) is linked to the last sample of Fragment 1 (Sample_count=1, Group_description_index=0x10010).

[0149] In other words, this sample-to-group entry indicates that the reference sample (marked sample) of the 24-sample "same type consecutive period" defined in the sample group description box is the 24th sample of fragment 1. In other words, that "same type consecutive period" is composed of the samples of fragment 2. Therefore, the stream type of the first 12 samples of fragment 2 is silent (i.e., slinet_flag = 1). Also, because the exact flag value is set to "1" (Exact_flag = 1) as described above, the stream type of the last 12 samples of fragment 2 is active (i.e., slinet_flag = 0).

[0150] In the case of "Method 1-6-1-1" in the table of Fig. 15, the reference sample for the "same type continuous period" may be fixed to the last sample of the fragment, as in the example of Fig. 30. This eliminates the need for a definition to specify the reference sample.

[0151] <Regarding Method 1-6-2> Furthermore, when "Method 1-6" in the table of Figure 15 is applied, as shown in the 11th row from the top of the table, a box (mhcs) may be defined to store information indicating a period of consecutive samples of the same type as "information regarding consecutive samples with no change in stream type," and the box may be stored in the user data box ("Method 1-6-2" in the table).

[0152] For example, a Haptics Contiguous Stream Type box may be defined as shown in A of FIG. 34. This Haptics Contiguous Stream Type box defines information about a continuous period of the same type, and the group type is set to 'mhcs'. This Haptics Contiguous Stream Type box is also stored in the User Data box ('utda'). This Haptics Contiguous Stream Type box is not a required sample group, and may be absent or multiple boxes may be present.

[0153] As shown in B of FIG. 34, this haptics contiguous stream type box defines the number of consecutive samples (number of samples in the same type consecutive period) that have the same stream type as the last sample (reference sample) of the fragment. In this case, the stream type of the next sample in the same type consecutive period is unknown. In other words, the stream type of the next sample in the same type consecutive period may be the same as or different from the stream type of the sample in the same type consecutive period. If this box does not exist, the number of samples in the same type consecutive period is unknown.

[0154] An example of the syntax of this haptics contiguous stream type box is shown in A of Fig. 35. "HapticsContiguousStreamTypeBox extends FullBox('mhcs',version,0)" may be defined. In addition, a grouping type (grouping_type), a contiguous sample count (contiguous_sample_count), etc. may be set.

[0155] An example of the semantics of this haptics contiguous stream type box is shown in B of Fig. 35. The grouping type (grouping_type) is identification information of the grouping type of the samples and is set as an integer. The contiguous sample count (contiguous_sample_count) is information indicating the number of consecutive samples having the same stream type as the last sample of this fragment (i.e., the number of samples in a consecutive period of the same type) and is set as an unsigned integer.

[0156] For example, in a first information processing device, the content file generation unit may define a box for storing information about the same type consecutive period and store the box in a general-purpose user data box of the content file. Also, in a second information processing device, the information about the same type consecutive period may be stored in a box within the general-purpose user data box of the content file.

[0157] By doing so, it is possible to suppress an increase in the load of the reproduction process.

[0158] Note that this "information about consecutive samples with the same stream type" may be stored in a meta box under a track, rather than in a user data box. Also, this "information about consecutive samples with the same stream type" may be stored in a timed metadata track, which is a track that stores metadata that can change in the time direction (also called timed metadata).

[0159] <Regarding Method 1-6-2-1> Note that in the case of "Method 1-6-2" in the table of FIG. 15, the stream type of the sample following the consecutive period of the same type was unknown, but the stream type of this sample may be explicitly indicated. For example, when "Method 1-6-2" in the table of FIG. 15 is applied, information regarding a change in the stream type of a sample after the consecutive period of the same type may be explicitly indicated, as shown in the bottom row of the table ("Method 1-6-2-1" in the table). In other words, information regarding consecutive samples whose stream type does not change may further include information regarding a change in the stream type of a sample after the consecutive period of the same type. This "information regarding a change in stream type" may be, for example, information indicating whether the stream type of a sample after the consecutive period of the same type changes from the stream type of the sample during the consecutive period of the same type, information indicating whether it does not change, or information indicating whether there is a change.

[0160] For example, a Haptics Contiguous Stream Type box may be defined as shown in A of FIG. 36. This Haptics Contiguous Stream Type box is a sample group indicating a continuous period of the same type, and the group type is set to 'mhcs'. This Haptics Contiguous Stream Type box is also stored in the User Data box ('utda'). This Haptics Contiguous Stream Type box is not a required sample group, and may be absent or multiple boxes may be present.

[0161] As shown in B of Figure 36, this haptics contiguous stream type box indicates the number of consecutive samples (number of samples in the same type consecutive period) that have the same stream type as the last sample (reference sample) of the fragment. It also indicates whether the stream type of the sample following the same type consecutive period is the same as the stream type of the sample in the same type consecutive period. Note that if this box does not exist, the length (number of samples) of the same type consecutive period is unknown.

[0162] An example of the syntax of this haptics contiguous stream type box is shown in A of Fig. 37. "HapticsContiguousStreamTypeBox extends FullBox('mhcs',version,0)" may be defined. In addition, a grouping type (grouping_type), an exact flag (exact_flag), a contiguous sample count (contiguous_sample_count), etc. may be set.

[0163] An example of the semantics of this haptics contiguous stream type box is shown in B of FIG. 37. The grouping type (grouping_type) is identification information of the grouping type of the samples and is set as an integer. The exact flag (exact_flag) is flag information indicating whether the stream type of the next sample in the same type consecutive period will be changed from the stream type of the sample in the same type consecutive period. For example, if the value of this exact flag is true (exact_flag = 1), this indicates that the stream type of the next sample in the same type consecutive period will be changed from the stream type of the sample in the same type consecutive period. On the other hand, if the value of this exact flag is false (exact_flag = 0), this indicates that the stream type of the next sample in the same type consecutive period will not be changed from the stream type of the sample in the same type consecutive period. The contiguous sample count (contiguous_sample_count) is information indicating the number of consecutive samples having the same stream type as the last sample of this fragment (i.e., the number of samples in the same type consecutive period) and is set as an unsigned integer. The reference sample may be arbitrarily set, and in that case, the sample number of the reference sample may be clearly indicated.

[0164] For example, in the first information processing device, the content file generation unit may generate, as information regarding the same type consecutive period, information indicating whether the stream type will change in the sample following the same type consecutive period. Also, in the second information processing device, the information regarding the same type consecutive period may include information indicating whether the stream type will change in the sample following the same type consecutive period.

[0165] By doing so, it is possible to suppress an increase in the load of the reproduction process.

[0166] <About Methods 1-7> In an ISOBMFF file, multiple media data can be stored on one track. This means that media data to be output to different output devices can be mixed on one track. In such cases, if it is not clear which output device the stream type information is for, it may not be possible to properly control the switching of the output device state.

[0167] Therefore, it may be possible to clearly indicate which output device the information on the stream type corresponds to. For example, when "Method 1" in the table of FIG. 15 is applied, as shown in the top row of the table of FIG. 16, device identification information corresponding to the target media data may be stored in the content file as management information ("Method 1-7" in the table of FIG. 16). In other words, the information on the stream type may further include device identification information for identifying the device corresponding to the target media data. Note that in this specification, the device corresponding to the target media data is also referred to as the target device. Furthermore, the device identification information may be any information as long as it is information for identifying the target device.

[0168] For example, in the first information processing device, the content file generation unit may generate, as the information on the stream type, device identification information for identifying an output device that outputs media data corresponding to the stream type, and in the second information processing device, the control unit may control the output device based on the device identification information for identifying the output device corresponding to the stream type, which is included in the information on the stream type.

[0169] By doing so, the output device corresponding to the target device data (i.e., the output device for which the "stream type information" is intended) can be easily identified during playback processing. Therefore, switching of the output device state can be easily and appropriately controlled. Furthermore, since there is no need to analyze the media bitstream for this control, an increase in the load of playback processing can be suppressed.

[0170] <Regarding Method 1-7-1> This device identification information may be applied to a sample group (mhps) whose stream type is active. For example, when "Method 1-7" in the table of Fig. 16 is applied, the device identification information corresponding to samples belonging to a sample group (mhps) whose stream type is active may be defined ("Method 1-7-1" in the table), as shown in the second row from the top of the table.

[0171] For example, a haptics packet stream sample group may be defined as shown in A of FIG. 38. This haptics packet stream sample group is a sample group with an active stream type, and the group type is set to 'mhps'. This haptics packet stream sample group is also stored in a sample group description box. This haptics silent stream sample group is not a required sample group, and may be absent or multiple samples may be present.

[0172] As shown in B of FIG. 38, this haptics packet stream sample group is a sample group in an MIHS track that is composed of samples that contain haptic data. In other words, the MIHS samples that belong to this sample group have a silent flag value of false (silent_flag = 0) and contain haptic data. Note that even if this haptics packet stream sample group does not exist, samples that do not contain haptic data may exist in an MIHS track. Similarly, samples that contain haptic data may also exist in an MIHS track.

[0173] In this example, the device identification information is defined as a grouping type parameter. For example, as shown in A of FIG. 39, a perception ID (perception_id), a perception modality (perception_modality), a channel ID (channel_id), a device ID (device_id), etc. may be defined as the grouping type parameter (device identification information). The perception ID is identification information of the haptic perception. The perception modality indicates the type of perception defined in ISO / IEC 23090-31, 32. The channel ID is identification information of the haptic channel. The device ID is identification information of the associated reference device.

[0174] The grouping type parameter (device identification information) may be any information, and may include information other than the above-mentioned perception ID, perception modality, channel ID, and device ID. Furthermore, the grouping type parameter (device identification information) does not need to include all of the above-mentioned perception ID, perception modality, channel ID, and device ID, but may include at least one of them.

[0175] An example of the syntax of this haptics packet stream sample group is shown in B of Fig. 39. "HapticsPacketStreamSampleGroupEntry extends HapticSampleGroupEntry('mhps')" may be defined.

[0176] For example, in the first information processing device, the content file generation unit may generate, as information on the stream type, information defining a group of samples in which media data exists, and generate device identification information for that group. Also, in the second information processing device, the device identification information may be information on the group of samples in which media data exists.

[0177] By doing so, during playback processing, the target device can be easily identified based on the device identification information in the management information. Therefore, switching of the output device state can be easily and appropriately controlled. Furthermore, since there is no need to analyze the media bitstream for this control, an increase in the load of playback processing can be suppressed.

[0178] In the case of a fragment movie, the device identification information may be stored in a distributed manner in the movie fragment box (moof). In this case, the device identification information may be linked to any sample using the information stored in the sample-to-group. If the stream type of all samples in the fragment is silent (silent_flag=1), there is no need to store the same sample information for each grouping type parameter, which reduces redundancy in multiple sample-to-groups.

[0179] <Regarding Method 1-7-2> The device identification information may also be applied to a haptics stream type sample group (mhst) that specifies the stream type. For example, when "Method 1-7" in the table of FIG. 16 is applied, the device identification information corresponding to a sample belonging to a sample group (mhst) that specifies that the stream type is active may be defined ("Method 1-7-2" in the table), as shown in the third row from the top of the table.

[0180] For example, a haptics stream type sample group may be defined as shown in A of FIG. 40. This haptics stream type sample group is a sample group that specifies the stream type, and the group type is set to 'mhst'. This haptics stream type sample group is also stored in a sample group description box. This haptics stream type sample group is not a required sample group, and may be absent or multiple groups may be present.

[0181] As shown in B of FIG. 40 , this haptics stream type sample group is a sample group in an MIHS track that does not contain haptic data or is composed of samples that do not contain haptic data. Whether or not the samples that make up this haptics stream type sample group contain haptic data is identified by a silent flag (silent_flag). For example, if the value of this silent flag is true (silent_flag = 1), this indicates that the MIHS samples belonging to this sample group do not contain haptic data. Also, if the value of this silent flag is false (silent_flag = 0), this indicates that the MIHS samples belonging to this sample group contain haptic data. Even if this haptics stream type sample group does not exist, samples that do not contain haptic data may exist in an MIHS track. Similarly, samples that include haptic data may also exist in an MIHS track.

[0182] In this example, the device identification information is defined as a grouping type parameter. For example, as shown in A of FIG. 41 , a perception ID (perception_id), a perception modality (perception_modality), a channel ID (channel_id), a device ID (device_id), etc. may be defined as the grouping type parameter (device identification information). The perception ID is identification information of the haptic perception. The perception modality indicates the type of perception defined in ISO / IEC 23090-31, 32. The channel ID is identification information of the haptic channel. The device ID is identification information of the associated reference device.

[0183] The grouping type parameter (device identification information) may be any information, and may include information other than the above-mentioned perception ID, perception modality, channel ID, and device ID. Furthermore, the grouping type parameter (device identification information) does not need to include all of the above-mentioned perception ID, perception modality, channel ID, and device ID, but may include at least one of them.

[0184] An example of the syntax of this haptics packet stream sample group is shown in B of Fig. 41. "HapticsStreamTypeGroupEntry extends HapticSampleGroupEntry('mhst')" may be defined. Also, a silent flag (silent_flag) may be set.

[0185] An example of the semantics of this haptics stream type sample group is shown in C of Figure 41. The silent flag (silent_flag), when true, indicates that the data packet does not contain haptics data. In this case, the data packet payload size (data_packet_payload_size) is zero.

[0186] For example, in the first information processing device, the content file generation unit may generate, as the information on the stream type, information defining a group of samples indicating the presence of media data in the samples, and generate device identification information for the group. Also, in the second information processing device, the device identification information may be information on the group of samples indicating the presence of media data in the samples.

[0187] By doing so, during playback processing, the target device can be easily identified based on the device identification information in the management information. Therefore, switching of the output device state can be easily and appropriately controlled. Furthermore, since there is no need to analyze the media bitstream for this control, an increase in the load of playback processing can be suppressed.

[0188] In the case of a fragment movie, the device identification information may be stored in a distributed manner in the movie fragment box (moof). In this case, the device identification information may be linked to any sample using the information stored in the sample-to-group. If the stream type of all samples in the fragment is silent (silent_flag=1), there is no need to store the same sample information for each grouping type parameter, which reduces redundancy in multiple sample-to-groups.

[0189] <About Method 1-8> Even if the ISOBMFF file is a fragment movie, multiple media data can be stored on one track. In other words, if the ISOBMFF file is a fragment movie, not only will the load on the playback process increase, but it may also become impossible to properly control the switching of the output device state.

[0190] Therefore, the information about the stream type may include information about the consecutive periods of the same type and device identification information. For example, when "Method 1" in the table of FIG. 15 is applied, as shown in the fourth row from the top of the table of FIG. 16, information about consecutive samples whose stream type does not change and device identification information corresponding to the target media data may be stored in the content file as management information ("Method 1-8" in the table of FIG. 16). The information about the consecutive periods of the same type is the same as in the case described in <Regarding Method 1-6>. Furthermore, the device identification information is the same as in the case described in <Regarding Method 1-7>.

[0191] For example, in a first information processing device, a content file generation unit may generate, as information about the stream type, information about a consecutive period of the same type in which successive samples of the same stream type occur, and device identification information for identifying an output device that outputs media data corresponding to that stream type. Also, in a second information processing device, a control unit may control an output device based on information about a consecutive period of the same type in which successive samples of the same stream type occur, which is included in the information about the stream type, and device identification information for identifying an output device that corresponds to that stream type.

[0192] By doing this, during playback processing, it is possible to easily determine the timing at which the target media data is to be output based on the "information regarding the same type continuous period." Furthermore, it is possible to easily identify the output device corresponding to the target device data (i.e., which output device the "information regarding stream type" is intended for) based on the "device identification information." Therefore, even if the ISOBMFF file is a fragment movie, it is possible to easily and appropriately control switching of the output device state. Furthermore, since there is no need to analyze the media bitstream for these controls, it is possible to suppress an increase in the load of the playback processing.

[0193] <Regarding Method 1-8-1> Device identification information may be applied to a haptics contiguous stream type sample group (mhcs). For example, when "Method 1-8" in the table of FIG. 16 is applied, device identification information corresponding to samples belonging to a sample group (mhcs) indicating a continuous period of the same type may be defined ("Method 1-8-1" in the table), as shown in the fifth row from the top of the table.

[0194] For example, a haptics contiguous stream type sample group may be defined as shown in A of FIG. 42. This haptics contiguous stream type sample group is a sample group that indicates a continuous period of the same type, and the group type is set to 'mhcs'. Furthermore, this haptics contiguous stream type sample group is stored in a sample group description box. Furthermore, this haptics contiguous stream type sample group is not a required sample group, and may be absent or multiple samples may be present.

[0195] As shown in FIG. 42B, this haptic contiguous stream type sample group indicates the number of consecutive samples (number of samples in the same type consecutive period) that have the same stream type as the marked sample (reference sample). The stream type of the next sample in the same type consecutive period is unknown. That is, the stream type of the next sample in the same type consecutive period may be the same as or different from the stream type of the sample in the same type consecutive period. If this sample group does not exist, the length (number of samples) of the same type consecutive period is unknown.

[0196] In this example, the device identification information is defined as a grouping type parameter. For example, as shown in A of FIG. 43, a perception ID (perception_id), a perception modality (perception_modality), a channel ID (channel_id), a device ID (device_id), etc. may be defined as the grouping type parameter (device identification information). The perception ID is identification information of the haptic perception. The perception modality indicates the type of perception defined in ISO / IEC 23090-31, 32. The channel ID is identification information of the haptic channel. The device ID is identification information of the associated reference device.

[0197] The grouping type parameter (device identification information) may be any information, and may include information other than the above-mentioned perception ID, perception modality, channel ID, and device ID. Furthermore, the grouping type parameter (device identification information) does not need to include all of the above-mentioned perception ID, perception modality, channel ID, and device ID, but may include at least one of them.

[0198] An example of the syntax of this haptics contiguous stream type sample group is shown in B of Fig. 43. "HapticsContiguousStreamTypeGroupEntry extends HapticSampleGroupEntry('mhcs')" may be defined. Also, a contiguous sample count (contiguous_sample_count) may be set.

[0199] An example of the semantics of this haptics contiguous stream type sample group is shown in C of Fig. 43. The contiguous sample count is information indicating the number of consecutive samples having the same stream type as the marked sample (reference sample) (i.e., the number of samples in a consecutive period of the same type), and is set as an unsigned integer.

[0200] For example, in the first information processing device, the content file generation unit may generate, as information about the same-type consecutive period, information defining a group of samples included in the same-type consecutive period, and generate device identification information for the group. Also, in the second information processing device, the control unit may control the output device based on information about the same-type consecutive period including information defining the group of samples included in the same-type consecutive period, and the device identification information for the group.

[0201] By doing this, during playback processing, it is possible to easily determine the timing at which the target media data will be output based on the "information regarding the same type continuous period." Furthermore, it is possible to easily identify the output device corresponding to the target device data based on the "device identification information." Therefore, even if the ISOBMFF file is a fragment movie, it is possible to easily and appropriately control switching of the output device state. Furthermore, since there is no need to analyze the media bitstream for these controls, it is possible to suppress an increase in the load of the playback processing.

[0202] In the case of a fragment movie, the device identification information may be stored in a distributed manner in the movie fragment box (moof). In this case, the device identification information may be linked to any sample using the information stored in the sample-to-group. If the stream type of all samples in the fragment is silent (silent_flag=1), there is no need to store the same sample information for each grouping type parameter, which reduces redundancy in multiple sample-to-groups.

[0203] <Regarding Method 1-8-1-1> Note that in the case of "Method 1-8-1" in the table of FIG. 16, the stream type of the sample following the consecutive period of the same type was unknown, but the stream type of this sample may be explicitly indicated. For example, when "Method 1-8-1" in the table of FIG. 16 is applied, information regarding a change in the stream type of a sample after the consecutive period of the same type may be explicitly indicated, as shown in the sixth row from the top of the table ("Method 1-8-1-1" in the table). In other words, information regarding consecutive samples whose stream type does not change may further include information regarding a change in the stream type of a sample after the consecutive period of the same type. This "information regarding a change in stream type" may be, for example, information indicating whether the stream type of a sample after the consecutive period of the same type changes from the stream type of the sample during the consecutive period of the same type, information indicating whether it does not change, or information indicating whether there is a change.

[0204] For example, a haptics contiguous stream type sample group may be defined as shown in A of FIG. 44. This haptics contiguous stream type sample group is a sample group that indicates a continuous period of the same type, and the group type is set to 'mhcs'. Furthermore, this haptics contiguous stream type sample group is stored in a sample group description box. Furthermore, this haptics contiguous stream type sample group is not a required sample group, and may be absent or multiple samples may be present.

[0205] As shown in B of Figure 44, this haptics contiguous stream type sample group defines the number of consecutive samples (number of samples in the same type consecutive period) that have the same stream type as the marked sample (reference sample). It also specifies whether the stream type of the next sample in the same type consecutive period is the same as the stream type of the sample in the same type consecutive period. If this sample group does not exist, the length (number of samples) of the same type consecutive period is unknown.

[0206] In this example, the device identification information is defined as a grouping type parameter. For example, as shown in A of FIG. 45, a perception ID (perception_id), a perception modality (perception_modality), a channel ID (channel_id), a device ID (device_id), etc. may be defined as the grouping type parameter (device identification information). The perception ID is identification information of the haptic perception. The perception modality indicates the type of perception defined in ISO / IEC 23090-31, 32. The channel ID is identification information of the haptic channel. The device ID is identification information of the associated reference device.

[0207] The grouping type parameter (device identification information) may be any information, and may include information other than the above-mentioned perception ID, perception modality, channel ID, and device ID. Furthermore, the grouping type parameter (device identification information) does not need to include all of the above-mentioned perception ID, perception modality, channel ID, and device ID, but may include at least one of them.

[0208] An example of the syntax of this haptics contiguous stream type sample group is shown in B of Fig. 45. "HapticsContiguousStreamTypeGroupEntry extends HapticSampleGroupEntry('mhcs')" may be defined. In addition, an exact flag (exact_flag), a contiguous sample count (contiguous_sample_count), etc. may be set.

[0209] An example of the semantics of this haptic contiguous stream type sample group is shown in C of FIG. 45. The exact flag (exact_flag) is flag information indicating whether the stream type of the next sample in the same type consecutive period indicated by the contiguous sample count (contiguous_sample_count) will be changed from the stream type of the sample in the same type consecutive period. For example, if the value of this exact flag is true (exact_flag = 1), this indicates that the stream type of the next sample in the same type consecutive period will be changed from the stream type of the sample in the same type consecutive period. On the other hand, if the value of this exact flag is false (exact_flag = 0), this indicates that the stream type of the next sample in the same type consecutive period will not be changed from the stream type of the sample in the same type consecutive period. The contiguous sample count is information indicating the number of consecutive samples having the same stream type as the marked sample (reference sample) (i.e., the number of samples in the same type consecutive period), and is set as an unsigned integer.

[0210] For example, in the first information processing device, the content file generation unit may generate, as the information regarding the consecutive periods of the same type, information indicating whether the stream type will change in the sample following the consecutive periods of the same type. Also, in the second information processing device, the information regarding the consecutive periods of the same type may further include information indicating whether the stream type will change in the sample following the consecutive periods of the same type.

[0211] By doing so, it is possible to suppress an increase in the load of the reproduction process.

[0212] <Regarding Method 1-8-2> Furthermore, when "Method 1-8" in the table of Figure 16 is applied, as shown in the seventh row from the top of the table, the device identification information corresponding to the samples belonging to the box (mhcs) in the user data box that stores information indicating the continuous period of the same type may be defined ("Method 1-8-2" in the table).

[0213] For example, a Haptics Contiguous Stream Type box may be defined as shown in A of FIG. 46. This Haptics Contiguous Stream Type box defines information about a continuous period of the same type, and the group type is set to 'mhcs'. This Haptics Contiguous Stream Type box is also stored in the User Data box ('utda'). This Haptics Contiguous Stream Type box is not a required sample group, and may be absent or multiple boxes may be present.

[0214] As shown in B of FIG. 46, this Haptics Contiguous Stream Type box defines the number of consecutive samples (number of samples in the same type consecutive period) that have the same stream type as the last sample (reference sample) of the fragment. The stream type of the next sample in the same type consecutive period is unknown. In other words, the stream type of the next sample in the same type consecutive period may be the same as or different from the stream type of the sample in the same type consecutive period. This stream type is indicated by a sample group. Note that if this box does not exist, the length (number of samples) of the same type consecutive period is unknown.

[0215] In this example, the device identification information is defined as a grouping type parameter. For example, as shown in A of FIG. 47, a perception ID (perception_id), a perception modality (perception_modality), a channel ID (channel_id), a device ID (device_id), etc. may be defined as the grouping type parameter (device identification information). The perception ID is identification information of the haptic perception. The perception modality indicates the type of perception defined in ISO / IEC 23090-31, 32. The channel ID is identification information of the haptic channel. The device ID is identification information of the associated reference device.

[0216] The grouping type parameter (device identification information) may be any information, and may include information other than the above-mentioned perception ID, perception modality, channel ID, and device ID. Furthermore, the grouping type parameter (device identification information) does not need to include all of the above-mentioned perception ID, perception modality, channel ID, and device ID, but may include at least one of them.

[0217] An example of the syntax of this haptics contiguous stream type box is shown in B of Fig. 47. "HapticsContiguousStreamTypeBox extends FullBox('mhcs',version,0)" may be defined. In addition, a grouping type (grouping_type), a grouping type parameter (grouping_type_parameter), a contiguous sample count (contiguous_sample_count), and the like may be set.

[0218] An example of the semantics of this haptics contiguous stream type box is shown in C of FIG. 47. The grouping type is identification information for identifying the grouping type of the sample and is set as an integer. The grouping type parameter indicates the subtype of the grouping type. The contiguous sample count is information indicating the number of consecutive samples having the same stream type as the last sample (reference sample) of this fragment (i.e., the number of samples in a continuous period of the same type) and is set as an unsigned integer. Note that the reference sample may be set arbitrarily. In that case, the sample number to be used as the reference sample may be specified explicitly.

[0219] For example, in a first information processing device, a content file generation unit may define a box for storing information about a consecutive period of the same type, store the box in a general-purpose user data box of the content file, and generate device identification information for samples included in the consecutive period of the same type. Also, in a second information processing device, a control unit may control an output device based on the information about the consecutive period of the same type stored in a box in the general-purpose user data box of the content file and the device identification information for samples included in the consecutive period of the same type.

[0220] By doing this, during playback processing, it is possible to easily determine the timing at which the target media data will be output based on the "information regarding the same type continuous period." Furthermore, it is possible to easily identify the output device corresponding to the target device data based on the "device identification information." Therefore, even if the ISOBMFF file is a fragment movie, it is possible to easily and appropriately control switching of the output device state. Furthermore, since there is no need to analyze the media bitstream for these controls, it is possible to suppress an increase in the load of the playback processing.

[0221] In the case of a fragment movie, the device identification information may be stored in a distributed manner in the movie fragment box (moof). In this case, the device identification information may be linked to any sample using the information stored in the sample-to-group. If the stream type of all samples in the fragment is silent (silent_flag=1), there is no need to store the same sample information for each grouping type parameter, which reduces redundancy in multiple sample-to-groups.

[0222] <Regarding Method 1-8-2-1> Note that in the case of "Method 1-8-2" in the table of FIG. 16, the stream type of the sample following the consecutive period of the same type was unknown, but the stream type of this sample may be explicitly indicated. For example, when "Method 1-8-2" in the table of FIG. 16 is applied, information regarding a change in the stream type of a sample after the consecutive period of the same type may be explicitly indicated, as shown in the eighth row from the top of the table ("Method 1-8-2-1" in the table). In other words, information regarding consecutive samples whose stream type does not change may further include information regarding a change in the stream type of a sample after the consecutive period of the same type. This "information regarding a change in stream type" may be, for example, information indicating whether the stream type of a sample after the consecutive period of the same type changes from the stream type of the sample during the consecutive period of the same type, information indicating whether it does not change, or information indicating whether there is a change.

[0223] For example, a Haptics Contiguous Stream Type box may be defined as shown in A of FIG. 48. This Haptics Contiguous Stream Type box is a sample group indicating a continuous period of the same type, and the group type is set to 'mhcs'. This Haptics Contiguous Stream Type box is also stored in the User Data box ('utda'). This Haptics Contiguous Stream Type box is not a required sample group, and may be absent or multiple boxes may be present.

[0224] As shown in B of Figure 48, this Haptics Contiguous Stream Type box defines the number of consecutive samples (number of samples in the same type consecutive period) that have the same stream type as the last sample (reference sample) of the fragment. It also specifies whether the stream type of the next sample in the same type consecutive period is the same as the stream type of the sample in the same type consecutive period. This stream type is indicated by a sample group. Note that if this box does not exist, the length (number of samples) of the same type consecutive period is unknown.

[0225] In this example, the device identification information is defined as a grouping type parameter. For example, as shown in A of FIG. 49, a perception ID (perception_id), a perception modality (perception_modality), a channel ID (channel_id), a device ID (device_id), etc. may be defined as the grouping type parameter (device identification information). The perception ID is identification information of the haptic perception. The perception modality indicates the type of perception specified in ISO / IEC 23090-31, 32. The channel ID is identification information of the haptic channel. The device ID is identification information of the associated reference device.

[0226] The grouping type parameter (device identification information) may be any information, and may include information other than the above-mentioned perception ID, perception modality, channel ID, and device ID. Furthermore, the grouping type parameter (device identification information) does not need to include all of the above-mentioned perception ID, perception modality, channel ID, and device ID, but may include at least one of them.

[0227] An example of the syntax of this haptics contiguous stream type box is shown in B of Fig. 49. "HapticsContiguousStreamTypeBox extends FullBox('mhcs',version,0)" may be defined. In addition, a grouping type (grouping_type), a grouping type parameter (grouping_type_parameter), an exact flag (exact_flag), a contiguous sample count (contiguous_sample_count), and the like may be set.

[0228] An example of the semantics of this haptics contiguous stream type box is shown in C of FIG. 49. The grouping type is identification information for identifying the grouping type of the samples and is set as an integer. The grouping type parameter indicates a subtype of the grouping type. The exact flag indicates that the next stream type in the same type consecutive period indicated by the contiguous sample count will be changed. The contiguous sample count is information indicating the number of consecutive samples having the same stream type as the last sample (reference sample) of this fragment (i.e., the number of samples in the same type consecutive period) and is set as an unsigned integer. Note that the reference sample may be set arbitrarily. In that case, the sample number to be used as the reference sample may be specified explicitly.

[0229] For example, in the first information processing device, the content file generation unit may generate, as information regarding the consecutive periods of the same type, information indicating whether the stream type will change in the sample following the consecutive periods of the same type. Also, in the second information processing device, the information regarding the consecutive periods of the same type may include information indicating whether the stream type will change in the sample following the consecutive periods of the same type.

[0230] By doing so, it is possible to suppress an increase in the load of the reproduction process.

[0231] <Regarding Methods 1-9> Whether or not "information on the sample stream type" to which the present technology is applied is present in a content file cannot be determined until the content file is acquired and analyzed. Therefore, for example, when controlling the distribution of an ISOBMFF file using an MPD, as in MPEG-DASH described in Non-Patent Document 4, it is difficult to determine, at the control stage, whether or not "information on the sample stream type" to which the present technology is applied is present in the content file. Therefore, at the control stage, it is difficult to determine the load of the playback process in advance and estimate the processing cost. As a result, it may be difficult to appropriately control distribution so as to suppress an increase in the load of the playback process. As a result, there is a risk of the load of the playback process increasing.

[0232] Therefore, when "Method 1" in the table of Figure 15 is applied, as shown in the ninth row from the top of the table of Figure 16, the distribution control file may indicate that information regarding the stream type of the sample exists as management information in the content file ("Method 1-9" in the table of Figure 16).

[0233] 50 is a diagram showing an example of an MPD description. In this example, a supplemental property is defined in a representation under an adaptation set, and its descriptor indicates that "information about the stream type exists as management information in the content file" (schemeIdUri="urn:mpeg:haptics:2024:hapticsStreamTypeInfo). In this way, the MPD representation may use the descriptor of the supplemental property to indicate that "information about the stream type exists as management information in the content file."

[0234] The location indicating that "information about the stream type exists as management information in the content file" may be anywhere in the MPD. Furthermore, the contents of the stream type, grouping type, grouping type parameters, etc. may be defined as elements of this supplemental property.

[0235] For example, the first information processing device may further include a delivery control file generation unit that generates a delivery control file that controls delivery of content files, and the delivery control file generation unit may store information in the generated delivery control file indicating that "information on the stream type exists as management information in the content file." The second information processing device may further include a file acquisition unit that acquires a delivery control file that controls delivery of content files, and a delivery control file analysis unit that analyzes the delivery control file to determine whether information on the stream type exists as management information in the content file. If the second information processing device determines, based on the analysis result of the delivery control file, that information on the stream type exists as management information in the content file, the control unit may control an output device based on the information on the stream type.

[0236] By doing this, during the distribution control stage, it is possible to easily and appropriately control the switching of the output device state based on the information stored in the distribution control file indicating that "information about the stream type exists as management information in the content file," thereby suppressing an increase in the load of the playback process.

[0237] <Regarding Method 1-10> In the above, an ISOBMF file has been used as an example of a container file, but the format and specifications of the container file that provides an extended definition of media associated with 3D data may be any format and are not limited to ISOBMFF. For example, when "Method 1" in the table of FIG. 15 is applied, information regarding the stream type of the sample may be stored in a Matryoshka Media Container, as shown in the bottom row of the table of FIG. 16 ("Method 1-10" in the table of FIG. 16). For example, a Matryoshka Media Container, as shown in FIG. 51, may be used. Of course, other formats are also possible.

[0238] <Application of Configuration> The first information processing device and the second information processing device may have any configuration and are not limited to the above-mentioned example. For example, in the second information processing device, the file acquisition unit may further acquire a content file. Then, the decoding unit may decode the media bitstream stored in the acquired content file to generate media data.

[0239] The second information processing device may further include an output processing unit that uses the generated media data to generate output media data that is data for output.

[0240] The second information processing device may further include an output unit that outputs output media data under the control of the control unit.

[0241] <Combinations> In this specification, a description of a higher-level method also applies to lower-level methods belonging to that method, unless a contradiction arises. For example, when it is described that "Method 1 may be applied," any one or more of the lower-level methods such as "Method 1-1" to "Method 1-10" in the tables of Figures 15 and 16 may be applied, or any one or more of the even lower-level methods such as "Method 1-3-1" and "Method 1-3-2" in the tables may be applied.

[0242] Each of the above-described methods may be applied in combination with any other method as long as no contradiction occurs. Three or more methods may be applied in combination. For example, any two or more of "Method 1-1" to "Method 1-10" in the tables of FIGS. 15 and 16 may be applied in combination. The methods shown in the tables of FIGS. 15 and 16 can be combined with other methods regardless of their hierarchy. Of course, they may also be combined with methods not mentioned above.

[0243] 4. First Embodiment File Generation Device The present technology described above can be applied to any device. Fig. 52 is a block diagram showing an example of the configuration of a file generation device, which is one aspect of an information processing device to which the present technology is applied. The file generation device 300 shown in Fig. 52 is a device that encodes media data and stores the resulting media bitstream in a container file.

[0244] Note that Fig. 52 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in file generation device 300, there may be processing units that are not shown as blocks in Fig. 52, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 52.

[0245] As shown in FIG. 52, the file generation device 300 has a media acquisition unit 311, a preprocessing unit 312, an encoding unit 313, a content file generation unit 314, a distribution control file generation unit 315, a storage unit 316, and a communication unit 317.

[0246] Media acquisition unit 311 performs processing related to the acquisition of data supplied from outside file generation device 300. For example, media acquisition unit 311 may acquire media data supplied to file generation device 300. Media acquisition unit 311 may also acquire other information related to the media data. Media acquisition unit 311 may supply the acquired media data, etc. to preprocessing unit 312.

[0247] The pre-processing unit 312 executes pre-processing on the data supplied from the media acquisition unit 311 before encoding. For example, the pre-processing unit 312 may acquire media data, etc. supplied from the media acquisition unit 311. The pre-processing unit 312 may acquire meta-information related to the media data from the acquired media data, etc. The pre-processing unit 312 may supply the acquired media data, etc. to the encoding unit 313. The pre-processing unit 312 may supply the meta-information acquired from the media data, etc. (meta-information related to the media data) to the content file generation unit 314.

[0248] The encoding unit 313 executes processing related to encoding. For example, the encoding unit 313 may acquire media data, etc. supplied from the preprocessing unit 312. The encoding unit 313 may encode the acquired media data, etc. using an encoding method corresponding to the data. For example, the encoding unit 313 may encode the media data and generate a media bitstream. The encoding unit 313 may supply the generated media bitstream, etc. to the content file generation unit 314.

[0249] The content file generation unit 314 performs processing related to the generation of a content file. For example, the content file generation unit 314 may acquire a media bitstream or the like supplied from the encoding unit 313. The content file generation unit 314 may generate a content file, such as an ISOBMFF file or a Matryoshka media container, as content. The content file generation unit 314 may store the media bitstream or the like in the generated content file. The content file generation unit 314 may acquire meta-information (meta-information related to media data) supplied from the pre-processing unit 312. The content file generation unit 314 may store the meta-information (meta-information related to media data) in the container file. The content file generation unit 314 may supply the generated container file to the storage unit 316. The content file generation unit 314 may also supply the generated container file to the distribution control file generation unit 315.

[0250] The delivery control file generation unit 315 performs processing related to the generation of a delivery control file. For example, the delivery control file generation unit 315 may acquire a content file supplied from the content file generation unit 314. The delivery control file generation unit 315 may generate a delivery control file such as an MPD based on information stored in the acquired content file. The delivery control file generation unit 315 may supply the generated delivery control file to the storage unit 316.

[0251] The storage unit 316 includes any storage medium, such as a hard disk or semiconductor memory, and performs processing related to data storage. For example, the storage unit 316 may acquire a content file supplied from the content file generation unit 314. The storage unit 316 may also acquire a delivery control file supplied from the delivery control file generation unit 315. The storage unit 316 may store the acquired content file or delivery control file. The storage unit 316 may supply the stored content file or delivery control file to the communication unit 317 in response to a request from a user, the communication unit 317, or the like, or at a predetermined timing. The storage unit 316 may be removable media that is detachable from the file generation device 300. The content file or delivery control file may be output from the file generation device 300 (supplied to the outside of the file generation device 300) by removing the storage unit 316 from the file generation device 300. That is, the content file and the distribution control file may be output from file generation device 300 (supplied to the outside of file generation device 300) via storage unit 316, which is removable media. Therefore, storage unit 316 can also be called an output unit (or supply unit).

[0252] The communication unit 317 has a communication function for communicating with a device outside the file generation device 300 (for example, a content file distribution server or a playback device), and executes processing related to the supply of content files and distribution control files via that communication. For example, the communication unit 317 may acquire content files and distribution control files supplied from the storage unit 316. The communication unit 317 may communicate with a device outside the file generation device 300 using the communication function, and supply the acquired content files and distribution control files to the communication partner via that communication (output them to the outside of the file generation device 300). Therefore, the communication unit 317 can also be called an output unit (or supply unit).

[0253] <Application of the Present Technology> The file generation device 300 configured as above may be the first information processing device described above, and one or more of the present technology described above in <3. Transmission of information related to stream type> may be applied.

[0254] For example, file generation device 300 (first information processing device) may include an encoding unit 313 that encodes media data related to haptics and generates a media bitstream, and a content file generation unit 314 that generates a content file for storing content made up of multiple samples that are played back continuously in the time direction, divides the media bitstream into samples and stores them in the content file, generates information about the stream type indicating the presence or absence of media data in the sample, and stores the information in the content file as management information. In other words, media acquisition unit 311, preprocessing unit 312, distribution control file generation unit 315, storage unit 316, and communication unit 317 may be omitted.

[0255] The content file generation unit 314 may also generate, as information related to the stream type, information defining a group of samples for which no media data exists. The content file generation unit 314 may also generate, as information related to the stream type, information defining a group of samples for which media data exists. The content file generation unit 314 may also generate, as information related to the stream type, information defining a group of samples indicating the presence or absence of media data in the samples. The content file generation unit 314 may also generate information defining a sample group for which the presence or absence of media data in the samples is indicated by flag information. The content file generation unit 314 may also generate information defining a sample group for which the presence or absence of media data in the samples is indicated by a grouping type parameter indicating the stream type of the sample group. The content file generation unit 314 may also generate, as information related to the stream type, information defining a sample for which the presence or absence of media data in the samples is indicated.

[0256] Alternatively, the content file may be configured to divide the multiple samples that make up the content into multiple fragments and manage them, and the content file generation unit 314 may generate information about the stream type for each fragment and store it in the content file as management information for each fragment.

[0257] The content file generation unit 314 may also generate, as information about the stream type, information about a same-type consecutive period in which consecutive samples of the same stream type occur. The content file generation unit 314 may also generate, as information about the same-type consecutive period, information defining a group of samples included in the same-type consecutive period. The content file generation unit 314 may also generate, as information about the same-type consecutive period, information indicating whether the stream type will change in the sample following the same-type consecutive period. The content file generation unit 314 may also define a box for storing information about the same-type consecutive period and store the box in a general-purpose user data box in the content file. The content file generation unit 314 may also generate, as information about the same-type consecutive period, information indicating whether the stream type will change in the sample following the same-type consecutive period.

[0258] Alternatively, the content file generation unit 314 may generate, as information about the stream type, device identification information for identifying an output device that outputs media data corresponding to the stream type. Alternatively, the content file generation unit 314 may generate, as information about the stream type, information defining a group of samples in which media data exists, and generate device identification information for the group. Alternatively, the content file generation unit 314 may generate, as information about the stream type, information defining a group of samples indicating the presence of media data in the samples, and generate device identification information for the group.

[0259] The content file generation unit 314 may also generate, as information about the stream type, information about a same-type consecutive period in which consecutive samples of the same stream type occur, and device identification information for identifying an output device that outputs media data corresponding to that stream type. The content file generation unit 314 may also generate, as information about the same-type consecutive period, information defining a group of samples included in the same-type consecutive period, and generate device identification information for that group. The content file generation unit 314 may also generate, as information about the same-type consecutive period, information indicating whether the stream type will change in the sample following the same-type consecutive period. The content file generation unit 314 may also define a box for storing information about the same-type consecutive period, store the box in a general-purpose user data box in the content file, and generate device identification information for the samples included in the same-type consecutive period. The content file generation unit 314 may also generate, as information about the same-type consecutive period, information indicating whether the stream type will change in the sample following the same-type consecutive period.

[0260] Furthermore, the file generation device 300 may further include a distribution control file generation unit 315 that generates a distribution control file that controls the distribution of content files. The distribution control file generation unit 315 may then store information in the generated distribution control file indicating that "information relating to the stream type exists in the content file as management information."

[0261] By having the above-described configuration, the file generation device 300 can obtain the same effects as those described above in <3. Transmission of information related to stream type>.

[0262] <File Generation Process Flow 1> An example of the flow of file generation process executed by this file generation device 300 will be described with reference to the flowchart of FIG.

[0263] When the file generation process is started, the media acquisition unit 311 of the file generation device 300 acquires media data supplied from outside the file generation device 300 in step S301.

[0264] In step S302, the preprocessing unit 312 preprocesses the acquired media data.

[0265] In step S303, the encoding unit 313 encodes the media data to generate a media bitstream.

[0266] In step S304, the content file generator 314 generates a content file as content and stores the generated media bitstream in the content file. In step S305, the content file generator 314 generates information about the stream type of samples for the media bitstream and stores the information in the content file as management information.

[0267] In step S306, the delivery control file generator 315 generates a delivery control file that stores information for controlling the delivery of the content file.

[0268] In step S307, the storage unit 316 stores the generated content file and distribution control file.

[0269] In step S308, the communication unit 317 supplies the content file and the distribution control file to a device external to the file generation device 300.

[0270] When the process of step S308 is completed, the file generation process ends.

[0271] <Application of the Present Technology> The file generation process flow as described above may be the first information processing method (or the first program) described above, and one or more of the present technology described above in <3. Transmission of information related to stream type> may be applied.

[0272] For example, this file generation process (first information processing method or first program) may include the following processes: step S303, which encodes media data related to haptics and generates a media bitstream; and steps S304 and S305, which generate a content file for storing content made up of a plurality of samples played back continuously in the time direction, divide the media bitstream into sample units and store them in the content file, generate information about the stream type indicating whether or not media data is present in the sample, and store the information in the content file as management information. In other words, the processes of steps S301, S302, S306, S307, and S308 may be omitted.

[0273] In step S305, the content file generation unit 314 may generate, as information related to the stream type, information defining a group of samples for which no media data exists. In step S305, the content file generation unit 314 may generate, as information related to the stream type, information defining a group of samples for which media data exists. In step S305, the content file generation unit 314 may generate, as information related to the stream type, information defining a group of samples indicating the presence or absence of media data in the samples. In step S305, the content file generation unit 314 may generate, as information related to the stream type, information defining a sample group for which the presence or absence of media data in the samples is indicated by flag information. In step S305, the content file generation unit 314 may generate, as information related to the stream type, information defining a sample group for which the presence or absence of media data in the samples is indicated by a grouping type parameter indicating the stream type of the sample group. In step S305, the content file generation unit 314 may generate, as information related to the stream type, information defining a sample for which the presence or absence of media data in the samples is indicated.

[0274] Alternatively, the content file may be configured to divide the multiple samples that make up the content into multiple fragments and manage them, and in step S305, the content file generation unit 314 may generate information regarding the stream type for each fragment and store it in the content file as management information for each fragment.

[0275] In step S305, the content file generation unit 314 may generate, as information about the stream type, information about a same-type consecutive period in which consecutive samples of the same stream type occur. In step S305, the content file generation unit 314 may generate, as information about the same-type consecutive period, information defining a group of samples included in the same-type consecutive period. In step S305, the content file generation unit 314 may generate, as information about the same-type consecutive period, information indicating whether the stream type will change in the sample following the same-type consecutive period. In step S305, the content file generation unit 314 may define a box for storing information about the same-type consecutive period and store the box in a general-purpose user data box in the content file. In step S305, the content file generation unit 314 may generate, as information about the same-type consecutive period, information indicating whether the stream type will change in the sample following the same-type consecutive period.

[0276] In step S305, the content file generation unit 314 may generate, as information about the stream type, device identification information for identifying an output device that outputs media data corresponding to the stream type. In step S305, the content file generation unit 314 may generate, as information about the stream type, information defining a group of samples in which media data exists, and generate device identification information for the group. In step S305, the content file generation unit 314 may generate, as information about the stream type, information defining a group of samples indicating the presence of media data in the samples, and generate device identification information for the group.

[0277] In step S305, the content file generation unit 314 may generate, as information about the stream type, information about a same-type consecutive period in which consecutive samples of the same stream type occur, and device identification information for identifying an output device that outputs media data corresponding to that stream type. In step S305, the content file generation unit 314 may generate, as information about the same-type consecutive period, information defining a group of samples included in the same-type consecutive period, and generate device identification information for that group. In step S305, the content file generation unit 314 may generate, as information about the same-type consecutive period, information indicating whether the stream type changes in the sample next to the same-type consecutive period. In step S305, the content file generation unit 314 may define a box for storing information about the same-type consecutive period, store the box in a general-purpose user data box in the content file, and generate device identification information for the samples included in the same-type consecutive period. In step S305, the content file generating unit 314 may generate, as information relating to the same type consecutive period, information indicating whether the stream type will change in the next sample of the same type consecutive period.

[0278] Also, in step S306, the distribution control file generation unit 315 may generate a distribution control file that controls the distribution of the content file, and store information in the generated distribution control file indicating that "information regarding the stream type exists in the content file as management information."

[0279] By performing each process as described above, the file generation device 300 can obtain the same effects as those described above in <3. Transmission of information related to stream type>.

[0280] <Flow 2 of file generation processing> In such file generation processing, for example, when "Method 1-1," "Method 1-2," or "Method 1-3" in the table of FIG. 15 is applied, each process may be executed in the flow shown in the flowchart of FIG. 54.

[0281] When the file generation process starts, in step S321, the media acquisition unit 311 generates haptic content synchronized with the AV media.

[0282] In step S322, the content file generation unit 314 determines the type of sample group to be added to the file and generates it.

[0283] In step S323, the encoding unit 313 encodes the haptic media.

[0284] In step S324, the content file generation unit 314 sets the sample time information and the access information.

[0285] In step S325, the content file generating unit 314 determines whether the sample corresponds to the type of the sample group. If it is determined that the sample corresponds to the type of the sample group, the process proceeds to step S326.

[0286] In step S326, the content file generating unit 314 sets sample-to-group information as the target sample of the sample group type. When the process of step S326 ends, the process proceeds to step S328.

[0287] If it is determined in step S325 that the sample does not correspond to the sample group type, the process proceeds to step S327. In step S327, the content file generation unit 314 sets the sample to group information as a non-target sample of the sample group type. When the process of step S327 is completed, the process proceeds to step S328.

[0288] In step S328, the content file generation unit 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S323. If it is determined in step S328 that all content has been processed, the process proceeds to step S329.

[0289] In step S329, the content file generating unit 314 generates a sample-to-group for the type of sample group.

[0290] In step S330, the communication unit 317 outputs the generated file. When the process of step S330 ends, the file generation process ends.

[0291] <Flow of File Generation Processing 3> In such file generation processing, for example, when "Method 1-4" in the table of FIG. 15 is applied, each process may be executed in the flow shown in the flowchart of FIG.

[0292] When the file generation process starts, in step S341, the media acquisition unit 311 generates haptic content synchronized with the AV media.

[0293] In step S342, the content file generation unit 314 determines the type of sample group to be added to the file and generates it.

[0294] In step S343, the encoding unit 313 encodes the haptic media.

[0295] In step S344, the content file generation unit 314 sets the sample time information and the access information.

[0296] In step S345, the content file generating unit 314 determines whether the sample is of the determined stream type. If it is determined that the sample is of the determined stream type, the process proceeds to step S346.

[0297] In step S346, the content file generator 314 indexes the sample entry having the determined stream type. After the process of step S346 is completed, the process proceeds to step S348.

[0298] If it is determined in step S345 that the sample does not have the determined stream type, the process proceeds to step S347. In step S347, the content file generator 314 indexes sample entries that do not have the determined stream type. When the process of step S347 is completed, the process proceeds to step S348.

[0299] In step S348, the content file generation unit 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S343. If it is determined in step S348 that all content has been processed, the process proceeds to step S349.

[0300] In step S349, the communication unit 317 outputs the generated file. When the process of step S349 ends, the file generation process ends.

[0301] <Flow 4 of File Generation Processing> In such file generation processing, for example, when "Method 1-5" in the table of FIG. 15 is applied, each process may be executed in the flow shown in the flowchart of FIG.

[0302] When the file generation process starts, in step S371, the media acquisition unit 311 generates haptic content synchronized with the AV media.

[0303] In step S372, the content file generation unit 314 determines the type of sample group to be added to the file and generates it.

[0304] In step S373, the content file generation unit 314 sets the fragment unit.

[0305] In step S374, the encoding unit 313 encodes the haptic media.

[0306] In step S375, the content file generation unit 314 sets the sample time information and the access information.

[0307] In step S376, the content file generation unit 314 determines whether the sample corresponds to the type of the sample group. If it is determined that the sample corresponds to the type of the sample group, the process proceeds to step S377.

[0308] In step S377, the content file generating unit 314 sets the sample to group information as a target sample of the sample group type. When the process of step S377 ends, the process proceeds to step S379.

[0309] If it is determined in step S376 that the sample does not correspond to the sample group type, the process proceeds to step S378. In step S378, the content file generation unit 314 sets the sample to group information as a non-target sample of the sample group type. When the process of step S378 ends, the process proceeds to step S379.

[0310] In step S379, the content file generating unit 314 generates sample-to-groups of the sample group type for each fragment.

[0311] In step S380, the content file generator 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S374. If it is determined in step S380 that all content has been processed, the process proceeds to step S381.

[0312] In step S381, the communication unit 317 outputs the generated file. When the process of step S381 ends, the file generation process ends.

[0313] <File Creation Process Flow 5> Another example of the file creation process flow when "Method 1-5" in the table of FIG. 15 is applied will be described with reference to the flowchart of FIG.

[0314] When the file generation process starts, in step S401, the media acquisition unit 311 generates haptic content synchronized with AV media.

[0315] In step S402, the content file generation unit 314 determines the stream type of the sample entry to be added to the file and generates it.

[0316] In step S403, the content file generation unit 314 sets a fragment unit.

[0317] In step S404, the encoding unit 313 encodes the haptic media.

[0318] In step S405, the content file generation unit 314 sets sample time information and access information.

[0319] In step S406, the content file generation unit 314 determines whether the sample is of the determined stream type. If it is determined that the sample is of the determined stream type, the process proceeds to step S407.

[0320] In step S407, the content file generator 314 indexes the sample entry having the determined stream type. After the process of step S407 is completed, the process proceeds to step S409.

[0321] If it is determined in step S406 that the sample does not have the determined stream type, the process proceeds to step S408. In step S408, the content file generation unit 314 indexes sample entries that do not have the determined stream type. When the process of step S408 is completed, the process proceeds to step S409.

[0322] In step S409, the content file generating unit 314 generates a sample entry for each fragment.

[0323] In step S410, the content file generation unit 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S404. If it is determined in step S410 that all content has been processed, the process proceeds to step S411.

[0324] In step S411, the communication unit 317 outputs the generated file. When the process of step S411 ends, the file generation process ends.

[0325] <File Creation Process Flow 6> An example of the file creation process flow when "Method 1-6-1" in the table of FIG. 15 is applied will be described with reference to the flowchart of FIG.

[0326] When the file generation process starts, in step S431, the media acquisition unit 311 generates haptic content synchronized with the AV media.

[0327] In step S432, the content file generation unit 314 determines the type of sample group to be added to the file and generates it.

[0328] In step S433, the content file generation unit 314 sets a fragment unit that is equal to or greater than the maximum value (max) of the rise time of the reference device.

[0329] In step S434, the encoding unit 313 encodes the haptic media.

[0330] In step S435, the content file generation unit 314 sets the sample time information and the access information.

[0331] In step S436, the content file generation unit 314 determines whether the sample corresponds to the type of the sample group. If it is determined that the sample corresponds to the type of the sample group, the process proceeds to step S437.

[0332] In step S437, the content file generating unit 314 sets sample-to-group information as the target sample of the sample group type. When the process of step S437 ends, the process proceeds to step S439.

[0333] If it is determined in step S436 that the sample does not correspond to the sample group type, the process proceeds to step S438. In step S438, the content file generation unit 314 sets the sample to group information as a non-target sample of the sample group type. When the process of step S438 ends, the process proceeds to step S439.

[0334] In step S439, the content file generating unit 314 generates sample-to-groups of the sample group type in units of fragments.

[0335] In step S440, the content file generating unit 314 generates sample-to-group information of the type of sample group of an arbitrary sample of the immediately preceding fragment in fragment units.

[0336] In step S441, the content file generation unit 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S434. If it is determined in step S441 that all content has been processed, the process proceeds to step S442.

[0337] In step S442, the communication unit 317 outputs the generated file. When the process of step S442 ends, the file generation process ends.

[0338] <File Creation Process Flow 7> An example of the file creation process flow when "Method 1-6-2" in the table of FIG. 15 is applied will be described with reference to the flowchart of FIG.

[0339] When the file generation process starts, in step S461, the media acquisition unit 311 generates haptic content synchronized with the AV media.

[0340] In step S462, the content file generation unit 314 determines the type of sample group to be added to the file and generates it.

[0341] In step S463, the content file generation unit 314 sets a fragment unit that is equal to or greater than the maximum value (max) of the rise time of the reference device.

[0342] In step S464, the encoding unit 313 encodes the haptic media.

[0343] In step S465, the content file generation unit 314 sets the sample time information and the access information.

[0344] In step S466, the content file generation unit 314 determines whether the sample corresponds to the type of the sample group. If it is determined that the sample corresponds to the type of the sample group, the process proceeds to step S467.

[0345] In step S467, the content file generating unit 314 sets sample-to-group information as the target sample of the sample group type. When the process of step S467 ends, the process proceeds to step S469.

[0346] If it is determined in step S466 that the sample does not correspond to the sample group type, the process proceeds to step S468. In step S468, the content file generation unit 314 sets the sample to group information as a non-target sample of the sample group type. When the process of step S468 ends, the process proceeds to step S469.

[0347] In step S469, the content file generating unit 314 generates sample-to-groups of the sample group type in units of fragments.

[0348] In step S470, the content file generating unit 314 generates a stream type box of the type continuity information of the sample group of any sample of the immediately preceding fragment in units of fragments.

[0349] In step S471, the content file generation unit 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S464. If it is determined in step S471 that all content has been processed, the process proceeds to step S472.

[0350] In step S472, the communication unit 317 outputs the generated file. When the process of step S472 ends, the file generation process ends.

[0351] <File Generation Process Flow 8> An example of the file generation process flow when "Method 1-7" in the table of FIG. 16 is applied will be described with reference to the flowchart of FIG.

[0352] When the file generation process starts, in step S491, the media acquisition unit 311 generates haptic content synchronized with the AV media.

[0353] In step S492, the content file generation unit 314 determines and generates the types of sample groups for each of a plurality of devices to be added to the file.

[0354] In step S494, the encoding unit 313 encodes the haptic media.

[0355] In step S494, the content file generation unit 314 sets the sample time information and the access information.

[0356] In step S495, the content file generation unit 314 determines whether the sample corresponds to the type of sample group corresponding to the device. If it is determined that the sample corresponds to the type of sample group corresponding to the device, the process proceeds to step S496.

[0357] In step S496, the content file generating unit 314 sets sample-to-group information as the target sample of the sample group type corresponding to the device. When the process of step S496 ends, the process proceeds to step S498.

[0358] If it is determined in step S495 that the sample does not correspond to the type of sample group corresponding to the device, the process proceeds to step S497. In step S497, the content file generation unit 314 sets the sample-to-group information as a non-target sample of the type of sample group corresponding to the device. When the process of step S497 is completed, the process proceeds to step S498.

[0359] In step S498, the content file generation unit 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S493. If it is determined in step S498 that all content has been processed, the process proceeds to step S499.

[0360] In step S499, the content file generating unit 314 generates a sample-to-group of a type of sample group according to the device.

[0361] In step S500, the communication unit 317 outputs the generated file. When the process of step S500 ends, the file generation process ends.

[0362] <File Creation Process Flow 9> An example of the file creation process flow when "Method 1-8-1" in the table of FIG. 16 is applied will be described with reference to the flowchart of FIG.

[0363] When the file generation process starts, in step S521, the media acquisition unit 311 generates haptic content synchronized with the AV media.

[0364] In step S522, the content file generation unit 314 determines and generates the types of sample groups of multiple devices to be added to the file.

[0365] In step S523, the encoding unit 313 encodes the haptic media.

[0366] In step S524, the content file generation unit 314 sets the sample time information and the access information.

[0367] In step S525, the content file generation unit 314 determines whether the sample corresponds to the type of sample group corresponding to the device. If it is determined that the sample corresponds to the type of sample group corresponding to the device, the process proceeds to step S526.

[0368] In step S526, the content file generating unit 314 sets sample-to-group information as the target sample of the sample group type corresponding to the device. When the process of step S526 ends, the process proceeds to step S528.

[0369] If it is determined in step S525 that the sample does not correspond to the type of sample group corresponding to the device, the process proceeds to step S527. In step S527, the content file generation unit 314 sets the sample-to-group information as a non-target sample of the type of sample group corresponding to the device. When the process of step S527 is completed, the process proceeds to step S528.

[0370] In step S528, the content file generating unit 314 generates sample-to-groups of the sample group type corresponding to the device in units of fragments.

[0371] In step S529, the content file generating unit 314 generates, for each fragment, sample-to-group of sample group type continuity information according to the device of any sample in the immediately preceding fragment.

[0372] In step S530, the content file generation unit 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S523. If it is determined in step S530 that all content has been processed, the process proceeds to step S531.

[0373] In step S531, the communication unit 317 outputs the generated file. When the process of step S531 ends, the file generation process ends.

[0374] <File Creation Process Flow 10> An example of the file creation process flow when "Method 1-8-2" in the table of FIG. 16 is applied will be described with reference to the flowchart of FIG.

[0375] When the file generation process starts, in step S551, the media acquisition unit 311 generates haptic content synchronized with the AV media.

[0376] In step S552, the content file generation unit 314 determines and generates the types of sample groups for each of a plurality of devices to be added to the file.

[0377] In step S553, the content file generation unit 314 sets a fragment unit that is equal to or greater than the maximum value (max) of the rise time of the reference device.

[0378] In step S554, the encoding unit 313 encodes the haptic media.

[0379] In step S555, the content file generation unit 314 sets the sample time information and the access information.

[0380] In step S556, the content file generation unit 314 determines whether the sample corresponds to the type of sample group corresponding to the device. If it is determined that the sample corresponds to the type of sample group corresponding to the device, the process proceeds to step S557.

[0381] In step S557, the content file generating unit 314 sets sample-to-group information as the target sample of the sample group type corresponding to the device. When the process of step S557 ends, the process proceeds to step S559.

[0382] If it is determined in step S556 that the sample does not correspond to the type of sample group corresponding to the device, the process proceeds to step S558. In step S558, the content file generation unit 314 sets the sample-to-group information as a non-target sample of the type of sample group corresponding to the device. When the process of step S558 is completed, the process proceeds to step S559.

[0383] In step S559, the content file generating unit 314 generates a sample-to-group of a type of sample group corresponding to the device in units of fragments.

[0384] In step S560, the content file generating unit 314 generates a stream type box of sample group type continuity information according to the device of any sample of the immediately preceding fragment in units of fragments.

[0385] In step S561, the content file generation unit 314 determines whether all content has been processed. If it is determined that unprocessed content exists, the process returns to step S554. If it is determined in step S561 that all content has been processed, the process proceeds to step S562.

[0386] In step S562, the communication unit 317 outputs the generated file. When the process of step S562 ends, the file generation process ends.

[0387] By performing each process as described above, the file generation device 300 can obtain the same effects as those described above in <3. Transmission of information related to stream type>.

[0388] 5. Second Embodiment Playback Device The present technology described above may be applied to any device. Fig. 63 is a block diagram showing an example of the configuration of a playback device, which is one aspect of an information processing device to which the present technology is applied. The playback device 600 shown in Fig. 63 is a playback device that performs playback processing of media data. For example, the playback device 600 acquires a content file generated by the file generation device 300, decodes a media bitstream stored in the content file, and plays back the media data.

[0389] Note that Fig. 63 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in playback device 600, there may be processing units that are not shown as blocks in Fig. 63, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 63.

[0390] As shown in FIG. 63, the playback device 600 has a communication unit 611 , a distribution control file analysis unit 612 , a control unit 613 , a decoding unit 614 , an output processing unit 615 , and an output unit 616 .

[0391] The communication unit 611 has a communication function for communicating with an external device of the playback device 600 (for example, a content file distribution server or the file generation device 300) and executes processing related to that communication. For example, the communication unit 611 may communicate with an external device of the playback device 600 and acquire a content file or a distribution control file supplied from the communication partner via that communication. The communication unit 611 may supply the acquired distribution control file to the distribution control file analysis unit 612. The communication unit 611 may supply the acquired content file to the control unit 613 or the decoding unit 614. Therefore, the communication unit 611 can also be said to be an acquisition unit that acquires a content file, a distribution control file, etc. (or an input unit that accepts input of a content file, a distribution control file, etc.).

[0392] The delivery control file analysis unit 612 performs processing related to the analysis of a delivery control file (e.g., an MPD). For example, the delivery control file analysis unit 612 may acquire a delivery control file supplied from the communication unit 611. The delivery control file analysis unit 612 may analyze control information stored in the delivery control file and control the communication unit 611 based on the analysis results. For example, the delivery control file analysis unit 612 may control processing related to the delivery of content files performed by the communication unit 611 (e.g., requests for content file delivery to a delivery server, acquisition of content files to be delivered, etc.). Furthermore, the delivery control file analysis unit 612 may control processing of the content files acquired by the communication unit 611 based on the analysis results.

[0393] The control unit 613 executes processing related to controlling the driving of the output unit 616. For example, the control unit 613 may acquire a content file supplied from the communication unit 611. The control unit 613 may control the driving of the output unit 616 based on management information stored in the content file.

[0394] The decoding unit 614 performs processing related to decoding of a media bitstream. For example, the decoding unit 614 may acquire a content file supplied from the communication unit 611. The decoding unit 614 may decode the media bitstream stored in the content file using an encoding method corresponding to the media to generate media data. The decoding unit 614 may supply the generated media data to the output processing unit 615.

[0395] The output processing unit 615 performs processing related to output control of media data. For example, the output processing unit 615 may acquire media data supplied from the decoding unit 614. The output processing unit 615 may render the media data to generate output data (hereinafter also referred to as output media data). For example, if the media data is image data, the output media data may be image information to be displayed on a display device such as a monitor. If the media data is audio data, the output media data may be an audio signal to be output from an audio output device such as a speaker. If the media data is haptic data, the output media data may be a vibration signal to vibrate a haptic device such as a vibration device. The output processing unit 615 may supply the generated output media data to the output unit 616. In other words, the output processing unit 615 may output the output media data from an output device (e.g., a monitor, a speaker, a vibration device, etc.) corresponding to the media. Therefore, the output processing unit 615 may also be referred to as a display control unit, an audio output control unit, a vibration output control unit, etc.

[0396] The output unit 616 has an output device and performs processing related to the output of media, etc. The output device of the output unit 616 may be any device that outputs media handled by the playback device 600. For example, it may be a display device (e.g., a monitor, etc.) that displays images, an audio output device (e.g., a speaker, etc.) that outputs audio, a vibration device that expresses tactile sensations by vibrating (e.g., a game console controller or glove, etc.), or other device. The output device may also be capable of outputting multiple types of media, such as a head-mounted display (HMD). For example, the output unit 616 may acquire output media data supplied from the output processing unit 615. The output unit 616 may output the output media data using an output device appropriate for the media. Therefore, the output unit 616 may also be referred to as a display unit, an audio output unit, a vibration output unit, etc. The output unit 616 may be driven under the control of the control unit 613.

[0397] <Application of the Present Technology> The playback device 600 configured as above may be the second information processing device described above, and one or more of the present technology described above in <3. Transmission of information related to stream type> may be applied.

[0398] For example, the playback device 600 (second information processing device) may include a control unit 613 that controls an output device (output unit 616) to output media data based on stream type information indicating the presence or absence of haptic media data in a sample, the information being stored as management information in a content file storing content made up of a plurality of samples played back consecutively in the time direction, and a decoding unit 614 that decodes the media bitstream stored in the content file to generate media data. In other words, the communication unit 611, the distribution control file analysis unit 612, the output processing unit 615, and the output unit 616 may be omitted.

[0399] The control unit 613 may also control the output device based on information included in the information on stream type that defines a group of samples for which no media data exists. The control unit 613 may also control the output device based on information included in the information on stream type that defines a group of samples for which media data exists. The control unit 613 may also control the output device based on information included in the information on stream type that defines a group of samples and indicates whether or not media data exists in the samples. The information on stream type may also include flag information that indicates whether or not media data exists in the samples. The information on stream type may also include a grouping type parameter that indicates the stream type of the sample group. The control unit 613 may also control the output device based on information included in the information on stream type that defines a sample and indicates whether or not media data exists in the samples.

[0400] Alternatively, the content file may manage the multiple samples that make up the content by dividing them into multiple fragments, and the control unit 613 may control the output device based on information about the stream type of each fragment, which is stored in the content file as management information for each fragment.

[0401] The control unit 613 may also control the output device based on information regarding a same-type consecutive period, which is a period in which consecutive samples of the same stream type are included in the information regarding the stream type. The information regarding the same-type consecutive period may also include information defining a group of samples included in the same-type consecutive period. The information regarding the same-type consecutive period may also further include information indicating whether the stream type will change in the sample following the same-type consecutive period. The information regarding the same-type consecutive period may also be stored in a box within a general-purpose user data box of the content file. The information regarding the same-type consecutive period may also include information indicating whether the stream type will change in the sample following the same-type consecutive period.

[0402] The control unit 613 may also control the output device based on device identification information for identifying an output device corresponding to the stream type, which is included in the information about the stream type. The device identification information may also be information about a group of samples in which media data exists. The device identification information may also be information about a group of samples indicating the presence of media data in the samples.

[0403] The control unit 613 may also control the output device based on information about a same-type consecutive period, which is a sequence of samples of the same stream type, included in the information about the stream type, and device identification information for identifying an output device corresponding to that stream type. The control unit 613 may also control the output device based on information about a same-type consecutive period, which includes information defining a group of samples included in the same-type consecutive period, and device identification information for that group. The information about the same-type consecutive period may further include information indicating whether the stream type will change in the sample following the same-type consecutive period. The control unit 613 may also control the output device based on information about the same-type consecutive period stored in a box in the general-purpose user data box of the content file and device identification information for the samples included in the same-type consecutive period. The information about the same-type consecutive period may also include information indicating whether the stream type will change in the sample following the same-type consecutive period.

[0404] The playback device 600 may further include a communication unit 611 (file acquisition unit) that acquires a distribution control file that controls distribution of content files, and a distribution control file analysis unit 612 that analyzes the distribution control file to determine whether the content file contains information related to the stream type as management information. If the playback device 600 determines, based on the analysis result of the distribution control file, that the content file contains information related to the stream type as management information, the control unit 613 may control the output device based on the information related to the stream type.

[0405] Furthermore, in the playback device 600, the communication unit 611 (file acquisition unit) may further acquire a content file. Then, the decoding unit 614 may decode the media bitstream stored in the acquired content file to generate media data.

[0406] Furthermore, the playback device 600 may further include an output processing unit 615 that uses the generated media data to generate output media data, which is data for output.

[0407] The playback device 600 may further include an output unit 616 that outputs output media data under the control of the control unit 613 .

[0408] With the above-described configuration, the playback device 600 can obtain the same effects as those described above in <3. Transmission of information related to stream type>.

[0409] <Playback Process Flow 1> An example of the flow of playback process executed by this playback device 600 will be described with reference to the flowchart in FIG.

[0410] When the playback process starts, in step S601, the communication unit 611 acquires a distribution control file.

[0411] In step S602, the delivery control file analysis unit 612 analyzes the delivery control file.

[0412] In step S603, the communication unit 611 acquires the content file.

[0413] In step S604, the control unit 613 analyzes the management information and controls the output device based on information about the stream type of the sample.

[0414] In step S605, the decoding unit 614 decodes the content bitstream.

[0415] In step S606, the output processing unit 615 generates output media data.

[0416] In step S607, the output unit 616 outputs the output media data. When the process of step S607 ends, the playback process ends.

[0417] <Application of the Present Technology> The above-described flow playback process may be the second information processing method (or second program) described above, and one or more of the present technology described above in <3. Transmission of information related to stream type> may be applied.

[0418] For example, this playback process (second information processing method (playback method) or second program) may include step S604 of controlling an output device to output media data based on stream type information indicating the presence or absence of haptic media data in a sample, the information being stored as management information in a content file storing content made up of a plurality of samples played back continuously in the time direction, and step S605 of decoding the media bitstream stored in the content file to generate media data. In other words, the processes of steps S601, S602, S603, S606, and S607 may be omitted.

[0419] In the processing of step S604, the control unit 613 may control the output device based on information included in the information on the stream type that defines a group of samples for which media data does not exist. In the processing of step S604, the control unit 613 may control the output device based on information included in the information on the stream type that defines a group of samples for which media data exists. In the processing of step S604, the control unit 613 may control the output device based on information included in the information on the stream type that defines a group of samples and indicates whether or not media data exists in the samples. In the processing of step S604, the information on the stream type may include flag information that indicates whether or not media data exists in the samples. In the processing of step S604, the information on the stream type may include a grouping type parameter that indicates the stream type of the sample group. In the processing of step S604, the control unit 613 may control the output device based on information included in the information on the stream type that defines a sample and indicates whether or not media data exists in the samples.

[0420] Alternatively, the content file may be configured to divide the multiple samples that make up the content into multiple fragments and manage them, and in the processing of step S604, the control unit 613 may control the output device based on information regarding the stream type for each fragment, which is stored in the content file as management information for each fragment.

[0421] In the processing of step S604, the control unit 613 may control the output device based on information regarding a same-type consecutive period, in which successive samples of the same stream type are included in the information regarding the stream type. In the processing of step S604, the information regarding the same-type consecutive period may include information defining a group of samples included in the same-type consecutive period. In the processing of step S604, the information regarding the same-type consecutive period may further include information indicating whether the stream type will change in the sample following the same-type consecutive period. In the processing of step S604, the information regarding the same-type consecutive period may be stored in a box within a general-purpose user data box of the content file. In the processing of step S604, the information regarding the same-type consecutive period may include information indicating whether the stream type will change in the sample following the same-type consecutive period.

[0422] In the process of step S604, the control unit 613 may control the output device based on device identification information for identifying the output device corresponding to the stream type, which is included in the information about the stream type. In the process of step S604, the device identification information may be information about a group of samples in which media data exists. In the process of step S604, the device identification information may be information about a group of samples indicating the presence of media data in the samples.

[0423] In step S604, the control unit 613 may control the output device based on information about a same-type consecutive period, which is a period of consecutive samples of the same stream type, included in the information about the stream type, and device identification information for identifying an output device corresponding to that stream type. In step S604, the control unit 613 may control the output device based on information about the same-type consecutive period, which includes information defining a group of samples included in the same-type consecutive period, and device identification information for that group. In step S604, the information about the same-type consecutive period may further include information indicating whether the stream type will change in the sample following the same-type consecutive period. In step S604, the control unit 613 may control the output device based on information about the same-type consecutive period stored in a box in the general-purpose user data box of the content file and device identification information for the samples included in the same-type consecutive period. In step S604, the information about the same-type consecutive period may include information indicating whether the stream type will change in the sample following the same-type consecutive period.

[0424] In step S601, the communication unit 611 (file acquisition unit) may acquire a distribution control file that controls distribution of a content file, and in step S602, the distribution control file analysis unit 612 may analyze the distribution control file to determine whether the content file contains information about the stream type as management information. If it is determined based on the analysis result of the distribution control file that the content file contains information about the stream type as management information, in step S604, the control unit 613 may control the output device based on the information about the stream type.

[0425] In addition, in the process of step S603, the communication unit 611 (file acquisition unit) may further acquire a content file. Then, in the process of step S605, the decoding unit 614 may decode the media bitstream stored in the acquired content file to generate media data.

[0426] In addition, in the process of step S606, the output processing unit 615 may use the generated media data to generate output media data, which is data for output.

[0427] In addition, in the process of step S607, the output unit 616 may be controlled by the control unit 613 to output the output media data.

[0428] By performing each process as described above, the playback device 600 can obtain the same effects as those described above in <3. Transmission of information related to stream type>.

[0429] <Reproduction process flow 2> In such reproduction process, for example, when "Method 1-1," "Method 1-2," "Method 1-3," or "Method 1-4" in the table of FIG. 15 is applied, each process may be executed in the flow shown in the flowchart of FIG. 65.

[0430] When the reproduction process starts, in step S621, the control unit 613 acquires haptic device start-up time information tx.

[0431] In step S622, the control unit 613 acquires information about the stream type of the sample group or sample entry of the haptic media, and analyzes the time series of whether or not playback is occurring.

[0432] In step S623, the control unit 613 and the like execute device media processing.

[0433] In step S624, the control unit 613 determines whether or not media is being processed. If it is determined that not all media have been played and that processing is being performed, the process returns to step S623. If it is determined in step S624 that all media have been played and that processing is not being performed, the playback process ends.

[0434] <Device Media Processing Flow 1> An example of the flow of device media processing executed in step S623 of FIG. 65 will be described with reference to the flowchart of FIG.

[0435] When device media processing starts, the control unit 613 determines in step S641 whether the driving state of the haptic device is active. In other words, it determines whether the output device for the target media is in a state where it can output the media data. If it is determined that the driving state of the haptic device is active (a state where it can output the media data), the process proceeds to step S642.

[0436] In step S642, the control unit 613 determines whether or not haptic media is being played during the haptic device startup time Tx. If it is determined that haptic media is not being played during the haptic device startup time Tx, the process proceeds to step S643.

[0437] In step S643, the control unit 613 sets the driving state of the haptic device to standby (a hibernating state or a shut-down state). That is, if the current driving state is active and it is determined that no haptic media is being played during the time Tx, the driving state of the haptic device is changed to standby (a state in which media data cannot be output). When the processing of step S643 ends, the processing proceeds to step S646.

[0438] Also, if it is determined in step S642 that haptic media is being played during the haptic device startup time Tx, the process of step S643 is omitted and the process proceeds to step S646. In other words, if the current drive state is active and it is determined that haptic media is being played during the time Tx, the current drive state (active) of the haptic device is maintained.

[0439] Also, if it is determined in step S641 that the driving state of the haptic device is not active (i.e., it is in standby), the process proceeds to step S644.

[0440] In step S644, the control unit 613 determines whether or not haptic media is being played after the haptic device startup time Tx. If it is determined that haptic media is being played after the haptic device startup time Tx, the process proceeds to step S645.

[0441] In step S645, the control unit 613 instructs the haptic device to start up. That is, if the current driving state is standby and it is determined that haptic media will be played after time Tx, the driving state of the haptic device is changed to active. When the processing of step S645 ends, the processing proceeds to step S646.

[0442] If it is determined in step S644 that haptic media is not being played after the haptic device startup time Tx, the process proceeds to step S646. In other words, if the current operating state is standby and it is determined that haptic media is not being played after the time Tx, the current operating state (standby) of the haptic device is maintained.

[0443] In step S646, the communication unit 611 acquires synchronized playback media, that is, the communication unit 611 acquires a content file and extracts the media bitstream of the haptic media stored in the content file.

[0444] In step S647, the decoding unit 614 decodes the synchronized playback media, that is, the decoding unit 614 decodes the extracted media bitstream to generate media data of the haptic media.

[0445] In step S648, the output processing unit 615 plays and renders the synchronized playback media. That is, the output processing unit 615 generates output media data of the haptic media by, for example, rendering the media data of the haptic media. The output processing unit 615 then supplies the output media data to the output unit 616, which then outputs the output media data.

[0446] When the process of step S648 ends, the device media process ends and the process returns to FIG.

[0447] <Reproduction Process Flow 3> For example, an example of the reproduction process flow when "Method 1-5" in the table of FIG. 15 is applied will be described with reference to the flowchart of FIG.

[0448] When the reproduction process starts, in step S661, the control unit 613 acquires haptic device start-up time information tx.

[0449] In step S662, the communication unit 611 acquires a fragment of the content file.

[0450] In step S663, the control unit 613 acquires information about the stream type of the sample group or sample entry of the haptic media of the acquired fragment, and analyzes the time series of whether playback is occurring.

[0451] In step S664, the control unit 613 determines whether or not there is time-series information for a time period equal to or greater than the haptic device rise time tx from the last sample of the fragment. If it is determined that there is no time-series information for a time period equal to or greater than tx, the process returns to step S662. If it is determined that there is time-series information for a time period equal to or greater than tx, the process proceeds to step S665.

[0452] In step S665, the control unit 613 etc. executes device media processing. The flow of the device media processing in this case is the same as the example explained with reference to the flowchart in Fig. 66, so explanation thereof will be omitted.

[0453] In step S666, the control unit 613 determines whether or not media is being processed. If it is determined that not all media have been played and that processing is in progress, the process returns to step S662. If it is determined in step S666 that all media have been played and that processing is not in progress, the playback process ends.

[0454] The flow of the playback process when "Method 1-6" in the table of FIG. 16 is applied is the same as the example described with reference to the flowchart of FIG.

[0455] <Reproduction Process Flow 4> For example, an example of the reproduction process flow when "Method 1-7" in the table of FIG. 16 is applied will be described with reference to the flowchart of FIG.

[0456] When the reproduction process starts, in step S681, the control unit 613 acquires information about the rise times tx of the plurality of haptic devices.

[0457] In step S682, the control unit 613 acquires information about the stream type of the sample group or sample entry of the haptic media, and analyzes the time series of whether or not playback is occurring.

[0458] In step S683, the control unit 613 and the like execute device media processing for each device.

[0459] In step S684, the control unit 613 determines whether or not media is being processed. If it is determined that not all media have been played and that processing is in progress, the process returns to step S683. If it is determined in step S684 that all media have been played and that processing is not in progress, the playback process ends.

[0460] <Device Media Processing Flow 2> An example of the flow of device media processing for each device executed in step S683 of FIG. 68 will be described with reference to the flowchart of FIG.

[0461] When device media processing is started, the control unit 613 determines in step S701 whether the operating state of the target device is active. That is, it determines whether the output device of the target media is in a state where it can output the media data. If it is determined that the operating state of the target device is active (a state where it can output the media data), the process proceeds to step S702.

[0462] In step S702, the control unit 613 determines whether or not the target device is playing back haptic media during the startup time Tx of the device. If it is determined that the target media is not playing back during the startup time Tx of the target device, the process proceeds to step S703.

[0463] In step S703, the control unit 613 sets the operating state of the target device to standby (hibernation or shutdown state). That is, if the current operating state is active and it is determined that no media is being played during the time Tx, the operating state of the target device is changed to standby (a state in which media data cannot be output). When the processing of step S703 ends, the processing proceeds to step S706.

[0464] If it is determined in step S702 that the media is being played during the startup time Tx of the target device, step S703 is skipped and the process proceeds to step S706. In other words, if the current operating state is active and it is determined that the target media is being played during the time Tx, the current operating state (active) of the target device is maintained.

[0465] Also, if it is determined in step S701 that the operating state of the target device is not active (i.e., it is in standby), the process proceeds to step S704.

[0466] In step S704, the control unit 613 determines whether the target media is being played after the startup time Tx of the target device. If it is determined that the target media is being played after the startup time Tx of the target device, the process proceeds to step S705.

[0467] In step S705, the control unit 613 instructs the target device to start up. That is, if the current operating state is standby and it is determined that the media will be played after the time Tx, the operating state of the target device is changed to active. When the processing of step S705 ends, the processing proceeds to step S706.

[0468] If it is determined in step S704 that the media is not being played after the startup time Tx of the target device, the process proceeds to step S706. In other words, if the current operating state is standby and it is determined that the media is not being played after the time Tx, the current operating state (standby) of the target device is maintained.

[0469] In step S706, the communication unit 611 acquires synchronized playback media. That is, the communication unit 611 acquires a content file and extracts the media bitstream of the target media stored in the content file.

[0470] In step S707, the decoding unit 614 decodes the synchronized playback media, that is, the decoding unit 614 decodes the extracted media bitstream to generate media data of the target media.

[0471] In step S708, the output processing unit 615 plays and renders the synchronized playback media. That is, the output processing unit 615 generates output media data for the target media by, for example, rendering the media data of the target media. The output processing unit 615 then supplies the output media data to the output unit 616, which then outputs the output media data.

[0472] When the process of step S708 ends, the device media process ends and the process returns to FIG.

[0473] <Reproduction Process Flow 5> For example, an example of the reproduction process flow when "Method 1-8" in the table of FIG. 16 is applied will be described with reference to the flowchart of FIG.

[0474] When the reproduction process starts, in step S721, the control unit 613 acquires information about the rise times tx of the plurality of haptic devices.

[0475] In step S722, the communication unit 611 acquires a fragment of the content file.

[0476] In step S723, the control unit 613 acquires information about the stream type of the sample group or sample entry corresponding to the device of the haptic media of the acquired fragment, and analyzes the time series of whether playback is occurring.

[0477] In step S724, the control unit 613 determines whether or not there is time-series information for a haptic device rise time tx or longer from the last sample of the fragment. If it is determined that there is no time-series information for a time tx or longer, the process returns to step S722. If it is determined that there is time-series information for a time tx or longer, the process proceeds to step S725.

[0478] In step S725, the control unit 613 etc. executes device media processing. The flow of the device media processing in this case is the same as the example explained with reference to the flowchart in Fig. 69, so explanation thereof will be omitted.

[0479] In step S726, the control unit 613 determines whether or not media is being processed. If it is determined that not all media have been played and that processing is being performed, the process returns to step S722. If it is determined in step S726 that all media have been played and that processing is not being performed, the playback process ends.

[0480] By performing each process as described above, the playback device 600 can obtain the same effects as those described above in <3. Transmission of information related to stream type>.

[0481] 6. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0482] FIG. 71 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0483] In a computer 900 shown in FIG. 71, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0484] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0485] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0486] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0487] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0488] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0489] Alternatively, this program may be pre-installed in the ROM 902 or the storage unit 913. The container file, which is generated by the file generation unit and stores media data and metadata, may be provided to the playback device via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. After being generated, the container file may be stored directly or via the transmission medium in a non-volatile storage medium such as a hard disk, optical disk, or flash memory, or in an electronic device incorporating a non-volatile storage medium.

[0490] <Applicable Targets of the Present Technology> The present technology can be applied to any encoding / decoding method.

[0491] Furthermore, the present technology can be applied to any configuration, for example, various electronic devices.

[0492] Furthermore, for example, the present technology can be implemented as a part of a device, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set (e.g., a video set) in which other functions are further added to a unit. In particular, each of the units in FIG. 34 , such as the preprocessing unit, encoding unit, and file generation unit, and each of the units in FIG. 38 , such as the parse processing unit, media acquisition unit, and decoding unit, may be configured to be independent modules or processors, or one or more modules or processors may be configured to perform the processing of multiple units.

[0493] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0494] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0495] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0496] For example, the present technology can be applied to systems and devices used to provide decorative content, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for transportation, such as monitoring traffic conditions and automatic driving control. Furthermore, for example, the present technology can also be applied to systems and devices used for security. Furthermore, for example, the present technology can also be applied to systems and devices used for automatic control of machines, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, the present technology can also be applied to systems and devices used to monitor natural conditions, such as volcanoes, forests, and oceans, and wildlife. Furthermore, for example, the present technology can also be applied to systems and devices used for sports.

[0497] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0498] Furthermore, various information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, making one piece of data available (linked) when processing the other piece of data. That is, data associated with each other may be combined into one piece of data or may be stored as separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only a portion of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0499] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0500] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0501] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0502] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0503] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0504] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0505] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0506] The present technology may also be configured as follows. (1) An information processing device comprising: an encoding unit that encodes media data related to haptics and generates a media bitstream; and a content file generation unit that generates a content file storing content composed of a plurality of samples played back continuously in a time direction, divides the media bitstream into sample units and stores them in the content file, generates information on a stream type indicating the presence or absence of the media data in the samples, and stores the information on the stream type in the content file as management information. (2) The information processing device described in (1), in which the content file generation unit generates, as the information on the stream type, information defining a group of the samples for which the media data does not exist. (3) The information processing device described in (1) or (2), in which the content file generation unit generates, as the information on the stream type, information defining a group of the samples for which the media data does exist. (4) The information processing device described in any of (1) to (3), in which the content file generation unit generates, as the information on the stream type, information defining the group of the samples for which the media data does exist. (5) The information processing device according to (4), wherein the content file generation unit generates information defining the group indicating the presence or absence of the media data in the sample by flag information. (6) The information processing device according to (4) or (5), wherein the content file generation unit generates information defining the group indicating the presence or absence of the media data in the sample by a grouping type parameter indicating the stream type of the group. (7) The information processing device according to any of (1) to (6), wherein the content file generation unit generates information defining the sample indicating the presence or absence of the media data in the sample as information related to the stream type.(8) The information processing device according to any one of (1) to (7), wherein in the content file, the plurality of samples constituting the content are divided into a plurality of fragments for management, and the content file generation unit generates information about the stream type for each fragment and stores the information in the content file as the management information for each fragment. (9) The information processing device according to any one of (1) to (8), wherein the content file generation unit generates, as the information about the stream type, information about a same type consecutive period in which samples having the same stream type continue. (10) The information processing device according to (9), wherein the content file generation unit generates, as the information about the same type consecutive period, information defining a group of the samples included in the same type consecutive period. (11) The information processing device according to (10), wherein the content file generation unit generates, as the information about the same type consecutive period, information indicating whether the stream type changes in the sample next to the same type consecutive period. (12) The information processing device according to any one of (9) to (11), wherein the content file generation unit defines a box for storing information about the consecutive periods of the same type and stores the box in a general-purpose user data box of the content file. (13) The information processing device according to (12), wherein the content file generation unit generates, as the information about the consecutive periods of the same type, information indicating whether the stream type will change in the sample next to the consecutive periods of the same type. (14) The information processing device according to any one of (1) to (13), wherein the content file generation unit generates, as the information about the stream type, device identification information for identifying an output device that outputs the media data corresponding to the stream type. (15) The information processing device according to (14), wherein the content file generation unit generates, as the information about the stream type, information defining a group of the samples in which the media data exists and generates the device identification information for the group.(16) The information processing device according to (14) or (15), wherein the content file generation unit generates, as the information about the stream type, information defining a group of the samples indicating the presence of the media data in the samples, and generates the device identification information for the group. (17) The information processing device according to any of (1) to (16), wherein the content file generation unit generates, as the information about the stream type, information about a same-type consecutive period during which the samples having the same stream type continue, and device identification information for identifying an output device that outputs the media data corresponding to the stream type. (18) The information processing device according to (17), wherein the content file generation unit generates, as the information about the same-type consecutive period, information defining a group of the samples included in the same-type consecutive period, and generates the device identification information for the group. (19) The information processing device according to (18), wherein the content file generation unit generates, as the information about the same-type consecutive period, information indicating whether the stream type changes in the sample next to the same-type consecutive period. (20) The information processing device according to any one of (17) to (19), wherein the content file generation unit defines a box for storing information relating to the same type consecutive period, stores the box in a general-purpose user data box of the content file, and generates the device identification information for the samples included in the same type consecutive period. (21) The information processing device according to (20), wherein the content file generation unit generates, as the information relating to the same type consecutive period, information indicating whether the stream type will change in the sample next to the same type consecutive period.(22) The information processing device according to any one of (1) to (21), further comprising a distribution control file generation unit that generates a distribution control file that controls distribution of the content file, wherein the distribution control file generation unit stores information indicating that information related to the stream type exists in the content file as the management information, in the generated distribution control file. (23) An information processing method including: encoding media data to generate a media bitstream, generating a content file that stores content made up of a plurality of samples that are played back continuously in a time direction, dividing the media bitstream into the sample units and storing them in the content file, generating information related to the stream type that indicates the presence or absence of the media data in the samples, and storing it in the content file as management information. (24) A program for causing a computer to execute a process including: encoding media data related to haptics to generate a media bitstream; generating a content file for storing content consisting of a plurality of samples played back continuously in the time direction; dividing the media bitstream into sample units and storing them in the content file; generating information regarding the stream type indicating the presence or absence of the media data in the samples; and storing the information in the content file as management information.

[0507] (31) An information processing device comprising: a control unit that controls an output device to output media data based on information on a stream type that indicates the presence or absence of predetermined media data in the samples, the information being stored as management information in a content file storing content made up of a plurality of samples played continuously in the time direction; and a decoding unit that decodes a media bitstream stored in the content file to generate the media data. (32) The information processing device according to (31), wherein the control unit controls the output device based on information that defines a group of the samples for which the media data does not exist, included in the information on the stream type. (33) The information processing device according to (31) or (32), wherein the control unit controls the output device based on information that defines a group of the samples for which the media data does exist, included in the information on the stream type. (34) The information processing device according to any of (31) to (33), wherein the control unit controls the output device based on information that defines a group of the samples, indicating the presence or absence of the media data in the samples, included in the information on the stream type. (35) The information processing device according to (34), wherein the information on the stream type includes flag information that indicates the presence or absence of the media data in the samples. (36) The information processing device according to (34) or (35), wherein the information on the stream type includes a grouping type parameter indicating the stream type of the group. (37) The information processing device according to any of (31) to (36), wherein the control unit controls the output device based on information defining the sample that indicates whether or not the media data is present in the sample and that is included in the information on the stream type.(38) The information processing device according to any one of (31) to (37), wherein in the content file, the plurality of samples constituting the content are managed by dividing them into a plurality of fragments, and the control unit controls the output device based on information on the stream type for each of the fragments, which is stored in the content file as the management information for each of the fragments. (39) The information processing device according to any one of (31) to (38), wherein the control unit controls the output device based on information on a same-type consecutive period in which samples having the same stream type continue, which information is included in the information on the stream type. (40) The information processing device according to (39), wherein the information on the same-type consecutive period includes information defining a group of the samples included in the same-type consecutive period. (41) The information processing device according to (40), wherein the information on the same-type consecutive period further includes information indicating whether the stream type changes in the sample next to the same-type consecutive period. (42) The information processing device according to any one of (39) to (41), wherein the information on the same-type consecutive period is stored in a box within a general-purpose user data box of the content file. (43) The information processing device according to (42), wherein the information related to the consecutive periods of the same type includes information indicating whether the stream type will change in the sample next to the consecutive periods of the same type. (44) The information processing device according to any of (31) to (43), wherein the control unit controls the output device based on device identification information for identifying the output device corresponding to the stream type, which is included in the information related to the stream type. (45) The information processing device according to (44), wherein the device identification information is information about a group of the samples in which the media data exists. (46) The information processing device according to (44) or (45), wherein the device identification information is information about a group of the samples indicating the presence of the media data in the samples.(47) The information processing device according to any one of (31) to (46), wherein the control unit controls the output device based on information on a same-type consecutive period in which the samples having the same stream type continue, the information being included in the information on the stream type, and device identification information for identifying the output device corresponding to the stream type. (48) The information processing device according to (47), wherein the control unit controls the output device based on information on the same-type consecutive period including information defining a group of the samples included in the same-type consecutive period, and the device identification information for the group. (49) The information processing device according to (48), wherein the information on the same-type consecutive period further includes information indicating whether the stream type changes in the sample next to the same-type consecutive period. (50) The information processing device according to any one of (47) to (49), wherein the control unit controls the output device based on information on the same-type consecutive period stored in a box in a general-purpose user data box of the content file, and the device identification information for the samples included in the same-type consecutive period. (51) The information processing device according to (50), wherein the information regarding the same type consecutive period includes information indicating whether the stream type will change in the sample next to the same type consecutive period. (52) The information processing device according to any of (31) to (51), further comprising: a file acquisition unit that acquires a distribution control file that controls distribution of the content file; and a distribution control file analysis unit that analyzes the distribution control file to determine whether the content file indicates that information regarding the stream type exists as the management information, wherein the control unit controls the output device based on the information regarding the stream type when it is determined based on the analysis result of the distribution control file that the content file indicates that information regarding the stream type exists as the management information.(53) The information processing device according to (52), wherein the file acquisition unit further acquires the content file, and the decoding unit decodes the media bitstream stored in the acquired content file to generate the media data. (54) The information processing device according to any of (31) to (53), further comprising an output processing unit that generates output media data that is data for output using the generated media data. (55) The information processing device according to (54), further comprising an output unit that outputs the output media data under the control of the control unit. (56) An information processing method comprising: controlling an output device that outputs the media data based on information on a stream type that indicates the presence or absence of media data related to haptics in the samples, the information being stored as management information in a content file that stores content made up of a plurality of samples that are played continuously in a time direction; and decoding the media bitstream stored in the content file to generate the media data. (57) A program for causing a computer to execute a process including: controlling an output device that outputs media data based on information regarding a stream type that indicates the presence or absence of media data related to haptics in a content file storing content consisting of a plurality of samples that are played back continuously in the time direction, the information being stored as management information in the content file; and decoding a media bitstream stored in the content file to generate the media data.

[0508] 300 File generation device, 311 Media acquisition unit, 312 Preprocessing unit, 313 Encoding unit, 314 Content file generation unit, 315 Distribution control file generation unit, 316 Storage unit, 317 Communication unit, 600 Playback device, 611 Communication unit, 612 Distribution control file analysis unit, 613 Control unit, 614 Decoding unit, 615 Output processing unit, 616 Output unit, 900 Computer

Claims

1. An information processing device comprising: an encoding unit that encodes media data related to haptics and generates a media bitstream; and a content file generation unit that generates a content file that stores content consisting of multiple samples that are played continuously in the time direction, divides the media bitstream into sample units and stores them in the content file, generates information regarding the stream type that indicates whether or not the media data is present in the sample, and stores it in the content file as management information.

2. The information processing device according to claim 1, wherein the content file generating unit generates, as the information relating to the stream type, information defining a group of the samples in which the media data does not exist.

3. The information processing device according to claim 1, wherein the content file generating unit generates, as the information relating to the stream type, information defining the group of the sample in which the media data exists.

4. The information processing device according to claim 1, wherein the content file generating unit generates, as the information relating to the stream type, information defining a group of the samples and indicating whether or not the media data is present in the samples.

5. The information processing device according to claim 4, wherein the content file generating section generates information defining the group that indicates the presence or absence of the media data in the sample by flag information.

6. The information processing device according to claim 4, wherein the content file generating unit generates information defining the group that indicates the presence or absence of the media data in the sample by a grouping type parameter that indicates the stream type of the group.

7. The information processing device according to claim 1, wherein the content file generating unit generates, as the information relating to the stream type, information defining the sample and indicating whether or not the media data is present in the sample.

8. An information processing device as described in claim 1, wherein in the content file, the multiple samples that make up the content are divided into multiple fragments and managed, and the content file generation unit generates information regarding the stream type for each fragment and stores it in the content file as the management information for each fragment.

9. The information processing device according to claim 1, wherein the content file generating section generates, as the information about the stream type, information about a period of consecutive identical samples in which the samples have the same stream type.

10. The information processing device according to claim 9, wherein the content file generating section generates, as information relating to the same type consecutive period, information defining a group of the samples included in the same type consecutive period.

11. The information processing device according to claim 10, wherein the content file generating unit generates, as the information relating to the same type consecutive period, information indicating whether the stream type will change in the sample following the same type consecutive period.

12. The information processing device according to claim 9, wherein the content file generation unit defines a box for storing information relating to the same type continuous period, and stores the box in a general-purpose user data box of the content file.

13. The information processing device according to claim 1, wherein the content file generating unit generates, as the information relating to the stream type, device identification information for identifying an output device that outputs the media data corresponding to the stream type.

14. The information processing device according to claim 13, wherein the content file generation unit generates, as the information relating to the stream type, information defining a group of the samples in which the media data exists, and generates the device identification information for the group.

15. The information processing device according to claim 13, wherein the content file generation unit generates, as information about the stream type, information defining a group of the sample indicating the presence of the media data in the sample, and generates the device identification information for the group.

16. The information processing device according to claim 1, wherein the content file generation unit generates, as information relating to the stream type, information relating to a period of consecutive identical samples of the same stream type, and device identification information for identifying an output device that outputs the media data corresponding to the stream type.

17. An information processing device as described in claim 1, further comprising a distribution control file generation unit that generates a distribution control file that controls the distribution of the content file, wherein the distribution control file generation unit stores information in the generated distribution control file indicating that information regarding the stream type exists in the content file as the management information.

18. An information processing method comprising: encoding media data relating to haptics to generate a media bitstream; generating a content file for storing content consisting of a plurality of samples played back continuously in the time direction; dividing the media bitstream into sample units and storing them in the content file; generating information regarding a stream type indicating whether or not the media data is present in the sample; and storing the information in the content file as management information.

19. An information processing device comprising: a control unit that controls an output device that outputs media data based on information regarding a stream type that indicates whether or not media data related to haptics is present in a content file that stores content consisting of multiple samples that are played continuously in the time direction, and a decoding unit that decodes a media bitstream stored in the content file and generates the media data.

20. An information processing method comprising: controlling an output device that outputs the media data based on information regarding a stream type that indicates whether or not haptic media data is present in a content file storing content consisting of multiple samples that are played back continuously in the time direction, and decoding a media bitstream stored in the content file to generate the media data.

Images