Information processing device, information processing method, playback processing device and playback processing method

Abstract

To provide an information processing apparatus, a reproduction processing apparatus, an information processing method, and a reproduction processing method that enable a client device to efficiently select a content configuration.SOLUTION: A preprocessing unit generates content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space. A file generation unit generates a file including data about the virtual space and the content configuration selection information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a playback processing device, and a playback processing method. [Background technology]

[0002] Currently, the mainstream of video distribution is 2D video, which is used for streaming movies and other content. 2D video is sometimes referred to as 2D (Dimension) content below. 360-degree video, which allows viewing in all directions, is also being distributed on online video distribution sites. Being able to view in all directions means that you can freely choose your viewing direction. 360-degree video is also called 3DoF (Degree of Freedom) video or 3DoF content. In both 2D and 3DoF content, video is basically encoded in 2D and distributed from a distribution server and displayed on the client.

[0003] There is also content called 3DoF+ content. 3DoF+ content allows you to look around in all directions like 3DoF content, but also allows you to move the viewpoint slightly. The range of movement of the viewpoint in 3DoF+ content is assumed to be the same as the range in which a user can move their head while sitting. 3DoF+ content achieves this by using one or more two-dimensionally encoded images.

[0004] Furthermore, the distribution of 6DoF video, known as 6DoF content, has been proposed as a video with even greater flexibility. 6DoF video allows you to look around in all directions in a three-dimensional space, and also allows you to walk around in the displayed three-dimensional space. Being able to walk around in a three-dimensional space means that you can freely select the viewpoint position. Hereinafter, three-dimensional space may also be referred to as 3D space.

[0005] 6DoF content is three-dimensional content that represents a three-dimensional space using one or more three-dimensional model data. Three-dimensional model data is also called 3D model data, and three-dimensional content is sometimes called 3D content.

[0006] One method of distributing 6DoF content is to construct a 3D space using multiple 3D model data and transmit it as multiple object streams. In this case, information on the structure of the 3D space called a scene description may be used. For example, there is the MPEG (Moving Picture Experts Group)-4 scene description. This scene description represents a scene as a graph with a hierarchical tree structure called a scene graph, and expresses this scene graph in binary format.

[0007] 6DoF content is video material that expresses 3D space using 3D model data at each time. The following three methods are examples of how 6DoF content can be expressed.

[0008] One method is an object-based representation method referred to in the present invention. The object-based representation method has a content configuration in which 6DoF content is created by arranging 3D model data for each 3D object, such as a person or object, that is an individual display object in a video, within the 3D space, thereby representing the entire 3D space. The object-based representation method has the advantage that, of the three methods, a client playing back 6DoF content simultaneously processes the largest amount of 3D model data. On the other hand, the object-based representation method allows for display of each individual 3D object, such as a person or object, with varying resolution. Therefore, of the three methods, this configuration method offers the highest degree of flexibility for client playback processing.

[0009] The other is a representation method that we refer to as space-based in this invention. In the space-based representation method, 6DoF content does not separate 3D model data for each 3D object, such as a person or an object, but has a content structure in which the entire target 3D space is represented as a single 3D model data. With the space-based representation method, the client processes a single 3D model data during playback, which is characterized by requiring the lowest processing power of the three methods. On the other hand, the resolution of the entire 6DoF content is fixed, and the client's degree of freedom in playback processing can be said to be extremely low.

[0010] The remaining method is a representation method that combines space-based and object-based representations. Hereinafter, this representation method will be referred to as a hybrid representation method. In the hybrid representation method, 6DoF content has a content configuration in which specific 3D objects are represented as individual 3D model data, and the 3D space that does not include those 3D objects is represented as a single 3D model data. In the hybrid representation method, multiple 3D model data are used in the client's playback process, but the number of 3D model data is fewer than the number used in the object-based representation method. In other words, the hybrid representation method requires a higher processing power from the client than the space-based representation method, but a lower processing power than the object-based representation method. Similarly, the client's degree of freedom in playback processing is higher than in the space-based representation method, but lower than in the object-based representation method.

[0011] As such, the content structure of 6DoF content differs for each expression method. Therefore, when a scene description is written to include several 6DoF contents with different expression methods, it is preferable for the client to select a content structure with as much freedom as possible, in order to further enhance the user's viewing experience. [Prior art documents] [Non-patent literature]

[0012] [Non-Patent Document 1] "ISO / IEC 14496-11:2015", Information technology. Coding of audio-visual objects. Part11:Sene description and application engine, 2015-11 Summary of the Invention [Problem to be solved by the invention]

[0013] However, when selecting an appropriate content configuration from a scene description, the client must perform various analyses before selecting the content configuration. This analysis includes, for example, an analysis of the entire scene description and an analysis of an Adaptation Set in an MPD (Media Presentation Description). Since such an analysis includes an analysis of parts that are not actually used, it can be said that the efficiency of content configuration selection by the client device is low.

[0014] Therefore, the present disclosure provides an information processing device, an information processing method, a playback processing device, and a playback processing method that enable a client device to efficiently select a content configuration. [Means for solving the problem]

[0015] According to the present disclosure, the preprocessing unit has a content configuration that represents a virtual space that is configured from one or more three-dimensional objects and their spatial arrangement information. Ruko Regarding the contents of each of the above 3D objects To determine whether or not recycling is possible 3D objects Generate Selection Information and a playback processing device that plays back the content includes capability identification information for determining whether decoding and rendering of the entire virtual space is executable, the capability identification information including information for determining the playback processing capability of the playback processing device used to reconstruct a three-dimensional space from a scene graph represented by a scene description, in the three-dimensional object selection information, and stores the three-dimensional object selection information for each content configuration in the scene description for a plurality of the content having different content configurations. The file generation unit generates the data of the virtual space and the 3D objects Generate a file containing the selection information. [Brief explanation of the drawings]

[0016] [Figure 1]FIG. 1 is a diagram showing the configuration of 6DoF content. [Figure 2] FIG. 1 is a system configuration diagram of an example of a distribution system. [Figure 3] FIG. 2 is a block diagram of a file generation device. [Figure 4] FIG. 3 is a diagram for explaining a method for storing content structure selection information according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of the syntax of an extended Switch node according to the first embodiment. [Figure 6] FIG. 2 is a block diagram of a client device. [Figure 7] 4 is a flowchart of a file generation process performed by the file generation device according to the first embodiment. [Figure 8] 4 is a flowchart of a playback process executed by a client device according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing an ISOBMFF file of a scene description in a modified example (1) of the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of the syntax of content configuration selection information stored in SampleEntry in the modified example (1) of the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of a group of content structure selection information. [Figure 12] 1 is a diagram showing the format of the Matroska Media Container. [Figure 13] FIG. 10 is a diagram for explaining a method for storing content structure selection information according to a modification (4) of the first embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of the syntax of a RequiredPerformance node. [Figure 15] FIG. 10 is a diagram illustrating an example of a 6DoFContentStruct Descriptor. [Figure 16] FIG. 10 is a diagram showing the semantics of CSC in a modification (5) of the first embodiment. [Figure 17]FIG. 10 is a diagram illustrating a method for storing content structure selection information according to the second embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of the syntax of an extended Switch node according to the second embodiment. [Figure 19] FIG. 13 is a diagram showing an example of the syntax of content configuration selection information stored in SampleEntry in the modified example (1) of the third embodiment. [Figure 20] FIG. 10 is a diagram showing the semantics of CSC in a modified example (2) of the third embodiment. [Figure 21] FIG. 10 is a diagram for explaining a method of using configuration information stored in an AdaptationSet of three-dimensional model data. [Figure 22] FIG. 2 is a hardware configuration diagram of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that in the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted. Furthermore, the scope of the disclosure of the present technology is not limited to the content of the embodiments, and also includes the content described in the following non-patent documents that were publicly known at the time of filing.

[0018] Non-patent document 1: (mentioned above) Non-patent document 2: "ISO / IEC 14496-12:2015", Information technology. Coding of audio-visual objects. Part 12: ISO base media file format, 2015-12 Non-patent document 3: "ISO / IEC 23009-1:2014", Information technology. Dynamic adaptive streaming over HTTP (DASH), Part 1: Media presentation description and segment formats, 2014-5

[0019] In other words, the contents of the above-mentioned non-patent documents are also incorporated herein by reference. In other words, the contents of the above-mentioned non-patent documents also serve as the basis for determining the support requirements. For example, even if the structure / terminology used in the Scene Description described in Non-Patent Document 1, the File Structure described in Non-Patent Document 2, and the terminology used in the MPEG-DASH standard described in Non-Patent Document 3 are not directly mentioned in the detailed description of the invention, they are considered to be within the scope of the disclosure of the present technology and to meet the support requirements of the claims. Similarly, for example, technical terms such as parsing, syntax, and semantics are considered to be within the scope of the disclosure of the present technology and to meet the support requirements of the claims, even if they are not directly mentioned in the detailed description of the invention. The present disclosure will be described in the following order.

[0020] 1. First embodiment 1.1 Modification of the First Embodiment (1) 1.2 Modification of the First Embodiment (2) 1.3 Modification of the First Embodiment (3) 1.4 Modification of the First Embodiment (4) 1.5 Modification of the First Embodiment (5) 2. Second embodiment 2.1 Modification of the second embodiment (1) 3. Third embodiment 3.1 Modification of the third embodiment (1) 3.2 Modification of the third embodiment (2) 4. Fourth Embodiment

[0021] [1. First embodiment] In the distribution of 6DoF content having content configurations based on object-based, space-based, and mixed representation methods, for example, a scene description file, an MPD file, and a 3D model data file are distributed in the configurations shown in Figure 1. Figure 1 shows the configuration of 6DoF content. In these current content configurations, clients may determine whether their own playback capabilities can be fully utilized based on the following three indicators:

[0022] The first index is an index for determining whether the scene description file and the 3D model data file can be decoded individually. The second index is an index for determining whether the scene description file and the 3D model data file can all be decoded. The third index is an index for determining whether rendering of the data after decoding is possible. Rendering refers to arranging and displaying in a 3D space. When a client device determines whether playback processing of each content configuration is possible based on these indexes, the following information can be used:

[0023] The first information is the @mimeType attribute and @codecs attribute stored in an AdaptationSet that represents a scene description among the AdaptationSets contained in the MPD file. This information allows the client device to determine whether it can decode the scene description. More specifically, the @mimeType attribute can determine whether the client device supports the file format of the scene description. The @codecs attribute can also determine whether the client device supports the codec that encoded the scene description. This allows the client device to determine in which format, such as MPEG-4 Scene Description or glTF (GL Transmission Format) 2.0, the scene description is generated, and thus to determine whether the client device can play back the scene description.

[0024] The second piece of information is the sceneProfileLevelIndication filed, which is stored when the scene description is represented in ISOBMFF (ISO Base Media File Format). This information allows a client device to determine whether it can render the data after decoding the scene description. This information includes information for determining the playback processing capabilities of the client device used to reconstruct a 3D space from the scene graph (hierarchical structure) represented by the scene description (a digital representation of the scene graph). For example, the sceneProfileLevelIndication filed includes the maximum number of points per scene in the case of a point cloud, and the maximum number of vertices, faces, and vertices per face in the case of a mesh. In other words, this information indicates how much playback processing capability is required for the entire scene.

[0025] The third piece of information is the number of external 3D model data files that make up the scene obtained from the scene graph represented by the scene description file. This information allows the client device to determine whether it can decode the scene description file and the 3D model data files. For example, the client device determines that playback is possible if the number of 3D model data decoders it possesses is greater than the number of external 3D model data files that make up the scene. In this case, the more decoders used, the higher the playback processing capability required of the client device.

[0026] The fourth information is the @mimeType attribute and @codecs attribute stored in the AdaptationSet representing each 3D model data among the AdaptationSets contained in the MPD file. The @mimeType attribute includes, for example, information about the file format in which the 3D model data is stored. The @codecs attribute includes information about the codec used to encode the 3D model data, as well as codec profile and level information. Based on this information, the client device determines whether it can decode each 3D model data. More specifically, the @mimeType attribute can determine whether the client device supports the file format of each 3D model data. The @codecs attribute can also determine whether the client device supports the codec used to encode each 3D model data.

[0027] Furthermore, if the @codecs attribute contains playback compatibility information for the 3D model data, the client device can determine whether it can render each piece of 3D model data. In this case, for example, the @codecs attribute contains the maximum number of points in the 3D model data for a point cloud, or the maximum number of vertices, faces, and vertices for a surface of the 3D model data for a mesh.

[0028] The fifth piece of information is the @bandwidth attribute stored in the Representation of each piece of 3D model data contained in the MPD file. This information allows the client device to determine whether it can decode each piece of 3D model data. For example, using this information, the client device can determine whether the bit rate allows playback of the 3D model data alone or the entire scene.

[0029] Of the above information, the first, fourth, and fifth information are used as the first index, the third, fourth, and fifth information are used as the second index, and the second and fourth information are used as the third index.

[0030] Here, content creators want to increase the value of their content by providing users with as much flexibility as possible in playback. On the other hand, it is preferable to have as many client devices capable of playback as possible. Therefore, content creators may consider preparing and distributing multiple content configurations as 6DoF content. Hereinafter, the content configurations for the object-based, space-based, and mixed-type representation methods will be referred to as the object-based content configuration, the space-based content configuration, and the mixed-type content configuration, respectively.

[0031] For example, if a content creator prepares a mixed-type content configuration and a spatial-based content configuration, a client device may select and play the mixed-type content configuration if its playback processing capability is high, but may select and play the spatial-based content configuration if its playback processing capability is low. In this case, the scene description is written to include two content configurations. In this case, a conventional client device would analyze all of the scene descriptions and the information described in the AdaptationSet of the MPD of the 3D model data that constitutes the scene in order to select a content configuration using the first to fifth pieces of information. This process is inefficient because it also analyzes parts of the content configuration that are not actually used in each scene.

[0032] However, in conventional 6DoF content distribution systems, client devices are not provided with information for determining which of the different content structures can be played back. Therefore, it is difficult for client devices to determine whether playback is possible without actually decoding and rendering the content. Therefore, we will describe a system that allows client devices to efficiently select a content structure.

[0033] [Configuration of the system according to the first embodiment] Fig. 2 is a system configuration diagram of an example of a distribution system. Distribution system 100 includes file generation device 1, which is an information processing device, client device 2, which is a playback processing device, and web server 3. File generation device 1, client device 2, and web server 3 are connected to network 4. File generation device 1, client device 2, and web server 3 can communicate with each other via network 4. Here, while Fig. 1 shows one of each device, distribution system 100 may include multiple file generation devices 1 and multiple client devices 2.

[0034] The file generation device 1 generates 6DoF content. The file generation device 1 uploads the generated 6DoF content to the Web server 3. Here, in this embodiment, a configuration in which the Web server 3 provides the 6DoF content to the client device 2 will be described, but the distribution system 100 can also have other configurations. For example, the file generation device 1 may be configured to include the functionality of the Web server 3, store the generated 6DoF content therein, and provide it to the client device 2.

[0035] The Web server 3 stores the 6DoF content uploaded from the file generation device 1. The Web server 3 then provides the specified 6DoF content in response to a request from the client device 2.

[0036] The client device 2 transmits a request to transmit 6DoF content to the Web server 3. Then, the client device 2 acquires the 6DoF content specified in the transmission request from the Web server 3. Then, the client device 2 decodes the 6DoF content to generate video, and displays the video on a display device such as a monitor.

[0037] Here, we will explain 6DoF content. 6DoF content represents a three-dimensional space using one or more three-dimensional objects. The three-dimensional objects are represented using a coordinate system within a bounding box normalized to the local coordinate system of the 6DoF content, and are compressed and encoded to become a bitstream. A scene description is used to place this bitstream in three-dimensional space.

[0038] There are multiple standards for this scene description. Basically, a scene displaying each 3D object at each time is represented by a hierarchical tree-structured graph called a scene graph, and this scene graph is represented in binary or text format. Here, a scene graph is spatial display control information, and is composed of information related to the display of 3D objects, with nodes defined as constituent units, and multiple nodes combined hierarchically. The nodes include nodes for coordinate conversion information that converts from one coordinate system to another, nodes for position information and size information of 3D objects, and nodes for access information to 3D objects and audio data.

[0039] In the following description, the 6DoF content is assumed to be composed of scene description data, which is spatial display control information, and media data of multiple 3D objects (for example, a representation of mesh data and texture data of the 3D objects). The 6DoF content may also include audio data. The media data of the 3D objects may also be in other formats, such as a point cloud. In this embodiment, the scene description file is assumed to conform to MPEG-4 Scene Description (ISO / IEC 14496-11).

[0040] MPEG-4 Scene Description data is a scene graph converted into a binary format called BIFS (Binary Format for Scenes). This scene graph can be converted into BIFS using a specific algorithm. Storing the scene description in ISOBMFF makes it possible to define a scene for each time period and to represent 3D objects whose position and size change.

[0041] [Configuration of the file generation device according to the first embodiment] Next, details of the file generation device 1 will be described. Fig. 3 is a block diagram of the file generation device. As shown in Fig. 3, the file generation device 1, which is an information processing device, has a generation processing unit 10 and a control unit 11. The control unit 11 executes processes related to the control of the generation processing unit 10. For example, the control unit 11 performs overall control of the operation timing of each unit of the generation processing unit 10. The generation processing unit 10 has a data input unit 101, a preprocessing unit 102, an encoding unit 103, a file generation unit 104, and a transmission unit 105.

[0042] The data input unit 101 accepts input of raw information for generating 3D objects, meta information, etc. The data input unit 101 outputs the acquired raw information to the pre-processing unit 102. The data input unit 101 accepts input of data. The data accepted by the data input unit 101 includes 3D objects and metadata such as placement information of the 3D objects. The data input unit 101 outputs the acquired data to the pre-processing unit 102.

[0043] The preprocessing unit 102 receives input of data including 3D objects and metadata such as placement information of the 3D objects from the data input unit 101. The preprocessing unit 102 then determines the bitstream configuration based on the acquired data, and generates a scene graph using the metadata of each 3D object and access information to the bitstream. The metadata includes control information such as the codec to be used for compression.

[0044] Furthermore, the preprocessing unit 102 generates, for each content structure, content structure selection information including one or more pieces of content structure information from the first to fifth pieces of information described above. This content structure selection information provides an index of the playback processing capability required to play back scenes of each content structure.

[0045] Then, the preprocessing unit 102 stores content structure selection information for each content structure in the scene description. This allows the client device 2 to select a content structure that can be played back using the content structure selection information. Storing the content structure selection information according to this embodiment will be described in detail below.

[0046] 4 is a diagram illustrating a method for storing content structure selection information according to the first embodiment. As shown in FIG. 4, the preprocessing unit 102 arranges child nodes for each content structure under a Switch node in the scene description. In FIG. 4, for example, content structure 301 is a content structure using a hybrid representation method, and content structure 302 is a content structure using a spatial-based representation method. Then, the preprocessing unit 102 extends the Switch node so that information used to determine whether or not to decode and render the entire scene for each content structure is stored as content structure selection information.

[0047] 5 is a diagram illustrating an example of the syntax of an extended Switch node in the first embodiment. For example, the preprocessing unit 102 indicates multiple content configurations in a Choice field in the Switch node. Furthermore, the preprocessing unit 102 newly adds a Points field, a VertivesParFace field, a Faces field, an Indices field, a Num3DmodeData field, a 3DmodeIDataMimeType field, a 3DmodeDataCodec field, and a Bitrate field, which indicate content configuration selection information for each content configuration. Then, the preprocessing unit 102 stores values ​​for each content configuration by storing values ​​in the newly added fields in the order of the content configurations indicated in the Choice field.

[0048] Points is the number of points in the point cloud. VertivesParFace is the number of vertices on the mesh faces. Faces is the number of faces in the mesh. Indices is the number of vertices in the mesh. Points, VertivesParFace, Faces, and Indices correspond to the second information. Num3DmodelData is the number of externally referenced 3D model data. This Num3DmodelData corresponds to the third information. 3DmodelDataMimeType is the MimeType of the externally referenced 3D model data. 3DmodelDataCodec is the codec of the externally referenced 3D model data. These Num3DmodelData and 3DmodelDataCodec correspond to the fourth information. Bitrate is the bitrate including the externally referenced 3D model data. This Bitrate corresponds to the fifth information.

[0049] The pre-processing unit 102 outputs the three-dimensional object and the generated scene graph to the encoding unit 103. The pre-processing unit 102 also outputs the metadata to the file generation unit 104.

[0050] The encoding unit 103 receives input of a 3D object and a scene graph from the preprocessing unit 102. Then, the encoding unit 103 encodes the 3D object to generate a bitstream. The encoding unit 103 also encodes the acquired scene graph to generate a scene description. Thereafter, the encoding unit 103 outputs the generated bitstream and scene description to the file generation unit 104.

[0051] The file generation unit 104 receives input of a bitstream and a scene description from the encoding unit 103. The file generation unit 104 also receives input of metadata from the preprocessing unit 102. The file generation unit 104 then files the acquired bitstream by storing it in an ISOBMFF file for each segment, and generates segment files of the bitstream. The file generation unit 104 also files scene description data by storing it in an ISOBMFF file for each segment, and generates segment files of the scene description.

[0052] Furthermore, the file generation unit 104 generates an MPD (Media Presentation Description) file based on the data acquired from the preprocessing unit 102. The MPD file stores meta information of the 6DoF content, such as the media type and information on video and audio segment files.

[0053] The transmission unit 105 acquires the bitstream, the ISOBMFF file of the scene description, and the MPD file from the file generation unit 104, and transmits them to the Web server 3 for uploading.

[0054] [Configuration of the client device according to the first embodiment] Fig. 6 is a block diagram of a client device. As shown in Fig. 6, the client device 2 has a playback processing unit 20 and a control unit 21. The control unit 21 controls the operation of each unit of the playback processing unit 20. For example, the control unit 21 comprehensively controls the timing of the operation of each unit of the playback processing unit 20. The playback processing unit 20 has a file acquisition unit 201, a measurement unit 202, a file processing unit 203, a decoding processing unit 204, a display control unit 205, a display information generation unit 206, and a display unit 207.

[0055] The file acquisition unit 201 acquires an MPD file corresponding to the 6DoF content to be played back from the Web server 3. Then, the file acquisition unit 201 acquires scene description information of the 6DoF content to be played back based on the MPD file.

[0056] The file acquisition unit 201 accesses the Web server 3 to acquire an ISOBMFF file that stores a scene description of the 6DoF content to be displayed. Then, the file acquisition unit 201 outputs the ISOBMFF file that stores the scene description to the file processing unit 203.

[0057] Furthermore, the file acquisition unit 201 acquires information about the bitstream selected by the file processing unit 203 from the file processing unit 203. Then, the file acquisition unit 201 accesses the Web server 3 to acquire segment files of the selected bitstream. Thereafter, the file acquisition unit 201 outputs the acquired segment files of the bitstream to the file processing unit 203.

[0058] The measurement unit 202 measures the transmission bandwidth of the transmission path between the client device 2 and the WEB server. Then, the measurement unit 202 outputs the measurement result of the transmission bandwidth to the file processing unit 203.

[0059] The file processing unit 203 receives input of an MPD file corresponding to the 6DoF content to be played back from the file acquisition unit 201. The file processing unit 203 then parses the acquired MPD file to acquire scene description information of the 6DoF content to be played back. The file processing unit 203 also recognizes multiple pieces of data to be used for adaptive distribution. For example, in the case of adaptive distribution that switches bit rates, the file processing unit 203 acquires information on bitstream segment files corresponding to each bit rate. In this case, the file processing unit 203 outputs scene description information of the 6DoF content to be played back to the file acquisition unit 201.

[0060] The file processing unit 203 receives an input of an ISOBMFF file storing a scene description from the file acquisition unit 201. The file processing unit 203 parses the acquired ISOBMFF file. Then, the file processing unit 203 acquires a Switch node of the scene description. After that, the file processing unit 203 acquires content structure selection information from the Switch node. Next, the file processing unit 203 selects a content structure to be used based on the acquired content structure selection information in accordance with the playback processing capability of the client device 2. After that, the file processing unit 203 acquires a child node in a scene graph corresponding to the selected content structure. Then, the file processing unit 203 acquires coordinate transformation information, placement information of three-dimensional objects, and access information for a scene with the selected content structure.

[0061] The file processing unit 203 also receives input of the measurement results of the transmission bandwidth from the measurement unit 202. Then, the file processing unit 203 selects a bitstream segment file to be played back based on the parsing results of the scene description and information indicating the transmission bandwidth acquired from the measurement unit 202. Then, the file processing unit 203 outputs information on the selected bitstream segment file to the file acquisition unit 201. At this time, adaptive distribution according to the bitrate is realized by changing the bitstream segment file to be selected depending on the transmission bandwidth.

[0062] Thereafter, the file processing unit 203 receives an input of the selected bitstream segment file from the file acquisition unit 201. Then, the file processing unit 203 extracts bitstream data from the acquired bitstream segment file and outputs the extracted data to the decoding processing unit 204.

[0063] The decoding processing unit 204 receives input of bitstream data from the file processing unit 203. Then, the decoding processing unit 204 performs decoding processing on the acquired bitstream data. Thereafter, the decoding processing unit 204 outputs the decoded bitstream data to the display information generation unit 206.

[0064] The display control unit 205 receives input of information on the operator's viewpoint position and line of sight direction from an input device (not shown). Then, the display control unit 205 outputs the acquired information on the viewpoint position and line of sight direction to the display information generation unit 206.

[0065] The display control unit 205 also receives input of information on what kind of three-dimensional objects are present from the file processing unit 203. The operator can also input designation information indicating the three-dimensional object of interest using an input device instead of the viewpoint position and line of sight information. The display control unit 205 acquires the designation information indicating the three-dimensional object of interest designated by the operator. Then, for example, in the case of tracking viewing of a three-dimensional object, the display control unit 205 outputs information on the viewpoint position and viewpoint direction to the display information generation unit 206 so as to track the three-dimensional object designated by the designation information over time. This makes it possible to display, for example, an image tracking the three-dimensional object designated by the operator. Furthermore, for example, in the case of displaying the position of a three-dimensional object, the display control unit 205 generates information identifying the designated three-dimensional object from the 6DoF content.

[0066] The display information generation unit 206 receives the scene description, the decoded bitstream data, and the acquired information on the viewpoint position and viewpoint direction, and generates display information. The display information generation unit 206 will be described in detail below.

[0067] The display information generation unit 206 receives input of bitstream data from the decoding processing unit 204. Then, based on the scene description, the display information generation unit 206 arranges the 3D objects, which are the acquired bitstream data, in 3D space. Furthermore, the display information generation unit 206 receives input of information on the operator's viewpoint position and line of sight direction from the display control unit 205. Then, the display information generation unit 206 renders the 3D objects arranged in 3D space in accordance with the viewpoint position and line of sight direction to generate an image for display. Thereafter, the display information generation unit 206 supplies the generated image for display to the display unit 207.

[0068] The display unit 207 has a display device such as a monitor. The display unit 207 receives an input of a display image generated by the display information generation unit 206. Then, the display unit 207 displays the acquired display image on the display device.

[0069] [File Generation Procedure According to the First Embodiment] Next, the flow of file generation processing by the file generation device 1 according to the first embodiment will be described in detail with reference to Fig. 7. Fig. 7 is a flowchart of file generation processing by the file generation device according to the first embodiment.

[0070] The preprocessing unit 102 generates content composition selection information (step S1). In this embodiment, the preprocessing unit 102 generates content composition selection information to be stored in a Switch node in a scene disgraph.

[0071] Then, the preprocessing unit 102 generates scene graph data in which the content composition selection information is stored in the Switch node (step S2). The preprocessing unit 102 outputs the scene graph data in which the content composition selection information is stored in the Switch node to the encoding unit 103.

[0072] Next, the encoding unit 103 encodes the data of the 3D object to generate a bitstream of the 3D object. The encoding unit 103 also encodes the acquired scene graph to generate a scene description (step S3).

[0073] Next, the file generation unit 104 stores the acquired bitstream in an ISOBMFF file for each segment to generate a bitstream segment file. Also, the file generation unit 104 stores scene description data in an ISOBMFF file for each segment to generate a scene description segment file (step S4).

[0074] The transmitting unit 105 outputs the segment file generated by the file generating unit 104 to the Web server 3 (step S5).

[0075] [Reproduction Processing Procedure According to the First Embodiment] Next, the flow of the playback process executed by the client device 2 according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart of the playback process executed by the client device according to the first embodiment.

[0076] The file acquisition unit 201 acquires an MPD file corresponding to the 6DoF content to be played back from the Web server 3 (step S11). The file acquisition unit 201 outputs the acquired MPD file to the file processing unit 203.

[0077] The file processing unit 203 parses the MPD file input from the file acquisition unit 201 and executes analysis processing (step S12). Based on the analysis result, the file processing unit 203 identifies the scene description of the 6DoF content to be played back.

[0078] Then, file acquisition unit 201 acquires the scene description identified by file processing unit 203. Next, file processing unit 203 acquires a Switch node by parsing the scene description acquired by file acquisition unit 201. Next, file processing unit 203 acquires content configuration selection information from the Switch node (step S13).

[0079] Then, the file processing unit 203 uses the acquired content structure selection information to select a content structure that is suited to the playback processing capability of the client device 2 (step S14).

[0080] Next, the file processing unit 203 parses the child node corresponding to the selected content configuration. After that, the file processing unit 203 acquires a bitstream segment file corresponding to the 6DoF content to be played back based on the parsing result (step S15). The decoding processing unit 204 performs a decoding process on the bitstream segment file. After that, the decoding processing unit 204 outputs the bitstream data to the display information generation unit 206.

[0081] The display control unit 205 outputs the input information on the viewpoint position and line of sight direction to the display information generation unit 206. The display information generation unit 206 performs viewing processing by using the information on the viewpoint position and line of sight acquired from the display control unit 205 to render a three-dimensional object and add position information to generate an image for display and display it on the display unit 207 (step S16).

[0082] As described above, the file generation device according to this embodiment stores content configuration selection information in the Switch node, which is the root node of the scene description, and provides the content configuration selection information to the client device. The client device acquires the Switch node, which is the root node of the scene description, to acquire the content configuration selection information, and selects a content configuration using the acquired content configuration selection information.

[0083] In a configuration in which content structure selection information is not stored, all scene descriptions are analyzed to collect information for selecting a content structure. In contrast, in the distribution system according to this embodiment, a client device can obtain content structure selection information by acquiring a Switch node, and can obtain information for selecting a content structure without analyzing other nodes in the scene description. This enables efficient content structure selection.

[0084] Content creators can prepare multiple content configurations to distribute 6DoF content compatible with client devices with different playback capabilities. Client devices can then efficiently select a content configuration that best suits their own playback processing capabilities.

[0085] [1.1 Modification of the first embodiment (1)] The file generation device 1 according to this modification differs from the first embodiment in that content configuration selection information is stored in the SampleEntry of the ISOBMFF file of the scene description.

[0086] The pre-processing unit 102 of the file generation device 1 according to this modification generates content structure selection information as meta. Then, the pre-processing unit 102 transmits the metadata including the content structure selection information to the file generation unit 104.

[0087] The file generation unit 104 receives input of the bitstream and scene description from the encoding unit 103. The file generation unit 104 also receives input of metadata including content structure selection information from the pre-processing unit .

[0088] As with the first embodiment, the file generating unit 104 stores the acquired bitstream in an ISOBMFF file for each segment to generate a file of the bitstream, thereby generating segment files of the bitstream.

[0089] On the other hand, when storing scene description data in an ISOBMFF file, the file generation unit 104 acquires content structure selection information from the metadata and stores the content structure selection information in the SampleEntry of the ISOBMFF file of the scene description.

[0090] For example, as shown in Fig. 9, the file generation unit 104 newly adds a 6DoFContentStructBox indicated by BOX303 to the SampleEntry of the ISOBMFF file of the scene description. Fig. 9 is a diagram showing the ISOBMFF file of the scene description in the modified example (1) of the first embodiment. Then, the file generation unit 104 stores content configuration selection information for each content configuration in the 6DoFContentStructBox.

[0091] For example, the file generation unit 104 stores content structure selection information expressed by the syntax shown in Fig. 10 in 6DoFContentStructBox. Fig. 10 is a diagram showing an example of the syntax of content structure selection information stored in SampleEntry in Modification (1) of the first embodiment. The file structure selection information in this case is the same information as the field newly added in the first embodiment shown in Fig. 5.

[0092] In this way, the file generation unit 104 stores the scene description data for each segment in an ISOBMFF file to generate a file of the scene description. The file generation unit 104 then outputs the ISOBMFF file of the scene description, including the file configuration information, to the transmission unit 105.

[0093] The file processing unit 203 of the client device 2 according to this modification receives an input of an ISOBMFF file of a scene description from the file acquisition unit 201. Then, the file processing unit 203 acquires an Initialization Segment of the ISOBMFF file of the scene description. Next, the file processing unit 203 acquires content structure selection information from the 6DoFContentStructBox in the acquired Initialization Segment. Then, the file processing unit 203 uses the acquired content structure selection information to select a content structure to use from the scene description.

[0094] As described above, the client device according to this modification can acquire content structure selection information before analyzing the scene description itself, and can select a content structure without analyzing the scene description itself. In this way, the client device according to this embodiment does not need to analyze the scene description itself to select a content structure. The configuration of modification (1) is effective when the playback processing capability required to play back the content structure does not change over time. In this case, there is no need to extend the scene description itself.

[0095] [1.2 Modification (2) of the first embodiment] The file generation device 1 of this modified example differs from the first embodiment in that it determines the value of each content configuration selection information, groups it, and indicates the group to which each content configuration belongs, thereby representing the content configuration selection information of each content configuration.

[0096] Fig. 11 is a diagram showing an example of groups of content structure selection information. In the table of Fig. 11, groups are set to which 01 to 03 are assigned as the identification number RequiredPerformanceID. For each group, the values ​​set for each piece of content structure selection information are shown.

[0097] The pre-processing unit 102 of the file generation device 1 according to this embodiment has information on the groups of the content configuration selection information shown in Fig. 11. Then, the pre-processing unit 102 stores information indicating which group each content configuration belongs to using a RequiredPerformanceID in the Switch node.

[0098] The file processing unit 203 of the client device 2 according to this embodiment analyzes the Switch node of the scene description and acquires the RequiredPerformanceID of the group to which each content configuration belongs. Then, the file processing unit 203 determines the playback processing capability required for playback of each content configuration from the content configuration selection information assigned to the group to which each content configuration belongs, and selects a content configuration.

[0099] As described above, the file generation device according to this embodiment notifies content configuration selection information for each content configuration using a group of content configuration selection information. This eliminates the need to generate detailed content configuration selection information for each content configuration, making it possible to reduce the load of the content configuration selection information generation process. This method is also applicable to the modified example (1) of the first embodiment.

[0100] [1.3 Modification (3) of the first embodiment] The file generation device 1 according to this modification differs from the first embodiment in that it uses Matroska Media Container (http: / / www.matroska.org / ) instead of ISOBMFF as the file format when transmitting three-dimensional model data.

[0101] Fig. 12 is a diagram showing the format of the Matroska Media Container. The file generation unit 104 of the file generation device 1 according to this modification stores a 6DoFContentStruct element having content structure selection information in a Track Entry element. At this time, the file generation unit 104 sets the Element Type to binary and stores SelectContentStructMetadata() shown in Fig. 10 as EBML (Extensible Binary Meta Language) data as binary data.

[0102] The file processing unit 203 of the client device 2 according to this modification acquires an Initialization Segment of a Matroska Media Container file including a scene description. Then, the file processing unit 203 acquires content configuration selection information from a 6DoFContentStruct element included in the Initialization Segment and selects a content configuration.

[0103] In this way, even when a container format different from that in variant example (1) is used, it is possible to provide content configuration selection information and allow the client device to select a container configuration. Also, in this variant example, container configuration selection information is generated for each container configuration, but this is not limiting. For example, it is also possible to store Element Type as an Integer to represent the RequiredPerformanceID, as in variant example (2).

[0104] [1.4 Modification (4) of the first embodiment] The file generation device 1 according to this embodiment differs from the first embodiment in that content structure selection information is stored for each content structure in the scene description. Fig. 13 is a diagram for explaining a method of storing content structure selection information according to a modification (4) of the first embodiment.

[0105] 4, file generation unit 104 of file generation device 1 according to this modification defines a new RequiredPerformance node 311 under the Group node of content structure 301, and defines a new RequiredPerformance node 312 under the Group node of content structure 302. Then, file generation unit 104 stores content structure selection information for content structure 301 in each RequiredPerformance node 311. Furthermore, file generation unit 104 stores content structure selection information for content structure 302 in each RequiredPerformance node 312.

[0106] FIG. 14 is a diagram showing an example of the syntax of a RequiredPerformance node. Specifically, the file generation unit 104 defines the RequiredPerformance node as a node having content configuration selection information for one content configuration using the syntax shown in FIG. 14. As shown in FIG. 14, the RequiredPerformance node registers information that serves as an index for determining whether the content configuration can be played. In this case, Points represents the number of points in the point cloud. VertivesParFace represents the number of vertices on a mesh face. Faces represents the number of faces in the mesh. Indices represents the number of vertices in the mesh. Num3DmodelData represents the number of externally referenced 3D model data. 3DmodelDataMimeType represents the MimeType of the externally referenced 3D model data. 3DmodelDataCodec represents the codec of the externally referenced 3D model data. Bitrate represents the bitrate including the externally referenced 3D model data.

[0107] The file processing unit 203 of the client device 2 according to this modification acquires a RequiredPerformance node for each content configuration. At this stage, the file processing unit 203 does not acquire other child nodes under this Group node. Next, the file processing unit 203 acquires content configuration selection information for each content configuration from each RequiredPerformance node. Then, the file processing unit 203 selects a content configuration. Thereafter, the file processing unit 203 acquires and parses the Group node and below of the selected content configuration.

[0108] As described above, the client device according to this modification can select a content configuration by acquiring and analyzing the RequiredPerformance node directly under the Group node of each content configuration. This reduces the amount of processing compared to analyzing the entire scene description. Furthermore, content configuration selection information can be provided to the client device without modifying existing nodes.

[0109] In this modification, a node that stores content configuration selection information is generated as a child node of the Group node, but it may be a child node of another node as long as it is the root node of the content configuration. Also, in this modification, container configuration selection information is generated for each container configuration, but this is not limiting. For example, the RequiredPerformance node may be configured to hold the RequiredPerformanceID in modification (2).

[0110] [1.5 Modification of the first embodiment (5)] The file generation device 1 according to this modification differs from the first embodiment in that content configuration selection information is stored in an AdaptationSet in an MPD file that indicates access information to a scene description.

[0111] The pre-processing unit 102 of the file generation device 1 according to this modification generates content structure selection information. Then, the pre-processing unit 102 transmits metadata including the content structure selection information to the file generation unit 104.

[0112] The file generation unit 104 receives input of the bitstream and scene description from the encoding unit 103. The file generation unit 104 also receives input of metadata including content structure selection information from the pre-processing unit .

[0113] The file generator 104 then stores the acquired bitstream in an ISOBMFF file for each segment to create a file, and generates a bitstream segment file. The file generator 104 also stores scene description data in an ISOBMFF file for each segment to create a file, and generates a scene description segment file.

[0114] Furthermore, the file generation unit 104 generates an MPD file based on the data acquired from the preprocessing unit 102. At this time, the file generation unit 104 acquires content structure selection information included in the metadata. Then, the file generation unit 104 defines the 6DoFContentStruct descriptor shown in FIG. 15 in AdaptationSet 320 of the scene description in the MPD file shown in FIG. 4. FIG. 15 is a diagram showing an example of a description of the 6DoFContentStruct Descriptor. Furthermore, the file generation unit 104 stores a CSC element for each content structure in the 6DoFContentStruct Descriptor in accordance with the acquired content structure selection information, thereby registering the content structure selection information in the attribute of the CSC element.

[0115] Fig. 16 is a diagram showing the semantics of a CSC in a modification (5) of the first embodiment. As shown in Fig. 16, a CSC describes the definition of an information element that indicates the capability of a content configuration. A CSC is composed of one or more elements up to a maximum of 255 elements. Here, @Use indicates attribute information of each element used, whether it is optional or mandatory.

[0116] If multiple CSC elements appear, they are registered in the order of the content composition described in the choice field of the Switch node of the scene description. In this case, CSC@points represents the number of points in the point cloud. CSC@VertivesParFace represents the number of vertices on a mesh face. CSC@Faces represents the number of faces on a mesh. CSC@Indices represents the number of vertices on a mesh. CSC@Num3DmodelData represents the number of externally referenced 3D model data. CSC@3DmodelDataMimeType represents the MimeType of the externally referenced 3D model data. CSC@3DmodelDataCodec represents the codec of the externally referenced 3D model data. CSC@Bitrate represents the bitrate including the externally referenced 3D model data.

[0117] The file processing unit 203 of the client device 2 according to this embodiment acquires content composition selection information by parsing the MPD file. That is, the file processing unit 203 can acquire the content composition selection information at the time of acquiring the MPD file, and can determine whether or not playable content exists in the scene description.

[0118] As described above, the client device according to this embodiment can select a content configuration that can be played back without acquiring a scene description. This enables efficient content configuration selection. However, in this case, since content configuration selection information is stored in the MPD file, it can be said to be effective when the playback processing capability required for playing back the content configuration does not change over time. Furthermore, while in this modification, container configuration selection information is generated for each container configuration, this is not limiting. For example, it is also possible to configure the attribute of the CSC element to store information representing the RequiredPerformanceID, as in modification (2).

[0119] [2. Second Embodiment] The file generation device 1 according to this embodiment differs from the first embodiment in that the file structure has a separate file for each scene description for each content structure, and content structure selection information is stored in the AdaptationSet of each scene description. The file generation device 1 according to this embodiment is also represented by the block diagram of FIG. 3. The client device 2 according to this embodiment is also represented by the block diagram of FIG. 6. In the following description, the operation of each unit similar to that of the first embodiment may be omitted.

[0120] 17 is a diagram illustrating a method for storing content structure selection information according to the second embodiment. As shown in FIG. 17, the pre-processing unit 102 of the file generation device 1 according to this modification generates scene descriptions 331 and 332 for each content structure. The scene description 331 is the scene description for content structure #1. The scene description 332 is the scene description for content structure #2. The pre-processing unit 102 then generates content structure selection information for each of content structures #1 and #2 and outputs it to the file generation unit 104.

[0121] The file generation unit 104 acquires the scene descriptions 331 and 332 from the encoding unit 103. The file generation unit 104 also acquires the content structure selection information for each of the content structures #1 and #2 from the pre-processing unit .

[0122] Then, the file generation unit 104 stores the scene descriptions 331 and 332 in the ISOBMFF file. Furthermore, the file generation unit 104 stores each piece of content configuration information in the AdaptationSet of each of the scene descriptions 331 and 332 in the MPD file using the 6DoFContentStruct Descriptor shown in Fig. 15.

[0123] The file processing unit 203 of the client device 2 according to this embodiment acquires an MPD file from the file acquisition unit 201. Then, the file processing unit 203 acquires content structure selection information included in each AdaptationSet of each scene description 331 and 332 of the MPD file. Then, the file processing unit 203 selects a content structure to use using the acquired content structure selection information. Thereafter, the file processing unit 203 acquires a scene description of the selected content structure from the Web server 3 via the file acquisition unit 201.

[0124] As described above, the client device according to this embodiment can select a content configuration when it acquires an MPD file. On the other hand, the methods according to the first embodiment and its modified examples acquire scene descriptions including content configurations that are not used, resulting in the acquisition of unused data. As such, the client device according to this embodiment does not need to acquire scene description data for unnecessary content configurations, enabling more efficient content configuration selection than the first embodiment and its modified examples. However, in this case, since content configuration selection information is stored in the MPD file, this method can be said to be effective when the playback processing capacity required for playback of the content configurations does not change over time.

[0125] [2.1 Modification (1) of the second embodiment] The file generation device 1 according to this modification differs from the second embodiment in that content configuration selection information is stored in the SampleEntry of the ISOBMFF file of the scene description.

[0126] The pre-processing unit 102 of the file generation device 1 according to this modification generates content structure selection information. Then, the pre-processing unit 102 transmits metadata including the content structure selection information to the file generation unit 104.

[0127] The file generation unit 104 receives input of the bitstream and scene description from the encoding unit 103. The file generation unit 104 also receives input of metadata including content configuration selection information from the pre-processing unit .

[0128] The file generator 104 then stores the acquired bitstream in an ISOBMFF file for each segment to create a file, and generates a bitstream segment file. The file generator 104 also stores scene description data in an ISOBMFF file for each segment to create a file, and generates a scene description segment file.

[0129] When storing this scene description in the ISOBMFF file, the file generation unit 104 stores the content structure selection information included in the metadata in the SampleEntry of the ISOBMFF file of the scene description. In this case, the file generation unit 104 can store the content structure selection information using the storage methods shown in Figures 9 and 10.

[0130] The file processing unit 203 of the client device 2 according to this embodiment acquires an ISOBMFF file of a scene description from the file acquisition unit 201. Next, the file processing unit 203 acquires an Initialization Segment of the ISOBMFF file of the acquired scene description. Then, the file processing unit 203 acquires content structure selection information from a 6DoFContentStructBox included in the Initialization Segment of the ISOBMFF file of the scene description. Then, the file processing unit 203 selects a content structure to use using the acquired content structure selection information. Thereafter, the file processing unit 203 acquires a scene description of the selected content structure from the Web server 3 via the file acquisition unit 201.

[0131] As described above, the client device according to this embodiment acquires content configuration selection information from the 6DoFContentStructBox included in the Initialization Segment of the ISOBMFF file of the scene description. After that, the client device only needs to acquire scene description data for the content configuration to be used, and does not need to acquire scene description data for other content configurations that are not used. This allows for efficient content configuration selection.

[0132] Here, instead of ISOBMFF, Matroska Media Container can be used as the file format when transmitting the 3D model data. When Matroska Media Container is used, the file generation unit 104 can store content configuration selection information in the same way as in the modification (3) of the first embodiment and provide it to the client device.

[0133] 3. Third Embodiment When delivering 6DoF content, it is possible to change the maximum processing power, bit rate, resolution, and other adaptations. Furthermore, the playback processing power required of the client changes depending on the bit rate adaptation for each 3D model data within a single content configuration. For example, if a stream with the largest variation in the number of mesh faces of 3D model data is selected for a certain content configuration, the client will be required to have high playback processing power, but if the minimum number of faces is selected, the playback processing power required of the client can be kept low.

[0134] In other words, content creators can increase the number of clients that can play content by providing content that has been adapted within the content structure. However, information that enables clients to determine whether the content that has been adapted within the content structure can be played back is not provided. Therefore, we will describe a system that allows clients to efficiently select a content structure when adaptation is performed within the content structure.

[0135] The file generation device 1 according to this embodiment differs from the first embodiment in that it generates and stores information indicating the minimum necessary playback processing capability as well as information indicating the maximum playback processing capability required for playback, and provides this to the client device 2. The file generation device 1 according to this embodiment is also represented by the block diagram of FIG. 3. The client device 2 according to this embodiment is also represented by the block diagram of FIG. 6. In the following explanation, explanations of the operations of the same parts as in the first embodiment may be omitted.

[0136] The pre-processing unit 102 of the file generation device 1 according to this embodiment generates content structure selection information for each content structure. Next, the pre-processing unit 102 expands the Switch node of the scene description as shown in Fig. 18 to store the content structure selection information for each content structure.

[0137] FIG. 18 is a diagram showing an example of the syntax of an extended Switch node in the second embodiment. MaxPoints is the maximum number of Points in a point cloud. MinPoints is the minimum number of Points in a point cloud. MaxVertivesParFace is the maximum number of vertices on a mesh face. MinVertivesParFace is the minimum number of vertices on a mesh face. MaxFaces is the maximum number of faces on a mesh. MinFaces is the minimum number of faces on a mesh. MaxIndices is the maximum number of vertices on a mesh. MinIndices is the minimum number of vertices on a mesh. MaxNum3DmodelData is the maximum number of externally referenced 3D model data. MinNum3DmodelData is the minimum number of externally referenced 3D model data. Max3DmodelDataCodec is the maximum value of the codec for externally referenced 3D model data. Min3DmodelDataCodec is the minimum value of the codec for externally referenced 3D model data. MaxBitrate is the maximum bitrate including externally referenced 3D model data. MinBitrate is the minimum bitrate including externally referenced 3D model data.

[0138] Here, the content configuration information indicating the maximum value is information indicating the playback processing capability capable of reliably playing back the content, while the content configuration information indicating the minimum value is information indicating the playback processing capability capable of playing back the content when adaptation is performed, i.e., information indicating the playback processing capability capable of playing back a portion of the content.

[0139] The preprocessing unit 102 outputs a scene graph including a Switch node represented by the syntax shown in FIG.

[0140] The file processing unit 203 of the client device 2 according to this embodiment acquires a Switch node from the acquired scene description. Then, the file processing unit 203 acquires system configuration selection information for each system configuration from the Switch node. Then, the file processing unit 203 selects a system configuration to use using the acquired system configuration selection information. In this case, even if the client device 2 does not satisfy the playback processing capability required by the content configuration selection information representing the maximum value in a certain system configuration, the file processing unit 203 can select that content configuration as long as the playback processing capability required by the content configuration selection information representing the minimum value is satisfied.

[0141] Thereafter, the file processing unit 203 parses the Group node and below of the selected system configuration in the scene description.

[0142] In the first embodiment, the maximum playback processing capability was required for the content configuration regardless of adaptation, and therefore a content configuration that can be played back if adaptation is performed is not selected. In contrast, the client device according to this embodiment can select and play back a content configuration that is difficult to play when the highest playback processing capability is required, as long as it becomes playable after adaptation.

[0143] This allows content creators to distribute 6DoF content that is compatible with client devices with different playback capabilities, taking into account adaptations within the content configuration. In other words, content creators can increase the number of client devices that can play a single content configuration. Furthermore, client devices can select a playable content configuration that suits their own playback processing capabilities, taking into account adaptations within the content configuration.

[0144] [3.1 Modification (1) of the third embodiment] The file generation device 1 according to this modification differs from the third embodiment in that content configuration selection information is stored in the SampleEntry of the ISOBMFF file of the scene description.

[0145] The file generation unit 104 receives an input of content structure selection information from the preprocessing unit 102. The file generation unit 104 newly adds a 6DoFContentStructBox to SampleEntry of the ISOBMFF file of the scene description. Then, the file generation unit 104 stores the content structure selection information expressed by the syntax shown in Fig. 19 in the 6DoFContentStructBox. Fig. 19 is a diagram showing an example of the syntax of the content structure selection information stored in SampleEntry in modification (1) of the third embodiment.

[0146] Thereafter, the file generation unit 104 outputs the ISOBMFF file of the scene description including the file configuration information to the transmission unit 105.

[0147] The file processing unit 203 of the client device 2 according to this modification receives an input of an ISOBMFF file of a scene description from the file acquisition unit 201. Then, it acquires an Initialization Segment of the ISOBMFF file of the scene description. Next, the file processing unit 203 acquires content configuration selection information from the 6DoFContentStructBox in the acquired Initialization Segment. Then, the file processing unit 203 selects a content configuration to use from the scene description using the acquired content configuration selection information. In this case, even if the client device 2 does not satisfy the playback processing capability required by the content configuration selection information representing the maximum value in a certain system configuration, the file processing unit 203 can select that content configuration as long as it satisfies the playback processing capability required by the content configuration selection information representing the minimum value.

[0148] As explained above, even with the method of storing content configuration selection information in the SampleEntry of the ISOBMFF file of the scene description, it is possible to have the client device select and play a content configuration that can be played back after adaptation has been performed. This enables content creators to distribute 6DoF content that takes into account adaptation within the content configuration and is compatible with playback on client devices with different playback capabilities.

[0149] [3.2 Modification (2) of the third embodiment] The file generation device 1 according to this modification differs from the second embodiment in that content configuration selection information is stored in an AdaptationSet in an MPD file that indicates access information to a scene description.

[0150] The pre-processing unit 102 of the file generation device 1 according to this modification generates content structure selection information. Then, the pre-processing unit 102 transmits metadata including the content structure selection information to the file generation unit 104.

[0151] The file generation unit 104 receives input of the bitstream and scene description from the encoding unit 103. The file generation unit 104 also receives input of metadata including content structure selection information from the pre-processing unit .

[0152] The file generator 104 then stores the acquired bitstream in an ISOBMFF file for each segment to create a file, and generates a bitstream segment file. The file generator 104 also stores scene description data in an ISOBMFF file for each segment to create a file, and generates a scene description segment file.

[0153] Furthermore, the file generation unit 104 generates an MPD file based on the data acquired from the preprocessing unit 102. At this time, the file generation unit 104 defines a 6DoFContentStruct descriptor in the AdaptationSet of the scene description in the MPD file as shown in FIG. 20. Then, the file generation unit 104 stores a CSC element for each content configuration in the 6DoFContentStruct Descriptor according to the content configuration selection information, and registers the content configuration selection information in the attribute of the CSC element. FIG. 20 is a diagram showing the semantics of a CSC in a modification (2) of the third embodiment. As shown in FIG. 20, the CSC describes a definition of an information element indicating the capability of a content configuration.

[0154] Thereafter, the file generation unit 104 outputs the ISOBMFF file of the scene description including the file configuration information to the transmission unit 105.

[0155] The file processing unit 203 of the client device 2 according to this embodiment acquires content configuration selection information by parsing an MPD file. Then, the file processing unit 203 selects a content configuration to use from the scene descriptions using the acquired content configuration selection information. In this case, even if the client device 2 does not satisfy the playback processing capability required by the content configuration selection information representing the maximum value in a certain system configuration, the file processing unit 203 can select that content configuration as long as it satisfies the playback processing capability required by the content configuration selection information representing the minimum value.

[0156] As described above, even with the method of storing content configuration selection information in the AdaptationSet of the scene description in the MPD file, it is possible to have the client device select and play a content configuration that can be played back after adaptation has been performed. This enables content creators to distribute 6DoF content that takes into account adaptations within the content configuration and is compatible with playback on client devices with different playback capabilities.

[0157] In the second embodiment and its modified examples (1) and (2), the container configuration selection information is generated for each container configuration, but this is not limiting, and it is also possible to express the maximum and minimum values ​​using, for example, RequiredPerformanceID.

[0158] [4. Fourth Embodiment] The file generation device 1 according to this embodiment differs from the first to third embodiments in that configuration information of each piece of 3D model data is stored in a Representation included in an AdaptationSet of the 3D model data in the MPD. The file generation device 1 according to this embodiment is also represented by the block diagram of Fig. 3. The client device 2 according to this embodiment is also represented by the block diagram of Fig. 6. In the following description, description of the operation of each unit similar to that of the first embodiment may be omitted.

[0159] The file generation unit 104 of the file generation device 1 according to this embodiment acquires content configuration selection information for each content configuration from the preprocessing unit 102. When generating an MPD file, the file generation unit 104 stores configuration information of the 3D model data for each Representation of the Adaptation Set of the 3D model data.

[0160] For example, the file generation unit 104 stores the number of points in the case of a point cloud in the attribute Representation@numPoint as configuration information of the 3D model data. Also, for a mesh, the file generation unit 104 stores the number of vertices of a face in the attribute Representation@vpf, the number of faces in the attribute Representation@numFase, and the number of vertices in the attribute Representation@numIndices.

[0161] In the first to third embodiments and their respective examples, storing this configuration information enables the file processing unit 203 of the client device 2 to select a content configuration that can be independently played back separately from the content configuration selection information. Here, an example has been described in which the configuration information of each 3D model data is stored in Representation, but the file generation unit 104 may store it in other elements such as AdaptationSet or Preselection.

[0162] Furthermore, in addition to storing this configuration information, the file generation unit 104 may store minimum and maximum values ​​in AdaptationSet of the 3D model data in the MPD. For example, in the case of a point cloud, the file generation unit 104 stores the maximum and minimum values ​​of the number of Points in the AdaptationSet@MaxNumPoint and AdaptationSet@MinNumPoint attributes. Furthermore, in the case of a mesh, the file generation unit 104 stores the maximum and minimum values ​​of the number of vertices of a face in the AdaptationSet@MaxVpf and AdaptationSet@MinVpf attributes, stores the maximum and minimum values ​​of the number of faces in the AdaptationSet@MaxNumFace and AdaptationSet@MinFace attributes, and stores the maximum and minimum values ​​of the number of vertices in the AdaptationSet@MaxNumIndices and AdaptationSet@MinIndices attributes.

[0163] The configuration according to this embodiment can be used to determine the playback processing capability required for playback when 3D model data is transmitted using an MPD file without using a scene description. For example, a case will be described in which an AdaptationSet 340 of 3D model data of a point cloud exists in an MPD file, as shown in Fig. 21. Fig. 21 is a diagram for explaining a method of using configuration information stored in an AdaptationSet of 3D model data.

[0164] 21, the file generation unit 104 stores information on the maximum and minimum values ​​of the number of points in AdaptationSet340 of the three-dimensional model data. The file generation unit 104 also stores the number of points in Representations 341 to 343 included in AdaptationSet340.

[0165] When parsing an MPD file, the file processing unit 203 of the client device 2 can determine whether or not there is a representation that can be played back at the level of the AdaptationSet 340. If there is a representation that can be played back, the file processing unit 203 can select the representation that can be played back from the representations 341 to 343 and play it back.

[0166] Alternatively, the file generation unit 104 may store the configuration information of the 3D model data in another location. For example, the file generation unit 104 may define a new 3DmModelDataMetadataBox in the SampleEntry of the ISOBMFF file of the 3D model data. The file generation unit 104 may then store the number of Points in the case of a point cloud of the 3D model data, and the number of vertices of faces, faces, and vertices in the case of a mesh, in the newly defined 3DmModelDataMetadataBox.

[0167] The file generation unit 104 can also use Matroska Media Container as the file format instead of ISOBMFF. In this case, the file generation unit 104 stores the configuration information in a new 3DmodelDataMetadata element in the Track Entry element. At this time, the file generation unit 104 sets the Element Type to binary, and stores, as EBML data, the number of points in the case of a PointCloud of D model data, and the number of vertices of a face, the number of faces, and the number of vertices in the case of a mesh, as binary data.

[0168] In the above embodiments and their respective modifications, the case where multiple content configurations exist has been described. However, the content configuration selection information can also be used to determine whether a single content configuration is a content configuration that can be played back. Furthermore, the content configuration selection information can be used even when the client device does not have the capability to play back 6DoF content of specific 3D model data. For example, a content creator may distribute a content configuration using one piece of 3D model data and a celestial sphere video. In this case, by adding information indicating that the content is a celestial sphere video and not 3D model data as the content configuration selection information, the client device can use the content configuration selection information to determine whether the content has a content configuration that can be played back.

[0169] [Hardware configuration] Fig. 22 is a hardware configuration diagram of a computer. The file generation device 1 and the client device 2 can be realized by a computer 90 shown in Fig. 22. In the computer 90, a processor 91, a memory 92, a network interface 93, a non-volatile storage 94, an input / output interface 95, and a display interface 86 are interconnected via a bus.

[0170] External devices such as an input device, an output device, a storage device, and a drive are connected to the input / output interface 95. Examples of the input device include a keyboard, a mouse, a microphone, a touch panel, and an input terminal. Examples of the output device include a speaker and an output terminal. Examples of the storage device include a hard disk, a RAM (Random Access Memory) disk, and the like. The drive drives removable media such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. Furthermore, a display 98, which is a display device, is connected to the display interface 96.

[0171] The network interface 93 is connected to an external network. The file generation device 1 and the client device 2 are connected to each other via the network interface 93. The file generation device 1 and the client device 2 are also connected to the Web server 3 via the network interface 93. The non-volatile storage 94 is an internal auxiliary storage device such as a hard disk or an SSD (Solid State Drive).

[0172] In the computer 90 configured as above, the processor 91 performs the above-described series of processes by, for example, loading a program stored in nonvolatile storage 94 into memory 92 via a bus and executing the program. The memory 92 also stores data necessary for the processor 91 to execute various processes as appropriate.

[0173] The program executed by the processor 91 can be applied by recording it on removable media such as package media, for example. In this case, the program can be installed in the nonvolatile storage 94 via the input / output interface 95 by inserting the removable media into a drive, which is an external device 97.

[0174] The program may also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting, in which case the program may be received by the network interface 93 and installed in the non-volatile storage 94.

[0175] Alternatively, this program may be pre-installed in the non-volatile storage 94 .

[0176] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0177] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0178] The present technology can also be configured as follows.

[0179] (1) a preprocessing unit that generates content configuration selection information for determining whether playback processing of one or more pieces of content is possible for each piece of content having a content configuration that represents a virtual space composed of one or more three-dimensional objects and their spatial arrangement information; a file generation unit that generates a file including the virtual space data and the content configuration selection information; An information processing device comprising: (2) The information processing device described in Appendix (1) in which the preprocessing unit includes feasibility identification information in the content configuration selection information for determining whether decoding and rendering of the entire virtual space can be performed in a playback processing device that plays the content. (3) The information processing device according to supplementary note (1) or (2), wherein the preprocessing unit stores the content configuration selection information in a scene description. (4) The information processing device according to supplementary note (3), wherein the preprocessing unit stores the content configuration selection information in a Switch node of the scene description. (5) The information processing device according to supplementary note (3), wherein the preprocessing unit stores the content structure selection information for each content structure in the scene description. (6) the preprocessing unit generates the content structure selection information as metadata; The file generation unit generates a content file that stores the content configuration selection information. The information processing device according to claim 2. (7) The information processing device according to appendix (6), wherein the file generation unit generates the content file as an ISOBMFF file and stores the content configuration selection information in a 6DoFContentStructBox in a SampleEntry of the content file. (8) The pre-processing unit has the content configuration selection information for each group whose content configuration is predetermined, and the content configuration selection information of the group to which each of the contents belongs is set as the content configuration selection information for each of the contents. (9) the preprocessing unit generates the content structure selection information as metadata; The file generation unit generates metadata that stores the content structure selection information. The information processing device according to claim 2. (10) The information processing device according to supplementary note (9), wherein the file generation unit generates the content file as an MPD file, and stores the content structure selection information in an AdaptationSet of the MPD file. (11) The preprocessing unit generates different scene descriptions for each of the plurality of contents having different content structures, The file generation unit stores the content configuration selection information in an AdaptationSet of an MPD file or a 6DoFContentStructBox in a SampleEntry of an ISOBMFF file for each of the scene descriptions. The information processing device according to claim 2. (12) The information processing device according to any one of appendices (1) to (11), wherein the preprocessing unit sets information indicating a playback processing capability capable of playing back the content as the content configuration information. (13) The information processing device according to appendix (12), wherein the preprocessing unit includes information indicating a playback processing capability capable of playing back a portion of the content in the content configuration selection information. (14) Generate content configuration selection information for determining whether playback processing of one or more pieces of content is possible for each piece of content having a content configuration that represents a virtual space composed of one or more three-dimensional objects and their spatial arrangement information; A file including the virtual space data and the content configuration selection information is generated. An information processing method that causes a computer to execute a process. (15) A file acquisition unit that acquires content configuration selection information for determining whether or not playback processing of one or more pieces of content having a content configuration representing a virtual space composed of one or more three-dimensional objects and their spatial arrangement information, and a file including data of the content; a file processing unit that acquires the content structure selection information from the file acquired by the file acquisition unit, determines whether playback processing of each of the contents is possible based on the acquired content structure selection information, and selects the content to be played; a playback unit that plays back the content selected by the file processing unit; A regeneration processing device comprising: (16) Acquire a file containing content configuration selection information for determining whether playback processing of one or more pieces of content having a content configuration representing a virtual space composed of one or more three-dimensional objects and their spatial arrangement information is possible; acquiring the content structure selection information from the acquired file, determining whether or not playback processing of each of the content items is possible based on the acquired content structure selection information, and selecting the content items to be played; Acquire data of the selected content; The selected content is played back using the acquired data. A reproduction processing method in which processing is executed by a computer. [Explanation of symbols]

[0180] 1 File generator 2. Client Device 3. Web Server 4 Network 10 Generation processing section 11 Control section 20 Reproduction processing section 21 Control section 101 Data Entry Section 102 Pre-processing section 103 Encoding section 104 File Generation Unit 105 Transmitter 201 File Acquisition Department 202 Measurement Department 203 File Processing Unit 204 Decryption processing unit 205 Display control unit 206 Display information generation section 207 Display section

Claims

1. generating three-dimensional object selection information for determining whether or not reproduction processing of each of the three-dimensional objects is possible for content having a content configuration that represents a virtual space composed of one or more three-dimensional objects and spatial arrangement information thereof; in the three-dimensional object selection information, capability identification information for determining whether decoding and rendering of the entire virtual space can be executed in a playback processing device that plays back the content, the capability identification information including information for determining a playback processing capability of the playback processing device used to reconstruct a three-dimensional space from a scene graph represented by a scene description; a pre-processing unit that stores the three-dimensional object selection information for each of the plurality of contents having different content configurations in the scene description; an information processing device comprising: a file generation unit that generates a file including the virtual space data and the three-dimensional object selection information;

2. the preprocessing unit generates the three-dimensional object selection information as metadata; The file generation unit generates a content file that stores the three-dimensional object selection information. The information processing device according to claim 1 .

3. 3. The information processing device according to claim 2, wherein the file generation unit generates the content file as an ISO Base Media File Format (ISOBMFF) file, and stores the three-dimensional object selection information in a 6DoFContentStructBox in a SampleEntry of the content file.

4. The information processing device according to claim 1, wherein the preprocessing unit has the three-dimensional object selection information for each group whose content configuration is predetermined, and the three-dimensional object selection information for the group to which each of the contents belongs is set as the three-dimensional object selection information for each of the contents.

5. the preprocessing unit generates the three-dimensional object selection information as metadata; The file generation unit generates a metadata file that stores the three-dimensional object selection information. The information processing device according to claim 1 .

6. The information processing device according to claim 5 , wherein the file generation unit generates the metadata file as an MPD (Media Presentation Description) file, and stores the three-dimensional object selection information in an AdaptationSet of the MPD file.

7. 2. The information processing apparatus according to claim 1, wherein the preprocessing unit sets information indicating a playback processing capability capable of playing back the content as the three-dimensional object selection information.

8. The information processing apparatus according to claim 7 , wherein the preprocessing unit includes information indicating a playback processing capability capable of playing back a portion of the content in the three-dimensional object selection information.

9. generating three-dimensional object selection information for determining whether or not reproduction processing of each of the three-dimensional objects is possible for content having a content configuration that represents a virtual space composed of one or more three-dimensional objects and spatial arrangement information thereof; in the three-dimensional object selection information, capability identification information for determining whether decoding and rendering of the entire virtual space can be executed in a playback processing device that plays back the content, the capability identification information including information for determining a playback processing capability of the playback processing device used to reconstruct a three-dimensional space from a scene graph represented by a scene description; For a plurality of the contents having different content configurations, the three-dimensional object selection information is stored for each content configuration in the scene description; A file including the virtual space data and the three-dimensional object selection information is generated. An information processing method that causes a computer to execute a process.

10. 3D object selection information for determining whether or not reproduction processing of each of the 3D objects is possible for content having a content configuration that represents a virtual space constituted by one or more 3D objects and spatial arrangement information thereof, in a playback processing device that plays back the content, capability identification information for determining whether decoding and rendering of the entire virtual space can be executed, the capability identification information including information for determining a playback processing capability of the playback processing device used to reconstruct a three-dimensional space from a scene graph represented by a scene description; a file acquisition unit that acquires three-dimensional object selection information stored for each content structure in the scene description for a plurality of the contents having different content structures, and a file including data of the content stored in the scene description; a file processing unit that acquires the three-dimensional object selection information from the file acquired by the file acquisition unit, determines whether or not reproduction processing of each of the three-dimensional objects is possible based on the acquired three-dimensional object selection information, and selects the three-dimensional object to be reproduced; a playback unit that plays back the three-dimensional object selected by the file processing unit; Equipped with Regeneration processing equipment.

11. 3D object selection information for determining whether or not reproduction processing of each of the 3D objects is possible for content having a content configuration that represents a virtual space constituted by one or more 3D objects and spatial arrangement information thereof, in a playback processing device that plays back the content, capability identification information for determining whether decoding and rendering of the entire virtual space can be executed, the capability identification information including information for determining a playback processing capability of the playback processing device used to reconstruct a three-dimensional space from a scene graph represented by a scene description; Acquire a file including three-dimensional object selection information stored for each content structure in the scene description for a plurality of the content pieces having different content structures; acquiring the three-dimensional object selection information from the acquired file, determining whether or not reproduction processing of each of the three-dimensional objects is possible based on the acquired three-dimensional object selection information, and selecting the three-dimensional objects to be reproduced; acquiring data of the selected three-dimensional object; Reconstructing the selected three-dimensional object using the acquired data. A reproduction processing method in which processing is executed by a computer.

Images