Information processing device and method

Abstract

The present disclosure pertains to information processing device and method which make it possible to suppress deterioration in subjective quality of a rendering image. With respect to a content file for storing data on a mesh having a hierarchical structure in which respective hierarchies have different bit rates, a control file which stores auxiliary information for adaptive control based on the topology of the mesh and which controls distribution of the content file is generated. Also, the data on the mesh is selected and acquired on the basis of the auxiliary information for adaptive control based on the topology of the mesh, the auxiliary information being stored in the control file which controls the distribution of the content file for storing the data on the mesh having a hierarchical structure in which the respective hierarchies have the different bit rates. The present disclosure can be applied to, for example, an information processing device or an information processing method.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that are capable of suppressing a reduction in the subjective quality of a rendered image.

[0002] Conventionally, V-DMC (Video-based Dynamic Mesh Coding) has been used as a method for encoding meshes, which are 3D data that represent the three-dimensional structure of an object using vertices and connections (see, for example, Non-Patent Document 1). In V-DMC, a base mesh, displacement vectors for vertex correction, attributes, and atlas information are encoded and transmitted as a bitstream. During decoding, each piece of information in the bitstream is decoded, the base mesh is subdivided, and displacement vectors are applied to the subdivided vertices to reconstruct a decoded mesh.

[0003] "WD 4.0 of V-DMC", ISO / IEC JTC 1 / SC 29 / WG 7, MDS23075, N00680, 2023-08-29

[0004] However, when distributing bitstreams generated using such V-DMC, there is a risk that the mesh topology may not be consistent, resulting in a so-called "flickering phenomenon" in the rendered image and a reduction in subjective quality.

[0005] The present disclosure has been made in light of such circumstances, and makes it possible to suppress a decrease in the subjective quality of a rendered image.

[0006] An information processing device according to one aspect of the present technology is an information processing device that includes a control file generation unit that stores auxiliary information for adaptive control based on the topology of a mesh in a content file that stores data of the mesh having a hierarchical structure with different bit rates for each layer, and generates a control file that controls the distribution of the content file.

[0007] An information processing method according to one aspect of the present technology is an information processing method that stores auxiliary information for adaptive control based on the topology of a mesh in a content file that stores data of a mesh having a hierarchical structure with different bit rates for each layer, and generates a control file that controls the distribution of the content file.

[0008] Another aspect of the information processing device of the present technology is an information processing device that includes a content acquisition unit that selects and acquires data of a mesh based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of a content file that stores data of a mesh having a hierarchical structure with different bit rates for each layer.

[0009] Another aspect of the information processing method of the present technology is an information processing method for selecting and acquiring data of a mesh based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of a content file storing data of a mesh having a hierarchical structure in which the bit rate differs for each layer.

[0010] In an information processing device and method according to one aspect of the present technology, a control file is generated that stores auxiliary information for adaptive control based on the topology of a mesh for a content file that stores data on the mesh having a hierarchical structure with different bit rates for each layer, and that controls the distribution of the content file.

[0011] In an information processing device and method according to another aspect of the present technology, data for a mesh is selected and acquired based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of a content file that stores data for a mesh having a hierarchical structure with different bit rates for each layer.

[0012] FIG. 1 is a diagram for explaining mesh. FIG. 1 is a diagram for explaining V-DMC. FIG. 1 is a diagram for explaining an example of a file structure. FIG. 2 is a diagram for explaining an example of the configuration of an MPD. FIG. 2 is a diagram for explaining an example of control using a scene description. FIG. 3 is a diagram for explaining a description of a scene description when MSFT_lod is used. FIG. 4 is a diagram for explaining an example of the main configuration of a scene description when MSFT_lod is used. FIG. 5 is a diagram for explaining an example of the main configuration of a scene description when MSFT_lod is used. FIG. 6 is a diagram for explaining an example of a method of adaptive delivery control. FIG. 7 is a diagram for explaining an example of topology match / mismatch. FIG. 8 is a diagram for explaining topology status information. FIG. 9 is a diagram for explaining an MPD description. FIG. 10 is a diagram for explaining delivery control using topology status information. FIG. 11 is a diagram for explaining adaptation behavior information. FIG. 12 is a diagram for explaining delivery control using adaptation behavior information. FIG. 13 is a diagram for explaining an example of using a stream for avoiding flicker. FIG. 14 is a diagram for explaining an example of delivery control using usage information. FIG. 15 is a block diagram for explaining an example of the main configuration of a file generation device. FIG. 16 is a block diagram for explaining an example of the main configuration of an encoding unit. 30 is a flowchart showing an example of the flow of a file generation process. 31 is a flowchart showing an example of the flow of a V-DMC encoding process. 32 is a block diagram showing an example of the main configuration of a playback device. 33 is a block diagram showing an example of the main configuration of a decoding unit. 34 is a flowchart showing an example of the flow of a playback process. 35 is a flowchart showing an example of the flow of a content acquisition process. 36 is a flowchart continuing from FIG. 30 showing an example of the flow of a content acquisition process. 37 is a flowchart showing an example of the flow of a V-DMC decoding process. 38 is a flowchart showing an example of the flow of a content acquisition process. 39 is a flowchart continuing from FIG. 30 showing an example of the flow of a content acquisition process. 39 is a flowchart showing an example of the flow of a content acquisition process. 39 is a flowchart continuing from FIG. 30 showing an example of the flow of a content acquisition process. 39 is a block diagram showing an example of the main configuration of a computer.

[0013] Modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described below. The description will be given in the following order: 1. Literature supporting technical content and technical terminology 2. Bandwidth-based adaptive distribution control of V-DMC data 3. Topology-based adaptive distribution control of V-DMC data 4. First embodiment (file generation device) 5. Second embodiment (playback device) 6. Supplementary notes

[0014] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0015] Unlicensed document 1: (above) Unlicensed document 2: "Information technology - Coding of audio-visual objects - Part 12: ISO base media file format, TECHNICAL CORRIGENDUM 1", ISO / IEC JTC 1 / SC 29 / WG 11, 14496_12_2022_7th_FDIS_MDS20295_WG03_N00233,7th edition, 2022-01 Non-licensed document 3: "Information technology - Coding of audio-visual objects - Part 15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format", ISO / IEC JISO / IEC JISO / IEC JTC 111 / SC 292929 / WG 11,ISO / IEC FDIS 14496-15:2014(E), w18241, 5th edition, 2014-01-13 Non-patentable document 4: "Information technology - Dynamic adaptive streaming over HTTP (DASH) - Part 1: Media presentation description and segment formats", ISO / IEC JTC 1 / SC 29 / WG 3, ISO 23009-1:2021(X), 5th edition, 2021-06-24 Non-patentable document 5: Lauri Ilola, Lukasz Kondrad, "[38.2] V-DMC base-mesh storage in ISOBMFF", ISO / IEC JTC1 / SC29 / WG3 m64933, October 2023, Hanover, Germany Special Specification:"Text of ISO / IEC FDIS 23090-10 Carriage of Visual Volumetric Video-based Coding Data", ISO / IEC JTC 1 / SC 29 / WG 03 N00241, Serial Number 20303, 2021-08-20 Functional Specifications 7:Saurabh Bhatia, Patrick Cozzi, Alexey Knyazev, Tony Parisi, "Khronos glTF2.0", https: / / github.com / KhronosGroup / glTF / tree / main / specification / 2.0, June 9, 2017 Note 8:"Information technology - Coded representation of immersive media - Part 14: Scene Description", ISO / IEC JTC 1 / SC 29 / WG 03, ISO / IEC DIS 23090-14:2021(E), N00485, MPEG 137th meeting, 2022-1 FUNDAMENTAL REPORT NINE:Saurabh Bhatia, Gary Hsu, Adam Gritt, John Copic, Marc Appelsmeier, Dag Frommhold, "MSFT_lod", "MSFT_lod". https: / / github.com / KhronosGroup / glTF / tree / main / extensions / 2.0 / Vendor / MSFT_lod 10:Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, Steve Sullivan, "High-Quality Streamable Free-Viewpoint Video."

[0016] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements. For example, even if syntax, terminology, etc. described in the above-mentioned non-patent documents are not directly defined in this disclosure, they are considered to be within the scope of this disclosure and meet the support requirements of the claims. Similarly, for example, technical terms such as parsing, syntax, and semantics are considered to be within the scope of this disclosure and meet the support requirements of the claims, even if they are not directly defined in this disclosure.

[0017] <2. Adaptive distribution control of V-DMC data based on bandwidth> <V-DMC> Conventionally, 3D data representing the three-dimensional structure of a three-dimensional structure (object with a three-dimensional shape) has been available as a mesh, which represents the three-dimensional shape of the object surface by forming polygons with vertices and connections (also called edges).

[0018] As shown in the upper left of Figure 1, in a mesh, vertices 11 and connections 12 connecting these vertices 11 form polygonal planes (polygons). These polygons (also called faces) represent the surface of a three-dimensional object, i.e., the three-dimensional shape of the object. A texture 13 can be applied to each face of this mesh.

[0019] Mesh data is composed of information such as that shown in the lower part of Figure 1. Vertex information 14, shown first from the left in the lower part of Figure 1, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 11 that constitutes the mesh. Connection information 15, shown second from the left in the lower part of Figure 1, is information indicating each connection (edge) 12 that constitutes the mesh. A texture image 16, shown third from the left in the lower part of Figure 1, is map information for the texture 13 that is applied to each face. A UV map 17, shown fourth from the left in the lower part of Figure 1, is information indicating the correspondence between the vertices 11 and the texture 13. The UV map 17 indicates the coordinates (UV coordinates) of each vertex 11 in the texture image 16.

[0020] As an example of such a mesh coding method, there is V-DMC (Video-based Dynamic Mesh Coding) as disclosed in Non-Patent Document 1.

[0021] In V-DMC, the mesh to be encoded (referred to in this specification as the original mesh) is represented as a base mesh that is less fine (i.e., coarser) than the original mesh, and displacement vectors of the division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are then encoded.

[0022] For example, assume that there is an original mesh as shown in the top row of Figure 2. The original mesh is a mesh composed of vertices and connections that represent the three-dimensional structure of an object, and is the target of encoding. For example, the original mesh is generated from a captured image of an object in real space (by camera capture). In Figure 2, black dots represent vertices, and lines connecting the black dots represent connections (edges). As described above, a mesh essentially forms polygons using vertices and edges, but for convenience of explanation, it is described here as a group of vertices connected linearly (in series).

[0023] By simplifying the original mesh, a coarse (low-resolution) mesh like the one shown in the second row from the top of Figure 2 is formed. This is called the base mesh. One simplification method is to thin out some of the vertices (decimate). In other words, the base mesh is a mesh with lower resolution than the original mesh, generated by thinning out vertices from the original mesh (i.e., simplifying the original mesh).

[0024] By subdividing each polygon of this base mesh, vertices and edges are added, as shown in the third row from the top of Figure 2. The degree of subdivision is arbitrary. That is, the number of vertices and edges added is arbitrary. For example, this subdivision can add vertices equal to the number of vertices thinned out from the original mesh. That is, subdivision can be used to maintain the same number of vertices as the original mesh. In this specification, these added vertices are also referred to as division points. This subdivision can also be repeated recursively. For example, in a technique called midpoint, the process of adding vertices to the midpoints of edges (subdivision) is repeated recursively. In other words, recursive subdivision increases the number of vertices and improves the resolution of the mesh. In this way, it is possible to perform subdivision up to any desired level of resolution (i.e., control the resolution of the subdivided mesh). In other words, the subdivided mesh can be layered according to its level of resolution. In other words, this can be considered a layering of the subdivision process and the vertices (division points) and edges obtained by the subdivision process.

[0025] However, the connections of the base mesh are updated when the vertices of the original mesh are thinned out. Therefore, the division points obtained by subdivision are formed on these updated connections (edges). As a result, the shape of the subdivided base mesh differs from the shape of the original mesh. More specifically, as shown in the bottom part of Figure 2, the positions of the division points (on the dotted line) differ from those of the original mesh. In addition, the positions of the vertices of the base mesh may differ from those of the original mesh.

[0026] In other words, by moving the positions of the vertices of the subdivided base mesh (the vertices or division points of the base mesh) closer to the vertex positions of the original mesh, the difference in shape between the subdivided base mesh and the original mesh can be reduced. In this specification, such movement of the vertices of the subdivided base mesh (the vertices or division points of the base mesh) is also referred to as displacement. Furthermore, the amount and direction of this displacement, expressed as a vector, is also referred to as a displacement vector. Ideally, by displacing each vertex of the subdivided base mesh, the shape of the subdivided base mesh can be made to match the shape of the original mesh. In other words, the original mesh can be expressed as a base mesh and a displacement vector.

[0027] In V-DMC, such base meshes and displacement vectors are coded instead of the original mesh (geometry). By coding the base meshes and displacement vectors in this way, it is possible to code with a reduced number of polygons (i.e., the number of vertices and edges) compared to coding the original mesh, which generally reduces the amount of code for the same quality. In other words, it is possible to improve coding efficiency.

[0028] During decoding, as described above, a mesh is restored (generated) by subdividing a base mesh and applying a displacement vector to each vertex of the subdivided base mesh to displace it. In this specification, this mesh is also referred to as a decoded mesh. Attributes are then applied to the decoded mesh, and the decoded mesh with the applied attributes is rendered. While the shape of the polygon (face) may be any polygonal shape, the following description will be given assuming that the polygon is triangular. Therefore, the polygon (face) will also be referred to as a triangle in the following description.

[0029] <Encoding and Decoding of V-DMC Data> In the case of V-DMC, mesh data consists of a base mesh, displacement vectors, attributes, and atlas information. This data group is also referred to as V-DMC data. The base mesh consists of information indicating vertices and connections, and is coded using an existing mesh coding method such as Draco. Note that the base mesh can be coded not only intra-coded but also inter-coded. When inter-coding, the base mesh is coded as a motion vector between frames.

[0030] Displacement vectors are arithmetically coded or packed into a two-dimensional plane and encoded as video data using a coding method for 2D video. For example, when encoded as video data, displacement vectors are converted into displacement coefficients using a predetermined method. The displacement coefficients are arranged as pixel values ​​in a two-dimensional area (also called a displacement map). This arrangement (mapping) of displacement coefficients is also called packing. Video (also called displacement video) using the displacement map as frame images is encoded using a coding method for 2D video. In other words, displacement coefficients are scalar values ​​corresponding to the displacement vectors. A displacement map is map information (also called image data) that stores displacement coefficients as pixel values. Displacement video is video image data using the displacement map as frame images.

[0031] An attribute is non-geometry information applied to a mesh (geometry), which is 3D data. For example, an attribute may include a texture applied to a face of the mesh (geometry). The attribute (e.g., texture) is divided into multiple subregions, each of which is projected in a predetermined projection direction, and the projected images (patches) are arranged in a two-dimensional region (also called an attribute map). In other words, attribute patches are packed into the attribute map. A video (also called attribute video) using the attribute map as frame images is encoded using a 2D video encoding method. In other words, the attribute map is map information (also called image data) that stores the patches (projected textures) as pixel values. Attribute video is video data using the attribute map as frame images.

[0032] Atlas information is information used when reconstructing a mesh. For example, atlas information may include correspondence between the base mesh and a displacement map or attribute map (such as a UV map), quantized values ​​of displacement vectors, etc. This atlas information is encoded using a predetermined encoding method.

[0033] The coded data (bitstream) of each data is decoded by a decoding method corresponding to the coding method. In other words, by decoding the coded data (bitstream), various information such as base meshes, displacement vectors, attributes, and atlas information is restored (generated).

[0034] <Use of ISOBMFF> Incidentally, as described in Non-Patent Documents 2 and 3, there is ISOBMFF (International Organization for Standardization Base Media File Format), which is a file container specification of MPEG-4 (Moving Picture Experts Group-4), an international standard technology for video compression.

[0035] In recent years, with the aim of improving the efficiency of playback processing from local storage and network distribution of bitstreams encoded using V-DMC, there has been anticipation for the establishment of a method of storing V-DMC bitstreams in this ISOBMFF and distributing them using MPEG-DASH (Moving Picture Experts Group Dynamic Adaptive Streaming over HTTP (Hypertext Transfer Protocol)), which is described in Non-Patent Document 4.

[0036] As an example of this technology, MPEG has proposed a standardization by extending ISO / IEC 23090-10, the V-PCC distribution technology standard described in Non-Patent Document 6, as described in Non-Patent Document 5. This standard adopts a multi-track structure in which the V-DMC bitstream is divided into tracks for each sub-bitstream. In this standard, a base mesh track is newly defined to store the base mesh sub-bitstream, and other sub-bitstreams are stored in the tracks defined in ISO / IEC 23090-10. That is, as shown in FIG. 3, a content file is composed of an atlas track that stores the atlas information sub-bitstream, a base mesh track that stores the base-mesh sub-bitstream, a geometry track that stores the displacement video sub-bitstream (using Video codec), and an attribute track that stores the attribute sub-bitstream.

[0037] Furthermore, the MPEG-DASH signaling structure of V-DMC ISOBMFF files may use the signaling structure of V-PCC ISOBMFF files described in Non-Patent Document 6. For example, in the case of V-PCC, each track of the ISOBMFF structure is represented as an individual adaptation set in the DASH manifest (MPD (Media Presentation Description)), and these are linked by a preselection element or a preselection descriptor. Similarly, in the case of V-DMC, each track may be signaled by an individual adaptation set, and linked by a preselection element or a preselection descriptor, as shown in FIG. 4.

[0038] <Scene Description> Furthermore, Non-Patent Document 8 describes an extension (MPEG-I Scene Description) that enables reference to an MPD adaptation set from a scene description (Scene Description file) written in glTF2.0 as described in Non-Patent Document 7, in order to perform adaptive streaming of a 3D scene including time-series 3D object data. For example, as shown in Fig. 5, in glTF2.0, a 3D object is represented by a JSON object called "mesh." Then, from there, the 3D object data shown in "buffer" is referenced via "accessor" and "bufferView."

[0039] In the extended MPEG-I Scene Description, this "buffer" references an MPEG-defined extended json object called "MPEG_media," which contains the AdaptationSet@id of the MPD to be referenced, thereby referencing the stream described in the MPD as 3D object data.

[0040] When referencing a multi-track V-PCC object as a 3D object, the AdaptationSet@id for the atlas information of that V-PCC object is written and referenced in the "MPEG_media" referenced by the "mesh" json object that references this 3D object.

[0041] Similarly, in V-DMC, it is assumed that the AdaptationSet@id for the atlas information of a V-DMC object will be referenced from the corresponding "mesh" json object via "MPEG_media".

[0042] <MTFS_lod> Non-patent document 9 describes, as one of the functions of scene description, a function that provides multiple variations of model data with different levels of detail (also called LOD (Level Of Detail)) for a 3D object to be placed in a scene, and switches to and displays model data with an appropriate level of detail depending on the distance from the viewpoint.

[0043] In glTF 2.0, this is achieved using an LOD extension called "MSFT_lod." For example, as shown in Figure 6, an extended JSON object called "MSFT_lod" is defined within a "node" that references a "mesh." This means that the defined "node" is the highest LOD model. The index array of "nodes" that are models of subsequent LODs is indicated by an extended JSON object called "ids" within "MSFT_lod." This index array is arranged in descending order of LOD. An extended JSON object called "MSFT_screencoverage" indicates the criteria by which the client uses each LOD "node" depending on the viewing situation, expressed as the ratio of the area where the corresponding object is displayed to the entire display screen. Figure 7 shows an example of the MSFT_lod structure in glTF 2.0, as expressed above.

[0044] By using the MPEG-I scene description described in Non-Patent Document 8 and MSFT_lod described in Non-Patent Document 9, adaptation streaming distribution can be realized in which the LOD of a V-DMC object is appropriately switched depending on the viewpoint position. For example, as shown in FIG. 8, multiple LOD variations of a V-DMC object are prepared, and each is described in the MPD. Furthermore, MSFT_lod is used in the MPEG-I scene description, and the AdaptationSet@id for the V-DMC atlas information corresponding to each LOD is written and referenced in the "MPEG_media" referenced by the "node" corresponding to each LOD.

[0045] In the MPD, the same content can be represented by representations in two or more different adaptation sets, and these contents may be time-aligned to allow seamless switching between representations in different adaptation sets.

[0046] In such a case, as shown in Fig. 9, by providing a supplemental property descriptor (AS Switching) in the adaptation set with @schmeIdURI="urn:mpeg:dash:adaptation-set-switching:2016" and @value="@id of the AdaptationSet that allows seamless switching from that adaptation set," it is possible to notify the client that seamless switching between representations in those adaptation sets is possible. Storing this supplemental property descriptor in the adaptation set for each V-DMC LOD enables bitrate adaptation across LODs.

[0047] <Mesh Tracking> The dynamic mesh stream generated by camera capture is a non-registered mesh stream with no consistent topology between frames. A consistent topology means that each frame has a common mesh structure. In this case, the number of faces is the same between frames and there is a corresponding relationship. The same is true for the number of vertices, which are also the same between frames and there is a corresponding relationship.

[0048] There is a topology alignment technology called Mesh Tracking. Applying this technology to align the topology enables mesh inter-compression. It also eliminates distortion of the fine shapes of objects caused by differences in topology between frames during playback.

[0049] However, mesh tracking processing may not be possible due to differences in the shape of objects between frames. This can result in frame intervals where the topology between frames is not consistent. In frame intervals where the topology is not consistent, if an object is displayed with low resolution relative to the viewing distance, which makes the difference in topology between frames visible, there is a risk of a "flickering phenomenon" occurring, in which the position of the object's edge fluctuates at high frequencies. Therefore, for content intended for local playback, this degradation in image quality is avoided by using a model with sufficient resolution relative to the viewing distance.

[0050] <For streaming> When streaming, adaptive streaming (i.e., adaptive streaming control of V-DMC data based on bandwidth) is performed to prevent playback interruptions when bandwidth fluctuates. If this is done as in the past, by selecting a combination that fits within the bandwidth from all representations in the adaptation set corresponding to the LOD appropriate for the current viewing situation and lower LODs, depending on the bandwidth, a mesh with a resolution that allows the topology to be visible may be selected and displayed. If the topology is not consistent at this time, there is a risk of image quality degradation specific to dynamic meshes, such as high-frequency flickering of object edges. This is more noticeable to viewers than traditional static degradation such as a decrease in resolution, and could lead to a more significant deterioration in the viewing experience.

[0051] This issue could not be addressed because the client could not determine whether such a flickering phenomenon would occur based on existing information alone. If the client could determine whether the mesh resolution is sufficient for the viewing distance and whether the mesh topology is consistent, it would be able to determine whether the flickering phenomenon would occur and take appropriate action. The status of whether the mesh resolution is sufficient for the viewing distance can be provided (notified) from the server (source) to the client (destination) using existing technology. For example, in glTF 2.0, the client can determine this status by referencing "MSFT_screencoverage" for MSFT_lod, as described in Non-Patent Document 9.

[0052] However, there was no way for the server (source) to notify the client (destination) of whether the mesh topology was consistent. Therefore, the client could not determine this status, and could not correctly select a combination of representations that would not cause flickering when performing bandwidth adaptation (bit rate adaptation). This could result in a decrease in the subjective quality of the rendered image.

[0053] 3. Adaptive Distribution Control of V-DMC Data Based on Topology Method 1 Therefore, when distributing such V-DMC data, as shown in the top row of the table in FIG. 10 , auxiliary information for performing adaptation to avoid flickering is provided (notified) from the server (distribution source) to the client (distribution destination) (Method 1).

[0054] For example, the first information processing device is provided with a control file generation unit that stores auxiliary information for adaptive control based on the topology of a mesh for a content file that stores data of the mesh having a hierarchical structure with different bit rates for each layer, and generates a control file that controls the distribution of the content file.

[0055] For example, in a first information processing method, a first information processing device stores auxiliary information for adaptive control based on the topology of a mesh in a content file that stores data of the mesh having a hierarchical structure in which the bit rate differs for each layer, and generates a control file that controls the distribution of the content file.

[0056] For example, the first program causes a computer to execute a process of storing auxiliary information for adaptive control based on the topology of a content file that stores data of a mesh having a hierarchical structure in which the bit rate differs for each layer, and generating a control file that controls the distribution of the content file.

[0057] Also, for example, the second information processing device is provided with a content acquisition unit that selects and acquires data of the mesh based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of content files that store data of the mesh having a hierarchical structure with different bit rates for each layer.

[0058] For example, in a second information processing method, a second information processing device selects and acquires data of a mesh based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of a content file that stores data of a mesh having a hierarchical structure with different bit rates for each layer.

[0059] For example, the second program causes the computer to execute a process of selecting and acquiring data for the mesh based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of a content file that stores data for a mesh having a hierarchical structure with different bit rates for each layer.

[0060] This allows auxiliary information to be provided from the source of a content file storing mesh data to the destination. Therefore, the destination can utilize the auxiliary information to achieve adaptive control based on mesh topology. This reduces flickering in rendered images and reduces degradation of the subjective quality of rendered images.

[0061] Note that this auxiliary information may be information for any data unit of mesh data. For example, this auxiliary information may be provided for each mesh stream. That is, the auxiliary information may correspond one-to-one to the mesh stream. That is, the auxiliary information may be information for each stream. Also, this auxiliary information may be provided for each segment of a mesh sequence. That is, multiple pieces of auxiliary information may be generated for one mesh stream, and each piece of auxiliary information may correspond to one of the segments of that mesh sequence. That is, the auxiliary information may be information for each segment. In this case, the auxiliary information may include information indicating the start position of the target segment.

[0062] The auxiliary information may be stored anywhere in the control file. For example, the control file may be an MPEG-DASH MPD. For example, the auxiliary information may be stored in a base mesh adaptation set. For example, in a first information processing device, a control file generation unit may store the auxiliary information in the base mesh adaptation set of the control file. In a second information processing device, a content acquisition unit may select and acquire mesh data based on the auxiliary information stored in the base mesh adaptation set of the control file. The auxiliary information may be stored in a supplemental property of the base mesh adaptation set. For example, in the first information processing device, a control file generation unit may store the auxiliary information in a supplemental property of the base mesh adaptation set. In a second information processing device, a content acquisition unit may select and acquire mesh data based on the auxiliary information stored in the supplemental property of the base mesh adaptation set. The auxiliary information may be stored in an adaptation set of atlas information. For example, in a first information processing device, a control file generation unit may store auxiliary information in an adaptation set of the atlas information in the control file. Furthermore, in a second information processing device, a content acquisition unit may select and acquire mesh data based on the auxiliary information stored in the adaptation set of the atlas information in the control file. Furthermore, the auxiliary information may be stored in a supplemental property of the adaptation set of the atlas information. For example, in a first information processing device, a control file generation unit may store auxiliary information in a supplemental property of the adaptation set of the atlas information. Furthermore, in a second information processing device, a content acquisition unit may select and acquire mesh data based on the auxiliary information stored in the supplemental property of the adaptation set of the atlas information.

[0063] Depending on how the LOD variations are created, the sections with consistent topology may or may not be the same among the LOD variations. For example, if mesh tracking is performed only on H-LOD (high-definition LOD) and reduction is performed based on the results to generate M-LOD (medium-definition LOD) and L-LOD (low-definition LOD), the sections with consistent topology may be the same among all LOD variations, as shown in A of Figure 11. In contrast, if each LOD variation is first generated and mesh tracking is performed individually, the sections with consistent topology may differ among all LOD variations, as shown in B of Figure 11.

[0064] Therefore, auxiliary information may be provided for each LOD. That is, auxiliary information may be information for each level (LOD) of a hierarchical structure. By doing so, even if the sections with consistent topology differ among all LOD variations, as in the example B of FIG. 11 , correct auxiliary information can be provided.

[0065] <Method 1-1> This auxiliary information may include any information necessary for performing adaptation to avoid flickering. For example, as shown in the second row from the top of the table in FIG. 10 , topology status information indicating the topology status of each frame may be stored in the control file as the auxiliary information (Method 1-1). For example, this topology status information may include information indicating whether the topology is consistent between frames. In other words, the auxiliary information may include topology status information indicating whether the topology is consistent between frames. Then, in the second information processing device, the content acquisition unit may select mesh data by lowering the acquisition priority of an adaptation set for a layer in the mesh hierarchical structure where the topology is not consistent between frames based on the topology status information. For example, the content acquisition unit may exclude from acquisition candidates an adaptation set for a layer whose topology is determined to be inconsistent according to the auxiliary information. Furthermore, the content acquisition unit may add such an adaptation set to acquisition candidates only if there are no acquisition candidates.

[0066] For example, topology status information may be stored in the MPEG-DASH MPD. In this case, the topology status information may be set using a supplemental property (SupplementalProperty) in the adaptation set of the base mesh for each LOD. For example, as shown in FIG. 12, a "TopologyStatus" element may be newly defined, and one or more "@status" elements may be stored in the supplemental property. Also, if there are multiple "TopologyStatus" elements, the "@start" element may be stored in the supplemental property.

[0067] As shown in FIG. 12 , the “TopologyStatus” element indicates topology status information for the LOD (hierarchy) corresponding to the adaptation set containing this information. When there is only one such element, it means that the topology status information is stream-based information. When there are two or more such elements, it means that the topology status information is segment-based information for a sequence. The “@status” element is flag information indicating the topology status of the corresponding section. For example, when the “@status” element is “0 (false),” it means that the section includes a frame-by-frame section in which the topology is not aligned between frames. When the “@status” element is “1 (true),” it means that the section does not include a frame-by-frame section in which the topology is not aligned between frames. The “@start” element is information indicating the index of the first segment of the section corresponding to the “@status” element (herein also referred to as the target “@status” element). If the next "@start" element exists, the section up to the previous segment will be the section corresponding to the target "@status" element. Also, if the next "@start" element does not exist, the section up to the end of the stream will be the section corresponding to the target "@status" element. The "@start" element is required when there are multiple "TopologyStatus" elements.

[0068] The topology status information may be stored in (the supplemental properties of) the adaptation set of the atlas information of each LOD in the MPD of MPEG-DASH.

[0069] For example, FIG. 13 shows an example of an MPD description in which topology status information is stored for each stream. In the example description in FIG. 13, the description in the portion enclosed by a bold frame is the topology status information. In this example, one piece of topology status information is stored for each stream. Therefore, this example is suitable for cases where there is little fluctuation in the topology state within the stream. Also, FIG. 14 shows an example of an MPD description in which topology status information is stored for each sequence segment. In the example description in FIG. 14, the description in the portion enclosed by a bold frame is the topology status information. In this example, multiple pieces of topology status information are stored for each stream. Therefore, this example is suitable for cases where there is much fluctuation in the topology state within the stream.

[0070] For example, suppose mesh data (V-DMC bitstream) has a hierarchical structure (LOD structure) as shown in Figure 15. The second LOD from the top is the level of detail appropriate for the current viewing distance (the distance from the viewpoint to the 3D object). The third LOD from the top is "@status=0" and is an LOD with an inconsistent topology, and the fourth LOD from the top is "@status=1" and is an LOD with a consistent topology.

[0071] Generally, in bandwidth-based adaptation control, combinations of representations that fit within the bandwidth are listed as acquisition candidates, and from among those candidates, the stream with the highest resolution is selected as the stream to actually acquire. For example, if only the third or fourth LOD from the top can fit within the bandwidth, adaptation sets for those LODs are listed as acquisition candidates. Then, from among those candidates, the highest resolution candidate, i.e., the stream with the third LOD from the top, is acquired.

[0072] 15, the LOD in the third row from the top is an LOD with an inconsistent topology, so the adaptation set of that LOD is excluded from the list of acquisition candidates. Therefore, the combination of representations of the LOD in the fourth row from the top is listed as an acquisition candidate, and the stream of that LOD is acquired.

[0073] In this way, adaptive control using topology status information can lower the acquisition priority of LODs that have low resolution relative to the viewing distance and inconsistent topology, thereby suppressing the occurrence of flickering in rendered images and reducing the degradation of the subjective quality of rendered images.

[0074] The degree of topology alignment between frames, which is the threshold for setting the value of the "@status" element to "0" or "1," can be any level. In other words, the level of topology alignment required for setting the value of the "@status" element to "1" can be set arbitrarily. For example, "@status=1" may be set only when all topologies are completely aligned between frames, or "@status=1" may be set when approximately X% or more of the topologies are aligned. The value of the "@status" element may also be set according to the magnitude of the topology deviation.

[0075] <Method 1-2> Furthermore, as shown in the third row from the top of the table in FIG. 10 , adaptation behavior information relating to an adaptive control method based on mesh topology may be stored as auxiliary information (Method 1-2). That is, the auxiliary information may include adaptation behavior information relating to an adaptive control method based on mesh topology for a target layer in a mesh hierarchical structure. Then, in the second information processing device, a content acquisition unit may select mesh data based on the adaptation behavior information. For example, this adaptation behavior information may indicate whether or not the layer that is the target of the adaptation behavior information in adaptive control based on mesh topology is to be included as a candidate for acquisition. Then, in the second information processing device, a content acquisition unit may select mesh data for a layer that is indicated to be included as a candidate for acquisition by the adaptation behavior information.

[0076] For example, the adaptation behavior information may be stored in the MPD of MPEG-DASH. In this case, the adaptation behavior information may be set using a supplemental property (SupplementalProperty) in the adaptation set of the base mesh of each LOD. For example, as shown in FIG. 16, a "TopologyStatus" element may be newly defined, and one or more "@adaptationBehavior" elements may be stored in the supplemental property. Also, if there are multiple "TopologyStatus" elements, the "@start" element may be stored in the supplemental property.

[0077] The "TopologyStatus" element and "@start" element are the same as in the example of Figure 12. The "@adaptationBehavior" element indicates how this LOD is handled when the player performs bandwidth adaptation for the corresponding section. For example, if this "@adaptationBehavior" element is "0 (false)", adaptation is performed excluding this LOD when the LOD is lowered from a level appropriate for the viewing distance and bandwidth adaptation is performed. If this "@adaptationBehavior" element is "1 (true)", adaptation is performed including this LOD when the LOD is lowered from a level appropriate for the viewing distance and bandwidth adaptation is performed.

[0078] The adaptation behavior information may be stored in (the supplemental properties of) the adaptation set of the atlas information of each LOD of the MPD of MPEG-DASH.

[0079] For example, FIG. 17 shows an example of an MPD description in which adaptation behavior information is stored for each stream. In the example description in FIG. 17, the description in the portion enclosed by a bold frame is the adaptation behavior information. In this example, one piece of adaptation behavior information is stored for each stream. Therefore, this example is suitable for cases where there is little fluctuation in the topology state within the stream. Also, FIG. 18 shows an example of an MPD description in which adaptation behavior information is stored for each sequence segment. In the example description in FIG. 18, the description in the portion enclosed by a bold frame is the adaptation behavior information. In this example, multiple pieces of adaptation behavior information are stored for each stream. Therefore, this example is suitable for cases where there is much fluctuation in the topology state within the stream.

[0080] For example, suppose mesh data (V-DMC bitstream) has a hierarchical structure (LOD structure) as shown in Fig. 19. The second-highest LOD is the level of detail appropriate for the current viewing distance (the distance from the viewpoint to the 3D object). The third-highest LOD is "@adaptationBehavior=0," and the fourth-highest LOD is "@adaptationBehavior=1."

[0081] Generally, in bandwidth-based adaptation control, combinations of representations that fit within the bandwidth are listed as acquisition candidates, and from among those candidates, the stream with the highest resolution is selected as the stream to actually acquire. For example, if only the third or fourth LOD from the top can fit within the bandwidth, adaptation sets for those LODs are listed as acquisition candidates. Then, from among those candidates, the highest resolution candidate, i.e., the stream with the third LOD from the top, is acquired.

[0082] However, in the example of Fig. 19, the LOD in the third row from the top has "@adaptationBehavior=0", so the adaptation set of that LOD is excluded from the list of acquisition candidates. Therefore, the combination of representations of the LOD in the fourth row from the top is listed as an acquisition candidate, and the stream of that LOD is acquired.

[0083] In this way, adaptive control according to the adaptation behavior information can lower the acquisition priority of LODs that have low definition for the viewing distance and have "@adaptationBehavior=0." In other words, adaptive control can be performed as specified by the adaptation behavior information. For example, by setting "@adaptationBehavior=0" for sections where the topology is not consistent between frames and "@adaptationBehavior=1" for sections where the topology is consistent between frames, control similar to that in the case of the topology status information described above can be performed. In other words, the occurrence of flickering in rendered images can be suppressed, and a reduction in the subjective quality of rendered images can be suppressed.

[0084] In the case of adaptation behavior information, it is possible to directly control the behavior of adaptation, i.e., whether or not to include the target LOD in the acquisition candidates, so it is possible to realize adaptive control based on any criteria, not just whether the topology is not consistent between frames.

[0085] Note that both the topology status information and the adaptation behavior information may be stored in the control file. For example, both the "@status" element and the "@adaptationBehavior" element may be defined (stored) in the MPD.

[0086] <Method 1-3> Alternatively, as shown in the fourth row from the top of the table in FIG. 10, a stream for avoiding flickering may be added (Method 1-3). For example, at least one of the above-described topology status information and adaptation behavior information may be stored in a control file, and a stream for avoiding flickering may be added to each LOD of an existing content set for adapting to a viewpoint position. The stream for avoiding flickering refers to a stream for lowering the overall rate without reducing the resolution of the geometry. For example, the stream may be an attribute (texture) stream with a lower rate corresponding to the same resolution of the geometry.

[0087] For example, the first information processing device may further include a content file generation unit that generates a content file and stores data for adaptive control based on mesh topology. Furthermore, in the second information processing device, if data for adaptive control based on mesh topology exists in a processing target layer of a mesh hierarchical structure, the content acquisition unit may prioritize the selection of the adaptive control data over control of a lower processing target layer. Note that the adaptive control data may include, for example, geometry data with a rate equivalent to the layer (LOD) corresponding to the adaptive control data and attribute data with a rate lower than the layer (LOD) corresponding to the adaptive control data.

[0088] The control file (MPD) settings are the same as those in the above-mentioned methods 1-1 and 1-2.

[0089] For example, as shown within the dotted line frame in FIG. 20 , adaptive control data, which is a stream for preventing flickering, is added to the adaptation set for each LOD of the attribute. As shown in FIG. 20 , "Representation 3" and "Representation 4" are added to the adaptation set for each LOD of the attribute. The streams corresponding to these are streams for preventing flickering. The bit rate of the stream corresponding to "Representation 3" is 10 Mbps, and the bit rate of the stream corresponding to "Representation 4" is 8 Mbps. In such a situation, even if a stream with a bit rate of 14 Mbps or a stream with a bit rate of 12 Mbps cannot fit into the bandwidth due to bandwidth reduction, for example, selection of the stream corresponding to "Representation 3" or "Representation 4" is prioritized over lowering the LOD. Therefore, reduction in the definition of the geometry can be suppressed.

[0090] In other words, by adding a flicker prevention stream, even if there is no existing stream that fits within the bandwidth and can prevent flicker, the flicker prevention stream can be selected to prevent the reduction in the definition of the geometry (topology) as described above. Therefore, the occurrence of flickering in the rendered image can be prevented. In other words, the reduction in the subjective quality of the rendered image can be prevented.

[0091] <Method 1-4> Furthermore, as shown in the bottom row of the table in FIG. 10 , information specifying the purpose of the flicker prevention stream may be stored in a control file (Method 1-4). For example, at least one of the above-described topology status information and adaptation behavior information may be stored in a control file, and the flicker prevention stream may be added to each LOD of an existing viewpoint-adapted content set. Furthermore, usage information specifying the purpose of the flicker prevention stream may be stored in the control file. The flicker prevention stream may then be made available only for the purpose indicated by the usage information. Note that this usage information may be stored anywhere in the control file. For example, the usage information may be stored in a representation that references the flicker prevention stream. Alternatively, the usage information may be stored in an essential property of the representation. Alternatively, the usage information may be stored in a supplemental property of the representation.

[0092] For example, in a first information processing apparatus, a control file generator may further store, in a control file, usage information indicating a usage of data for adaptive control based on a mesh topology. The control file generator may then store the usage information in a representation of the control file that references the data for adaptive control. For example, the control file generator may store the usage information in an essential property of the representation. The control file generator may also store the usage information in a supplemental property of the representation.

[0093] For example, in a second information processing device, a control file may further store usage information indicating the usage of adaptive control data based on a mesh topology. The content acquisition unit may then lower the acquisition priority of adaptive control data corresponding to the usage information among the mesh data present in a processing target layer of the mesh hierarchical structure. For example, the content acquisition unit may lower the acquisition priority of adaptive control data referenced by a representation in the control file that stores the usage information. For example, the content acquisition unit may lower the acquisition priority of adaptive control data referenced by a representation in the control file that stores the usage information in an essential property. Furthermore, the content acquisition unit may lower the acquisition priority of adaptive control data referenced by a representation in the control file that stores the usage information in a supplemental property.

[0094] By using the usage information in this manner, it is possible to prevent the added adaptive control data based on the mesh topology from being used for purposes other than its intended purpose. As described above, this adaptive control data (flicker avoidance stream) is a stream for reducing the overall rate without reducing the resolution of the geometry. For example, this adaptive control data (flicker avoidance stream) includes geometry data with a rate equivalent to that of the layer corresponding to the adaptive control data and attribute data with a rate lower than that of the layer corresponding to the adaptive control data. In other words, this adaptive control data (flicker avoidance stream) has its rate reduced in a biased manner toward the attribute. In a situation where flicker does not occur, a stream in which the bit rate is reduced uniformly, including the geometry, will result in less degradation of subjective quality (i.e., less degradation) than a stream in which the rate is reduced in a biased manner toward such an attribute, even at the same total rate. In other words, if flicker does not occur, it is desirable to avoid using this flicker avoidance stream as much as possible.

[0095] Therefore, as described above, by providing usage information and controlling selection of the flicker prevention stream only for an appropriate usage based on the usage information (i.e., when a flicker phenomenon occurs), it is possible to suppress use of the flicker prevention stream in situations where a flicker phenomenon does not occur, thereby suppressing a reduction in the subjective quality of the rendered image.

[0096] An example of an MPD description when this usage information is stored is shown in Fig. 21. In the example description in Fig. 21, the description in the portion enclosed in a bold frame is the usage signal for flicker avoidance, i.e., the usage information. In this example, an essential property is used in the representation that references the stream for flicker avoidance to set a descriptor "ForAvoidFlickering" that clearly indicates that this stream is for flicker avoidance. This descriptor "ForAvoidFlickering" is the usage information. As mentioned above, a supplemental property may be used instead of an essential property.

[0097] In this case, as shown in FIG. 22 , similar to the example in FIG. 20 , adaptive control data, which is a stream for preventing flickering, is added to the adaptation set for each LOD of the attribute (streams corresponding to "Representation 3" and "Representation 4" within the dotted line frame). However, in the example in FIG. 22 , a combination of representations that does not include a representation with "ForAvoidFlickering" set is preferentially selected and acquired. By controlling in this manner, it is possible to suppress the use of this stream for preventing flickering in situations where no flickering phenomenon occurs. Therefore, it is possible to suppress a reduction in the subjective quality of the rendered image.

[0098] <Combination> Each of the above-described methods may be applied in combination with any other method as long as no contradiction occurs. Three or more methods may be applied in combination. Furthermore, techniques that can be combined may include not only those shown in the table of FIG. 10 as "methods," but also all elements described in this specification. Furthermore, each of the above-described methods may be applied in combination with methods other than those described above.

[0099] <4. First Embodiment> <File Generation Device> The present technology may be applied to any device. FIG. 23 is a block diagram showing an example of the configuration of a file generation device, which is one aspect of an information processing device to which the present technology is applied. The file generation device 300 (first information processing device) shown in FIG. 23 is a device that encodes mesh data to generate a bitstream and stores the bitstream in a content file (e.g., ISOBMFF). The file generation device 300 also generates a control file (e.g., MPD) and a scene description that control the distribution of the content file. The file generation device 300 also uploads the generated content file, control file, and scene description to a server or the like.

[0100] Note that Fig. 23 shows the main processing units, data flows, etc., and does not necessarily include everything shown in Fig. 23. In other words, in file generation device 300, there may be processing units that are not shown as blocks in Fig. 23, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 23.

[0101] As shown in FIG. 23 , the file generation device 300 (first information processing device) has a preprocessing unit 311, an encoding unit 312, a content file generation unit 313, an SD generation unit 314, an MPD generation unit 315, a recording unit 316, and an upload unit 317.

[0102] The pre-processing unit 311 performs processing on mesh data before encoding. For example, the pre-processing unit 311 may acquire mesh data to be supplied to the file generation device 300. As pre-processing (processing before encoding), the pre-processing unit 311 may convert the acquired mesh data into V-DMC data. That is, the pre-processing unit 311 may generate a base mesh, a displacement vector, atlas information, etc. from the acquired mesh data. The pre-processing unit 311 may supply the generated V-DMC data (atlas information, base mesh, displacement vector, original mesh, attribute map, etc.) to the encoding unit 312.

[0103] The encoding unit 312 executes processing related to encoding of V-DMC data. For example, the encoding unit 312 may acquire V-DMC data supplied from the preprocessing unit 311. The encoding unit 312 may encode the V-DMC data using the V-DMC format to generate a V-DMC bitstream. The encoding unit 312 may supply the generated V-DMC bitstream to the content file generation unit 313.

[0104] The content file generation unit 313 executes processing related to the generation of a content file. For example, the content file generation unit 313 may acquire a V-DMC bitstream supplied from the encoding unit 312. The content file generation unit 313 may generate a content file (e.g., ISOBMFF) and store the V-DMC bitstream in the content file. Of course, this content file may have any specifications, but the following description will use ISOBMFF as an example. The content file generation unit 313 may supply the generated content file (ISOBMFF) to the SD generation unit 314. The content file generation unit 313 may supply the generated content file to the MPD generation unit 315. The content file generation unit 313 may supply the generated content file to the recording unit 316.

[0105] The SD generation unit 314 executes processing related to the generation of a scene description. For example, the SD generation unit 314 may acquire a content file (e.g., ISOBMFF) supplied from the content file generation unit 313. The SD generation unit 314 may generate a scene description (e.g., glTF2.0) corresponding to the content file. This scene description describes scene information that describes the situation (scene) of the 3D space in which the mesh data is placed. Of course, this scene description may be of any specification, but the following description will use glTF2.0 as an example. The SD generation unit 314 may supply the generated scene description (glTF2.0) to the recording unit 316.

[0106] The MPD generation unit 315 executes processing related to the generation of a control file. For example, the MPD generation unit 315 may acquire a content file (e.g., ISOBMFF) supplied from the content file generation unit 313. The MPD generation unit 315 may generate a control file (e.g., an MPD of MPEG-DASH) that controls the distribution of the content file. Therefore, the MPD generation unit 315 can also be said to be a control file generation unit. Of course, the control file may have any specifications, but the following description will be given taking an MPD as an example. The MPD generation unit 315 may supply the generated MPD to the recording unit 316.

[0107] The recording unit 316 has, for example, a recording medium and executes processing related to recording of information. For example, the recording unit 316 may acquire a content file supplied from the content file generation unit 313 and record it on the recording medium. The recording unit 316 may acquire a scene description supplied from the SD generation unit 314 and record it on the recording medium. The recording unit 316 may acquire an MPD supplied from the MPD generation unit 315 and record it on the recording medium. Furthermore, the recording unit 316 may read at least one of the content file, scene description, and MPD stored on the recording medium at a predetermined timing or based on an external request or the like, and supply it to the upload unit 317.

[0108] The upload unit 317 has, for example, a communication function and performs processing related to the supply of information to an external device. For example, the upload unit 317 may acquire a content file supplied from the recording unit 316. The upload unit 317 may also acquire a scene description supplied from the recording unit 316. The upload unit 317 may also acquire an MPD supplied from the recording unit 316. For example, the upload unit 317 may communicate with another device or the like using the communication function, and upload (supply) at least one of the content file, the scene description, and the MPD to the other device via the communication at a predetermined timing or based on an external request or the like. For example, the upload unit 317 uploads (supplies) at least one of the content file, the scene description, and the MPD to a server or the like that distributes the content file. Therefore, the upload unit 317 can also be considered a communication unit that communicates with another device. The upload unit 317 can also be considered a providing unit that provides at least one of the content file, the scene description, and the MPD. The upload unit 317 can also be said to be a supply unit that supplies at least one of a content file, a scene description, and an MPD.

[0109] <Encoding Unit> Fig. 24 is a block diagram showing an example of the main configuration of the encoding unit 312 in Fig. 23. Note that Fig. 24 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in the encoding unit 312, there may be processing units that are not shown as blocks in Fig. 24, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 24.

[0110] As shown in FIG. 24, the encoding unit 312 has an atlas information encoding unit 351, a base mesh encoding unit 352, a displacement vector correction unit 353, a displacement vector encoding unit 354, a mesh reconstruction unit 355, an attribute map conversion unit 356, an attribute encoding unit 357, and a multiplexing unit 358.

[0111] The atlas information encoding unit 351 performs processing related to encoding of atlas information. For example, the atlas information encoding unit 351 may acquire atlas information supplied from the pre-processing unit 311. The atlas information may include information about geometry patches, information about the geometry patch coordinate system, and the like. The atlas information encoding unit 351 may also encode the acquired atlas information using a predetermined encoding method to generate encoded data of the atlas information. The atlas information encoding unit 351 may also supply the generated encoded data of the atlas information to the multiplexing unit 358.

[0112] The base mesh encoding unit 352 performs processing related to encoding of the base mesh. For example, the base mesh encoding unit 352 may acquire a base mesh supplied from the pre-processing unit 311. The base mesh encoding unit 352 may acquire atlas information supplied from the pre-processing unit 311. The base mesh encoding unit 352 may quantize the acquired base mesh and encode it using a predetermined encoding method (e.g., Draco) to generate encoded data of the base mesh. The base mesh encoding unit 352 may supply the generated encoded data of the base mesh to the displacement vector correction unit 353. The base mesh encoding unit 352 may also supply the generated encoded data of the base mesh to the multiplexing unit 358.

[0113] The displacement vector correction unit 353 performs processing related to the correction of the displacement vector. For example, the displacement vector correction unit 353 may acquire a base mesh and a displacement vector supplied from the pre-processing unit 311. Alternatively, the displacement vector correction unit 353 may acquire encoded data of the base mesh supplied from the base mesh encoding unit 352. The displacement vector correction unit 353 may correct the displacement vector based on this information. For example, the displacement vector correction unit 353 may decode the acquired encoded data of the base mesh, compare the base mesh before and after encoding to determine encoding distortion of the base mesh, and correct the displacement vector in accordance with the encoding distortion. The displacement vector correction unit 353 may supply the corrected displacement vector to the displacement vector encoding unit 354. Alternatively, the displacement vector correction unit 353 may dequantize the decoded base mesh and supply it to the mesh reconstruction unit 355.

[0114] The displacement vector encoding unit 354 performs processing related to encoding of displacement vectors. For example, the displacement vector encoding unit 354 may acquire displacement vectors supplied from the displacement vector correction unit 353. The displacement vector encoding unit 354 may also acquire atlas information supplied from the pre-processing unit 311. The displacement vector encoding unit 354 may generate a displacement map by wavelet transforming the displacement vectors, quantizing them, and packing them into a two-dimensional region. In this case, the displacement vector encoding unit 354 may use the acquired atlas information. The displacement vector encoding unit 354 may also generate a displacement video in which the displacement map is used as frame images. In other words, the displacement video is a moving image in which the frame images are displacement maps, which are two-dimensional regions in which displacement vectors are packed. The displacement vector encoding unit 354 may encode the generated displacement video using a predetermined encoding method for 2D moving images to generate encoded data of displacement vectors (displacement video). The displacement vector encoding unit 354 may also supply the encoded data of the displacement vectors generated in this manner to the multiplexing unit 358. The displacement vector encoding unit 354 may also decode the generated encoded data, unpack the displacement vectors from the displacement map, and dequantize the displacement vectors. The displacement vector encoding unit 354 may also provide the dequantized displacement vectors to the mesh reconstruction unit 355.

[0115] The displacement vector encoding unit 354 may arithmetically encode the displacement vector to generate encoded data of the displacement vector. In this case, the displacement vector encoding unit 354 may arithmetically decode the encoded data to generate a displacement vector and supply the generated displacement vector to the mesh reconstruction unit 355. In this case, the displacement vector encoding unit 354 may use the acquired atlas information. Furthermore, the displacement vector encoding unit 354 may supply the used atlas information to the mesh reconstruction unit 355.

[0116] The mesh reconstruction unit 355 performs processing related to mesh reconstruction. For example, the mesh reconstruction unit 355 may acquire a base mesh supplied from the displacement vector correction unit 353. The mesh reconstruction unit 355 may also acquire a displacement vector supplied from the displacement vector encoding unit 354. The mesh reconstruction unit 355 may also acquire atlas information supplied from the displacement vector encoding unit 354. The mesh reconstruction unit 355 may reconstruct a mesh using the atlas information. The mesh reconstruction unit 355 may also supply the reconstructed mesh to the attribute map conversion unit 356.

[0117] The attribute map conversion unit 356 performs processing related to attribute map conversion. For example, the attribute map conversion unit 356 may acquire a reconstructed mesh supplied from the mesh reconstruction unit 355. The attribute map conversion unit 356 may also acquire atlas information supplied from the preprocessing unit 311. The attribute map conversion unit 356 may also acquire an original mesh and an attribute map input to the file generation device 300. The attribute map conversion unit 356 may convert the acquired attribute map based on other acquired information. For example, the attribute map conversion unit 356 may convert the attribute map based on the atlas information, the original mesh, etc., so that it corresponds to the reconstructed mesh. In other words, the attribute map conversion unit 356 can be said to generate a converted attribute map. Therefore, the attribute map conversion unit 356 can also be said to be an attribute map generation unit. The attribute map conversion unit 356 may supply the converted attribute map to the attribute encoding unit 357 .

[0118] The attribute encoding unit 357 performs processing related to encoding of attributes. For example, the attribute encoding unit 357 may acquire an attribute map supplied from the attribute map conversion unit 356. The attribute encoding unit 357 may also generate attribute video using the acquired attribute map as frame images. The attribute encoding unit 357 may also encode the generated attribute video using a predetermined encoding method for 2D video to generate encoded data of attributes. The attribute encoding unit 357 may also supply the generated encoded data of attributes to the multiplexing unit 358.

[0119] The multiplexing unit 358 performs processing related to multiplexing of encoded data (substreams). For example, the multiplexing unit 358 may acquire encoded data of atlas information supplied from the atlas information encoding unit 351. Alternatively, the multiplexing unit 358 may acquire encoded data of base meshes supplied from the base mesh encoding unit 352. Alternatively, the multiplexing unit 358 may acquire encoded data of displacement vectors supplied from the displacement vector encoding unit 354. Alternatively, the multiplexing unit 358 may acquire encoded data of attributes supplied from the attribute encoding unit 357. The multiplexing unit 358 may multiplex these pieces of encoded data as substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 358 can also be referred to as a bitstream generation unit (or V-DMC bitstream generation unit). The multiplexing unit 358 may supply the generated V-DMC bitstream to the content file generation unit 313. Therefore, the multiplexing unit 358 can also be said to be a supply unit (providing unit) of the V-DMC bitstream.

[0120] <Application of the Present Technology> The above-described present technology may be applied to the file generation device 300 configured as described above. For example, in the file generation device 300 (first information processing device), the MPD generation unit 315 may store auxiliary information for adaptive control based on the topology of a mesh in a content file storing data of a mesh having a hierarchical structure in which each layer has a different bit rate, and generate an MPD for controlling distribution of the content file. This auxiliary information may include topology status information indicating whether the topology is consistent between frames. The auxiliary information may also include adaptation behavior information regarding the adaptive control method for a target layer in the mesh hierarchical structure. The adaptation behavior information may also indicate whether the target layer is included as a candidate for acquisition in the adaptive control. The content file generation unit 313 may also generate a content file and store adaptive control data for the mesh. The adaptive control data may include geometry data with a rate equivalent to that of the layer corresponding to the adaptive control data and attribute data with a rate lower than that of the layer corresponding to the adaptive control data. The MPD generation unit 315 may also store usage information indicating the usage of the adaptation control data in the MPD. The MPD generation unit 315 may also store the usage information in a representation of the MPD that references the adaptation control data. For example, the MPD generation unit 315 may store the usage information in an essential property of the representation. The MPD generation unit 315 may also store the usage information in a supplemental property of the representation.

[0121] The auxiliary information may be information on a stream-by-stream basis. The auxiliary information may be information on a segment-by-segment basis. The auxiliary information may include information indicating the start position of a target segment. The MPD generation unit 315 may store the auxiliary information in an adaptation set of a base mesh of the MPD. The MPD generation unit 315 may store the auxiliary information in a supplemental property of the adaptation set. The MPD generation unit 315 may store the auxiliary information in an adaptation set of atlas information of the MPD. The MPD generation unit 315 may store the auxiliary information in a supplemental property of the adaptation set. The auxiliary information may be information for each layer in the hierarchical structure of mesh data.

[0122] Furthermore, in the file generation device 300 (first information processing device), the content file generation unit 313 may generate a content file and store mesh data in it. Then, the MPD generation unit 315 may generate an MPD corresponding to the generated content file.

[0123] In addition, in the file generation device 300 (first information processing device), the encoding unit 312 may encode mesh data to generate a bit stream, and the content file generation unit 313 may store the generated bit stream in a content file.

[0124] In addition, in the file generation device 300 (first information processing device), the encoding unit 312 may generate a bitstream by encoding and multiplexing data of a base mesh, a displacement vector, and attributes corresponding to a mesh.

[0125] Furthermore, in the file generation device 300 (first information processing device), the recording unit 316 may record the content file and the MPD. Furthermore, the upload unit 317 may supply the content file and the MPD.

[0126] With this configuration, the file generation device 300 can suppress the occurrence of flickering in the rendered image, i.e., the file generation device 300 can suppress a decrease in the subjective quality of the rendered image.

[0127] <Flow of File Generation Process> An example of the flow of the file generation process executed by the file generation device 300 will be described with reference to the flowchart of FIG.

[0128] When the file generation process is started, in step S301, the pre-processing unit 311 performs pre-processing on the data of the mesh to be processed, and generates V-DMC data.

[0129] In step S302, the encoding unit 312 performs a V-DMC encoding process, encodes the V-DMC data, and generates a V-DMC bitstream.

[0130] In step S303, the content file generation unit 313 generates a content file and stores the V-DMC bitstream in the content file.

[0131] In step S304, the SD generation unit 314 generates a scene description corresponding to the content file.

[0132] In step S305, the MPD generation unit 315 generates an MPD corresponding to the content file. Then, in step S306, the MPD generation unit 315 generates auxiliary information for performing adaptation to avoid flickering and stores it in the MPD. For example, for a content file storing data of meshes having a hierarchical structure in which the bit rate varies for each layer, the MPD generation unit 315 stores auxiliary information for adaptation control based on the topology of the meshes, and generates an MPD that controls the distribution of the content file.

[0133] In step S307, the recording unit 316 records the scene description, the MPD, and the content file.

[0134] In step S308, the upload unit 317 reads at least one of the scene description, the MPD, and the content file from the recording unit 316 at a predetermined timing or based on an external request, and uploads (supplies) the content file to a server or the like that distributes the content file.

[0135] When the process of step S307 ends, the file generation process ends.

[0136] <Flow of V-DMC Encoding Process> Next, an example of the flow of the V-DMC encoding process executed in step S302 of FIG. 25 will be described with reference to the flowchart of FIG.

[0137] When the V-DMC encoding process starts, the atlas information encoding unit 351 encodes the atlas information in step S341.

[0138] In step S342, the base mesh encoding unit 352 encodes the base mesh.

[0139] In step S343, the displacement vector correction unit 353 corrects the displacement vector.

[0140] In step S344, the displacement vector encoding unit 354 encodes the corrected displacement vector. For example, the displacement vector encoding unit 354 may pack the displacement vector into a displacement video and encode it using a 2D encoding method. Alternatively, the displacement vector encoding unit 354 may arithmetically encode the displacement vector.

[0141] In step S345, the mesh reconstructing unit 355 reconstructs the mesh.

[0142] In step S346, the attribute map conversion unit 356 converts the attribute map.

[0143] In step S347, the attribute encoding unit 357 encodes the attribute video using the attribute map as a frame image.

[0144] In step S348, the multiplexing unit 358 multiplexes the coded data of the atlas information, the coded data of the base mesh, the coded data of the displacement vector, and the coded data of the attribute to generate a V-DMC bitstream.

[0145] When the process of step S348 ends, the V-DMC encoding process ends, and the process returns to FIG.

[0146] By performing the processes described above, the file generation device 300 can suppress the occurrence of flickering in the rendered image, i.e., the file generation device 300 can suppress a decrease in the subjective quality of the rendered image.

[0147] 27 is a block diagram showing an example of the configuration of a playback device, which is one aspect of an information processing device to which the present technology is applied. The playback device 400 (second information processing device) shown in Fig. 27 is a device that decodes and plays back a V-DMC bitstream stored in a content file (e.g., ISOBMFF) (i.e., generates and displays a rendering image thereof).

[0148] Note that Fig. 27 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in playback device 400, there may be processing units that are not shown as blocks in Fig. 27, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 27.

[0149] As shown in FIG. 27 , the playback device 400 (second information processing device) has an SD acquisition unit 411 , an MPD acquisition unit 412 , a content acquisition unit 413 , a decoding unit 414 , a display control unit 415 , and a display unit 416 .

[0150] The SD acquisition unit 411 has, for example, a communication function and executes processing related to acquiring a scene description. For example, the SD acquisition unit 411 may use the communication function to communicate with an external device (another device, etc.) and acquire a scene description supplied from the external device (another device, etc.) via the communication. This scene description may be generated by, for example, the file generation device 300. Alternatively, this scene description may be acquired from, for example, a server that distributes content files generated by the file generation device 300. The SD acquisition unit 411 may supply the acquired scene description to the content acquisition unit 413.

[0151] The MPD acquisition unit 412 has, for example, a communication function, and executes processing related to acquisition of an MPD, which is a control file that controls distribution of content files. Therefore, the MPD acquisition unit 412 can also be called a control file acquisition unit. For example, the MPD acquisition unit 412 may communicate with an external device (another device, etc.) using the communication function, and acquire an MPD supplied from the external device (another device, etc.) via the communication. This MPD may be generated by, for example, the file generation device 300. Furthermore, this MPD may be acquired from, for example, a server that distributes content files generated by the file generation device 300. The MPD acquisition unit 412 may supply the acquired MPD to the content acquisition unit 413.

[0152] The content acquisition unit 413 has, for example, a communication function and executes processing related to content acquisition. For example, the content acquisition unit 413 may use the communication function to communicate with an external device (another device, etc.) and acquire a V-DMC bitstream stored in a content file from the external device (another device, etc.) via that communication. This content file (and the V-DMC bitstream) may be generated by, for example, the file generation device 300. Furthermore, this V-DMC bitstream may be acquired from, for example, a server that distributes content files generated by the file generation device 300. The content acquisition unit 413 may supply the acquired V-DMC bitstream to the decoding unit 414.

[0153] The decoding unit 414 executes processing related to decoding of a V-DMC bitstream. For example, the decoding unit 414 may acquire a V-DMC bitstream supplied from the content acquisition unit 413. The decoding unit 414 may decode the V-DMC bitstream and reconstruct (restore) mesh data. The decoding unit 414 may supply the mesh data to the display control unit 415.

[0154] The display control unit 415 executes processing related to image display control. For example, the display control unit 415 may acquire mesh data supplied from the decoding unit 414. The display control unit 415 may render the mesh data to generate a rendering image (display image). The display control unit 415 may supply the generated display image (rendering image) to the display unit 416 to display it.

[0155] The display unit 416 has a display device such as an LCD (Liquid Crystal Display) or an OLED (Organic Electro Luminescence Display), and executes processing related to image display. For example, the display unit 416 is controlled by the display control unit 415, and displays a display image (a rendering image of mesh data) supplied from the display control unit 415 using the display device.

[0156] <Decoding Unit> Fig. 28 is a block diagram showing an example of the main configuration of the decoding unit 414 in Fig. 27. Note that Fig. 28 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in the decoding unit 414, there may be processing units that are not shown as blocks in Fig. 28, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 28.

[0157] As shown in FIG. 28, the decoding unit 414 has a demultiplexing unit 451, an atlas information decoding unit 452, a base mesh decoding unit 453, a subdivision unit 454, a displacement vector decoding unit 455, a displacement vector application unit 456, an attribute decoding unit 457, and an attribute application unit 458.

[0158] The demultiplexing unit 451 performs demultiplexing processing. For example, the demultiplexing unit 451 may acquire a V-DMC bitstream supplied from the content acquisition unit 413. The demultiplexing unit 451 may also demultiplex the acquired V-DMC bitstream to extract coded data of atlas information, coded data of base meshes, coded data of displacement vectors, and coded data of attributes. Therefore, the demultiplexing unit 451 can also be considered an acquisition unit for the V-DMC bitstream or various information contained in the V-DMC bitstream. The demultiplexing unit 451 may supply the coded data of the extracted atlas information to the atlas information decoding unit 452. The demultiplexing unit 451 may also supply the coded data of the extracted base meshes to the base mesh decoding unit 453. The demultiplexing unit 451 may also supply the coded data of the extracted displacement vectors to the displacement vector decoding unit 455. Furthermore, the demultiplexing unit 451 may supply the coded data of the extracted attributes to the attribute decoding unit 457 .

[0159] The atlas information decoding unit 452 performs processing related to decoding of the atlas information. For example, the atlas information decoding unit 452 may acquire encoded data of the atlas information supplied from the demultiplexing unit 451. The atlas information decoding unit 452 may also decode the acquired encoded data of the atlas information to generate (restore) the atlas information. The atlas information decoding unit 452 may supply the generated atlas information to the base mesh decoding unit 453. The atlas information decoding unit 452 may supply the generated atlas information to the subdivision unit 454. The atlas information decoding unit 452 may supply the generated atlas information to the displacement vector decoding unit 455. The atlas information decoding unit 452 may supply the generated atlas information to the displacement vector application unit 456. The atlas information decoding unit 452 may supply the generated atlas information to the attribute decoding unit 457. The atlas information decoder 452 may provide the generated atlas information to the attribute application unit 458 .

[0160] The base mesh decoding unit 453 performs processing related to decoding of the base mesh. For example, the base mesh decoding unit 453 may acquire coded data of the base mesh supplied from the demultiplexing unit 451. The base mesh decoding unit 453 may also decode the acquired coded data (bit stream) of the base mesh using a predetermined decoding method (e.g., Draco) to generate (restore) the base mesh (e.g., a vertex list, a triangle list, etc.). In this case, the base mesh decoding unit 453 may acquire atlas information supplied from the atlas information decoding unit 452 and decode the coded data of the base mesh based on the atlas information. The base mesh decoding unit 453 may also supply the generated base mesh to the subdivision unit 454.

[0161] The subdivision unit 454 performs processing related to subdivision of the faces of the base mesh. For example, the subdivision unit 454 may obtain a base mesh supplied from the base mesh decoding unit 453. The subdivision unit 454 may subdivide (the faces of) the base mesh to generate division points. In this case, the subdivision unit 454 may obtain atlas information supplied from the atlas information decoding unit 452 and perform subdivision of the base mesh based on the atlas information. The subdivision unit 454 may supply the subdivided base mesh to the displacement vector application unit 456.

[0162] The displacement vector decoding unit 455 performs processing related to the decoding of displacement vectors. For example, the displacement vector decoding unit 455 may acquire encoded data of displacement vectors (i.e., a displacement bitstream) supplied from the demultiplexing unit 451. The displacement vector decoding unit 455 may decode the encoded data (bitstream) of displacement vectors to generate (restore) displacement vectors. For example, if the displacement vectors are encoded as displacement video, the displacement vector decoding unit 455 may decode the encoded data of the displacement vectors using a predetermined decoding method for 2D video, generate (restore) the displacement video, and unpack the displacement vectors from a displacement map, which is a frame image of the displacement video. Furthermore, if the displacement video is arithmetically coded, the displacement vector decoding unit 455 may arithmetically decode the encoded data of the displacement vectors to generate displacement vectors. In this case, the displacement vector decoding unit 455 may acquire atlas information supplied from the atlas information decoding unit 452 and decode the encoded data of the displacement vectors based on the atlas information. The displacement vector decoding unit 455 may supply the displacement vector obtained in this manner to the displacement vector application unit 456 .

[0163] The displacement vector application unit 456 performs processing related to application of a displacement vector to a subdivided base mesh. For example, the displacement vector application unit 456 may acquire a subdivided base mesh supplied from the subdivision unit 454. The displacement vector application unit 456 may acquire a displacement vector supplied from the displacement vector decoding unit 455. The displacement vector application unit 456 may apply a displacement vector to the vertices of the subdivided base mesh. In other words, the displacement vector application unit 456 may generate (reconstruct) a decoded mesh. In this case, the displacement vector application unit 456 may acquire atlas information supplied from the atlas information decoding unit 452 and apply a displacement vector based on the atlas information. The displacement vector application unit 456 may supply the decoded mesh generated in this manner to the attribute application unit 458.

[0164] The attribute decoding unit 457 executes processing related to attribute decoding. For example, the attribute decoding unit 457 may acquire coded data of attributes supplied from the demultiplexing unit 451. The attribute decoding unit 457 may also decode the acquired coded data of attributes using a predetermined decoding method for 2D video to generate (restore) attribute video. In this case, the attribute decoding unit 457 may acquire atlas information supplied from the atlas information decoding unit 452 and decode attributes based on the atlas information. The attribute decoding unit 457 may also supply an attribute map, which is a frame image of the generated attribute video, to the attribute application unit 458.

[0165] The attribute application unit 458 performs processing related to the application of attributes to the decoded mesh. For example, the attribute application unit 458 may acquire the decoded mesh supplied from the displacement vector application unit 456. The attribute application unit 458 may acquire an attribute map supplied from the attribute decoding unit 457. The attribute application unit 458 may apply the attributes of the attribute map to the decoded mesh. In this case, the attribute application unit 458 may acquire atlas information supplied from the atlas information decoding unit 452 and apply the attributes to the decoded mesh based on the atlas information. The attribute application unit 458 may supply the decoded mesh to which the attributes have been applied (i.e., the data of the decoded mesh) to the display control unit 415.

[0166] <Application of the Present Technology> The present technology described above may be applied to the playback device 400 configured as described above. For example, in the playback device 400 (second information processing device), the content acquisition unit 413 may select and acquire mesh data based on auxiliary information for adaptation control based on the topology of the mesh, which is stored in an MPD (control file) that controls the distribution of a content file storing mesh data having a hierarchical structure with different bit rates for each layer. This auxiliary information may include topology status information indicating whether the topology is consistent between frames. Then, in the playback device 400 (second information processing device), the content acquisition unit 413 may select mesh data by lowering the acquisition priority of an adaptation set for a layer in the mesh hierarchical structure whose topology is not consistent between frames based on the topology status information. Furthermore, this auxiliary information may include adaptation behavior information regarding the adaptation control method for a target layer in the mesh hierarchical structure. Then, in the playback device 400 (second information processing device), the content acquisition unit 413 may select mesh data based on the adaptation behavior information. Furthermore, the adaptation behavior information may indicate whether or not a target hierarchical layer is to be included as a candidate for acquisition in the adaptive control. Then, in the playback device 400 (second information processing device), the content acquisition unit 413 may select mesh data indicated by the adaptation behavior information to be included as a candidate for acquisition. Furthermore, in the playback device 400 (second information processing device), if adaptive control data for the mesh exists in a processing target hierarchical layer of the mesh, the content acquisition unit 413 may prioritize selection of the adaptive control data over control of a lower hierarchical layer to be processed. Note that the adaptive control data may include geometry data with a rate equivalent to that of the hierarchical layer corresponding to the adaptive control data and attribute data with a rate lower than that of the hierarchical layer corresponding to the adaptive control data. Furthermore, the MPD may further store usage information indicating a usage of the adaptive control data.Then, in the playback device 400 (second information processing device), the content acquisition unit 413 may lower the acquisition priority of adaptive control data corresponding to the usage information, among the data of meshes present in a processing target hierarchical layer of the mesh hierarchical structure. Also, in the playback device 400 (second information processing device), the content acquisition unit 413 may lower the acquisition priority of adaptive control data referenced by a representation in the MPD that stores the usage information. Also, in the playback device 400 (second information processing device), the content acquisition unit 413 may lower the acquisition priority of adaptive control data referenced by a representation in the MPD that stores the usage information in an essential property. Also, in the playback device 400 (second information processing device), the content acquisition unit 413 may lower the acquisition priority of adaptive control data referenced by a representation in the MPD that stores the usage information in a supplemental property.

[0167] The auxiliary information may be information on a stream-by-stream basis. The auxiliary information may be information on a segment-by-segment basis. The auxiliary information may include information indicating the start position of a target segment. In the playback device 400 (second information processing device), the content acquisition unit 413 may select and acquire mesh data based on auxiliary information stored in an adaptation set of a base mesh in the MPD. In the playback device 400 (second information processing device), the content acquisition unit 413 may select and acquire mesh data based on auxiliary information stored in a supplemental property of the adaptation set. In the playback device 400 (second information processing device), the content acquisition unit 413 may select and acquire mesh data based on auxiliary information stored in an adaptation set of atlas information in the MPD. In the playback device 400 (second information processing device), the content acquisition unit 413 may select and acquire mesh data based on auxiliary information stored in a supplemental property of the adaptation set. The auxiliary information may be information for each layer in the hierarchical structure of mesh data.

[0168] Furthermore, in the playback device 400 (second information processing device), the MPD acquisition unit 412 may acquire the MPD. Then, the content acquisition unit 413 may select and acquire mesh data based on the auxiliary information stored in the acquired control file.

[0169] In addition, in the playback device 400 (second information processing device), the content acquisition unit 413 may be configured to select and acquire a bitstream stored in a content file based on the auxiliary information. Then, the decoding unit 414 may decode the acquired bitstream to generate mesh data.

[0170] In addition, in the playback device 400 (second information processing device), the decoding unit 414 may generate mesh data by demultiplexing the bitstream, decoding sub-bitstreams of the base mesh, displacement vector, and attributes, and reconstructing them.

[0171] Furthermore, in the playback device 400 (second information processing device), the display control unit 415 may render mesh data to generate a display image, and cause the display unit 416 to display the display image.

[0172] With this configuration, the playback device 400 can suppress the occurrence of flickering in rendered images, i.e., the playback device 400 can suppress degradation of the subjective quality of the rendered images.

[0173] Note that processing units of the playback device 400 that are not essential for each method may be omitted.

[0174] <Playback Process Flow> An example of the process flow executed by the playback device 400 having the above configuration will be described with reference to the flowcharts of FIGS. 25 to 29.

[0175] When playback processing starts, the SD acquisition unit 411 acquires a scene description in step S401. In step S402, the MPD acquisition unit 412 acquires an MPD.

[0176] In step S403, the content acquisition unit 413 executes content acquisition processing to acquire desired content (V-DMC bitstream) based on the scene description and MPD. That is, the content acquisition unit 413 selects and acquires data of a mesh based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of a content file that stores data of a mesh having a hierarchical structure with a different bit rate for each layer.

[0177] In step S404, the decoding unit 414 executes a V-DMC decoding process to decode the acquired V-DMC bitstream and generate mesh data.

[0178] In step S405, the display control unit 415 renders the generated mesh data to generate a rendering image (display image), and causes the display unit 416 to display it.

[0179] When the process of step S405 is completed, the playback process ends.

[0180] <Flow of Content Acquisition Processing> Next, an example of the flow of the content acquisition processing executed in step S403 of Fig. 29 will be described with reference to the flowcharts of Fig. 30 and Fig. 31. Note that here, the case where Method 1-1 is applied, that is, the case where topology status information is applied as auxiliary information, will be described.

[0181] When the content acquisition process starts, the content acquisition unit 413 acquires information on the viewpoint position and the object position in step S421 of Fig. 30. In step S422, the content acquisition unit 413 identifies an adaptation set of atlas information corresponding to an LOD suitable for the current viewing situation based on the scene description.

[0182] In step S423, the content acquisition unit 413 identifies the adaptation sets of all related components based on the preselection associated with the adaptation set.

[0183] In step S424, the content acquisition unit 413 identifies an adaptation set of all components corresponding to an LOD lower than the current LOD for bit rate adaptation.

[0184] In step S425, the content acquisition unit 413 acquires the current bandwidth information.

[0185] In step S426, the content acquisition unit 413 acquires topology status information of each LOD from the adaptation set of each base mesh.

[0186] In step S427, the content acquisition unit 413 lists combinations of representations that fit within the bandwidth from among all representations in the adaptation set that correspond to an LOD that is appropriate for the current viewing situation and an LOD that is lower than that and has a TopologyStatus@status value of 1.

[0187] When the process of step S427 ends, the process proceeds to FIG.

[0188] 31, the content acquisition unit 413 determines whether or not one or more combinations have been listed. If it is determined that one or more combinations have not been listed, the process proceeds to step S432.

[0189] In step S432, the content acquisition unit 413 lists combinations of representations that fit within the bandwidth from among all representations in the adaptation set corresponding to the LOD for which the value of TopologyStatus@status is 0.

[0190] When the process of step S432 ends, the process proceeds to step S433. If it is determined in step S431 that one or more combinations have been listed, the process proceeds to step S433.

[0191] In step S433, the content acquisition unit 413 selects the combination of representations with the highest rate from the list.

[0192] In step S434, the content acquisition unit 413 acquires the stream of the selected representation.

[0193] In step S435, the content acquisition unit 413 determines whether or not the stream is at its end. If it is determined that the stream is not at its end, the process proceeds to step S436.

[0194] In step S436, the content acquisition unit 413 determines whether the viewpoint position or the object position has changed. If it is determined that the viewpoint position or the object position has changed, the process returns to step S421 in Fig. 30, and the subsequent processes are repeated. Also, if it is determined in step S436 in Fig. 31 that the viewpoint position or the object position has not changed, the process proceeds to step S437 in Fig. 31.

[0195] In step S437, the content acquisition unit 413 determines whether or not the bandwidth has changed. If it is determined that the bandwidth has changed, the process returns to step S425 in Fig. 30, and the subsequent processes are repeated.

[0196] Also, if it is determined in step S437 of FIG. 31 that the bandwidth has not fluctuated, the process returns to step S434, and the subsequent processes are repeated.

[0197] If it is determined in step S435 that the processing target has reached the end of the stream, the content acquisition process ends and the process returns to FIG.

[0198] <Flow of V-DMC Decoding Process> Next, an example of the flow of the V-DMC decoding process executed in step S404 of FIG. 29 will be described with reference to the flowchart of FIG.

[0199] When the decoding process starts, the demultiplexing unit 451 of the playback device 400 demultiplexes the V-DMC bitstream in step S451.

[0200] In step S452, the atlas information decoding unit 452 decodes the coded data of the atlas information to generate (restore) the atlas information.

[0201] In step S453, the base mesh decoding unit 453 decodes the coded data of the base mesh and generates (restores) the base mesh.

[0202] In step S454, the subdivision unit 454 subdivides the base mesh to generate division points.

[0203] In step S455, the disparity vector decoding unit 455 decodes the coded data (bit stream) of the disparity vector to generate (restore) the disparity vector.

[0204] In step S456, the displacement vector application unit 456 applies the displacement vectors to the vertices of the subdivided base mesh to generate a decoded mesh.

[0205] In step S457, the attribute decoding unit 457 decodes the coded data of the attribute to generate (restore) the attribute video.

[0206] In step S458, the attribute application unit 458 applies the attributes included in the attribute map included in the attribute video to the decoded mesh.

[0207] When the process of step S458 ends, the V-DMC decoding process ends, and the process returns to FIG. 29 .

[0208] By performing the processes described above, the playback device 400 can suppress the occurrence of flickering in rendered images, i.e., the playback device 400 can suppress degradation of the subjective quality of rendered images.

[0209] <Flow of Content Acquisition Processing> Next, another example of the flow of the content acquisition processing executed in step S403 of Fig. 29 will be described with reference to the flowcharts of Fig. 33 and 34. Note that here, the case where Method 1-2 is applied, that is, the case where adaptation behavior information is applied as auxiliary information, will be described.

[0210] When the content acquisition process is started, the content acquisition unit 413 executes the processes of steps S501 to S505 in FIG. 33 in the same manner as the processes of steps S421 to S425 in FIG.

[0211] In step S506, the content acquisition unit 413 acquires adaptation behavior information for each LOD from the adaptation set of each base mesh.

[0212] In step S507, the content acquisition unit 413 lists combinations of representations that fit within the bandwidth from among all representations in the adaptation set that correspond to an LOD that is suitable for the current viewing situation and an LOD that is lower than that and has a TopologyStatus@adaptationBehavior value of 1.

[0213] When the process of step S507 ends, the process proceeds to FIG.

[0214] 34, the content acquisition unit 413 determines whether or not one or more combinations have been listed. If it is determined that one or more combinations have not been listed, the process proceeds to step S512.

[0215] In step S512, the content acquisition unit 413 lists combinations of representations that fit within the bandwidth from among all representations in the adaptation set corresponding to the LOD for which the value of TopologyStatus@adaptationBehavior is 0.

[0216] When the process of step S512 ends, the process proceeds to step S513. If it is determined in step S511 that one or more combinations have been listed, the process proceeds to step S513.

[0217] The content acquisition unit 413 executes the processes of steps S513 to S517 in FIG. 34 in the same manner as the processes of steps S433 to S437 in FIG.

[0218] If it is determined in step S515 that the processing target has reached the end of the stream, the content acquisition process ends and the process returns to FIG.

[0219] By performing each process in this manner, the playback device 400 can suppress the occurrence of flickering in the rendered image, i.e., the playback device 400 can suppress degradation of the subjective quality of the rendered image.

[0220] <Flow of Content Acquisition Processing> Next, yet another example of the flow of the content acquisition processing executed in step S403 of Fig. 29 will be described with reference to the flowcharts of Fig. 35 and 36. Note that here, the case where Method 1-4 is applied, that is, the case where a flicker prevention stream and its usage information are applied in addition to auxiliary information, will be described.

[0221] When the content acquisition process is started, the content acquisition unit 413 executes the processes of steps S561 to S567 in FIG. 35 in the same manner as the processes of steps S421 to S427 in FIG.

[0222] When the process of step S567 ends, the process proceeds to Fig. 36. The content acquisition unit 413 executes the processes of steps S571 and S572 in Fig. 36 in the same manner as the processes of steps S431 and S432 in Fig. 31.

[0223] In step S573, the content acquisition unit 413 selects the combination of representations with the highest rate from the list, giving priority to combinations that do not use representations for which the "ForAvoidFlickering" signal is set.

[0224] The content acquisition unit 413 executes the processes of steps S574 to S577 in FIG. 36 in the same manner as the processes of steps S434 to S437 in FIG.

[0225] If it is determined in step S575 that the processing target has reached the end of the stream, the content acquisition process ends and the process returns to FIG.

[0226] By performing each process in this manner, the playback device 400 can suppress the occurrence of flickering in the rendered image, i.e., the playback device 400 can suppress degradation of the subjective quality of the rendered image.

[0227] 6. Supplementary Notes Polygon Shape In the above description, the polygon shape is a triangle, but this shape is just an example. The polygon shape may be any polygonal shape.

[0228] <Encoding Method> In the above, V-DMC has been used as an example of an encoding method to which the present technology can be applied, but the present technology is not limited to this example, and can be applied to any encoding method that encodes a base mesh, a displacement vector, an attribute map including texture, atlas information, or information equivalent thereto.

[0229] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0230] FIG. 37 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0231] In a computer 900 shown in FIG. 37, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0232] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0233] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0234] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0235] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0236] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0237] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0238] <Applicable Targets of This Technology> This technology can be applied to any encoding / decoding method. Also, this technology can be applied to any distribution method or file container.

[0239] Furthermore, the present technology can be applied to any configuration, for example, various electronic devices.

[0240] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0241] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0242] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0243] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0244] For example, the present technology can be applied to systems and devices used to provide viewing content, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for transportation, such as monitoring traffic conditions and controlling automatic driving. Furthermore, for example, the present technology can also be applied to systems and devices used for security. Furthermore, for example, the present technology can also be applied to systems and devices used for automatic control of machines, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, the present technology can also be applied to systems and devices used to monitor natural conditions, such as volcanoes, forests, and oceans, and wildlife. Furthermore, for example, the present technology can also be applied to systems and devices used for sports.

[0245] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0246] Furthermore, various information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, making one piece of data available (linked) when processing the other piece of data. That is, data associated with each other may be combined into one piece of data or may be stored as separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only a portion of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0247] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0248] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0249] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0250] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0251] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0252] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0253] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0254] The present technology can also be configured as follows. (1) An information processing device including a control file generation unit that stores auxiliary information for adaptive control based on the topology of a mesh for a content file storing data of a mesh having a hierarchical structure in which each layer has a different bit rate, and generates a control file for controlling distribution of the content file. (2) The information processing device according to (1), in which the auxiliary information includes topology status information indicating whether the topology is consistent between frames. (3) The information processing device according to (1) or (2), in which the auxiliary information includes adaptation behavior information regarding the adaptive control method for a target layer in the hierarchical structure. (4) The information processing device according to (3), in which the adaptation behavior information indicates whether the layer is included in a candidate for acquisition in the adaptive control. (5) The information processing device according to any of (1) to (4), further including a content file generation unit that generates the content file and stores data for the adaptive control of the mesh. (6) The information processing device according to (5), wherein the adaptive control data has geometry data with a rate equivalent to that of a hierarchical layer corresponding to the adaptive control data, and attribute data with a rate lower than that of a hierarchical layer corresponding to the adaptive control data. (7) The information processing device according to (5) or (6), wherein the control file generation unit further stores, in the control file, usage information indicating a usage of the adaptive control data. (8) The information processing device according to (7), wherein the control file generation unit stores the usage information in a representation of the control file that references the adaptive control data. (9) The information processing device according to (8), wherein the control file generation unit stores the usage information in an essential property of the representation. (10) The information processing device according to (8) or (9), wherein the control file generation unit stores the usage information in a supplemental property of the representation. (11) The information processing device according to any of (1) to (10), wherein the auxiliary information is information on a stream-by-stream basis.(12) The information processing device according to any one of (1) to (11), wherein the auxiliary information is information on a segment-by-segment basis. (13) The information processing device according to (12), wherein the auxiliary information includes information indicating a start position of the target segment. (14) The information processing device according to any one of (1) to (13), wherein the control file generation unit stores the auxiliary information in an adaptation set of a base mesh in the control file. (15) The information processing device according to (14), wherein the control file generation unit stores the auxiliary information in a supplemental property of the adaptation set. (16) The information processing device according to any one of (1) to (15), wherein the control file generation unit stores the auxiliary information in an adaptation set of atlas information in the control file. (17) The information processing device according to (16), wherein the control file generation unit stores the auxiliary information in a supplemental property of the adaptation set. (18) The information processing device according to any one of (1) to (17), wherein the auxiliary information is information for each layer of the hierarchical structure. (19) The information processing device according to any one of (1) to (18), further comprising a content file generation unit that generates the content file and stores data of the mesh, wherein the control file generation unit generates the control file corresponding to the generated content file. (20) The information processing device according to (19), further comprising an encoding unit that encodes data of the mesh and generates a bit stream, wherein the content file generation unit stores the generated bit stream in the content file. (21) The information processing device according to (20), wherein the encoding unit generates the bit stream by encoding and multiplexing data of a base mesh, a displacement vector, and attributes corresponding to the mesh. (22) The information processing device according to any one of (1) to (21), further comprising a recording unit that records the content file and the control file. (23) The information processing device according to any one of (1) to (22), further comprising a supply unit that supplies the content file and the control file.(24) An information processing method for storing auxiliary information for adaptive control based on the topology of a mesh in a content file storing mesh data having a hierarchical structure with a different bit rate for each layer, and generating a control file for controlling distribution of the content file. (25) A program for causing a computer to execute a process for storing auxiliary information for adaptive control based on the topology of a mesh in a content file storing mesh data having a hierarchical structure with a different bit rate for each layer, and generating a control file for controlling distribution of the content file.

[0255] (41) An information processing device comprising: a content acquisition unit that selects and acquires mesh data based on auxiliary information for adaptation control based on the topology of the mesh, the auxiliary information being stored in a control file that controls distribution of a content file storing mesh data having a hierarchical structure in which the bit rate varies for each layer. (42) The information processing device described in (41), wherein the auxiliary information includes topology status information indicating whether the topology is consistent between frames, and the content acquisition unit selects the mesh data by lowering the acquisition priority of an adaptation set for a layer in the hierarchical structure in which the topology is not consistent between the frames based on the topology status information. (43) The information processing device described in (41) or (42), wherein the auxiliary information includes adaptation behavior information regarding the adaptation control method for a target layer in the hierarchical structure, and the content acquisition unit selects the mesh data based on the adaptation behavior information. (44) The information processing device according to (43), wherein the adaptation behavior information indicates whether the hierarchical level is to be included in candidates for acquisition in the adaptive control, and the content acquisition unit selects data of the mesh indicated by the adaptation behavior information to be included in the candidates for acquisition. (45) The information processing device according to any of (41) to (44), wherein, when data for adaptive control of the mesh exists in a hierarchical level to be processed in the hierarchical structure, the content acquisition unit prioritizes selection of the data for adaptive control over control of lowering the hierarchical level to be processed. (46) The information processing device according to (45), wherein the data for adaptive control includes geometry data having a rate equivalent to that of the hierarchical level corresponding to the data for adaptive control and attribute data having a rate lower than that of the hierarchical level corresponding to the data for adaptive control. (47) The information processing device according to (45) or (46), wherein the control file further stores usage information indicating a usage of the data for adaptive control, and the content acquisition unit lowers the acquisition priority of the data for adaptive control corresponding to the usage information among data of the meshes existing in a hierarchical level to be processed in the hierarchical structure.(48) The information processing device according to (47), wherein the content acquisition unit lowers the acquisition priority of the adaptive control data referenced by a representation of the control file that stores the usage information. (49) The information processing device according to (48), wherein the content acquisition unit lowers the acquisition priority of the adaptive control data referenced by a representation of the control file that stores the usage information in an essential property. (50) The information processing device according to (48) or (49), wherein the content acquisition unit lowers the acquisition priority of the adaptive control data referenced by a representation of the control file that stores the usage information in a supplemental property. (51) The information processing device according to any of (41) to (50), wherein the auxiliary information is information in units of streams. (52) The information processing device according to any of (41) to (51), wherein the auxiliary information is information in units of segments. (53) The information processing device according to (52), wherein the auxiliary information includes information indicating a start position of the target segment. (54) The information processing device according to any one of (41) to (53), wherein the content acquisition unit selects and acquires data of the mesh based on the auxiliary information stored in an adaptation set of a base mesh in the control file. (55) The information processing device according to (54), wherein the content acquisition unit selects and acquires data of the mesh based on the auxiliary information stored in a supplemental property of the adaptation set. (56) The information processing device according to any one of (41) to (55), wherein the content acquisition unit selects and acquires data of the mesh based on the auxiliary information stored in an adaptation set of atlas information in the control file. (57) The information processing device according to (56), wherein the content acquisition unit selects and acquires data of the mesh based on the auxiliary information stored in a supplemental property of the adaptation set. (58) The information processing device according to any one of (41) to (57), wherein the auxiliary information is information for each hierarchical level in the hierarchical structure.(59) The information processing device according to any one of (41) to (58), further comprising a control file acquisition unit that acquires the control file, wherein the content acquisition unit selects and acquires data for the mesh based on the auxiliary information stored in the acquired control file. (60) The information processing device according to any one of (41) to (59), further comprising a decoding unit that is configured to select and acquire a bitstream stored in the content file based on the auxiliary information and decodes the acquired bitstream to generate data for the mesh. (61) The information processing device according to (60), wherein the decoding unit demultiplexes the bitstream and decodes and reconstructs sub-bitstreams of a base mesh, a displacement vector, and attributes to generate data for the mesh. (62) The information processing device according to any one of (41) to (61), further comprising a display control unit that renders data for the mesh to generate a display image and displays the display image. (63) An information processing method for selecting and acquiring mesh data based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls distribution of a content file storing mesh data having a hierarchical structure with a different bit rate for each layer. (64) A program for causing a computer to execute a process for selecting and acquiring mesh data based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls distribution of a content file storing mesh data having a hierarchical structure with a different bit rate for each layer.

[0256] 300 File generation device, 311 Preprocessing unit, 312 Encoding unit, 313 Content file generation unit, 314 SD generation unit, 315 MPD generation unit, 316 Recording unit, 317 Upload unit, 351 Atlas information encoding unit, 352 Base mesh encoding unit, 353 Displacement vector correction unit, 354 Displacement vector encoding unit, 355 Mesh reconstruction unit, 356 Attribute map conversion unit, 357 Attribute encoding unit, 358 Multiplexing unit, 400 Playback device, 411 SD acquisition unit, 412 MPD acquisition unit, 413 Content acquisition unit, 414 Decoding unit, 415 Display control unit, 416 Display unit, 451 Demultiplexing unit, 452 Atlas information decoding unit, 453 Base mesh decoding unit, 454 Subdivision unit 455 Displacement vector decoding unit, 456 Displacement vector application unit, 457 Attribute decoding unit, 458 Attribute application unit, 900 Computer

Claims

1. An information processing device having a control file generation unit that stores auxiliary information for adaptive control based on the topology of a mesh for a content file that stores mesh data having a hierarchical structure in which the bit rate differs for each layer, and generates a control file that controls the distribution of the content file.

2. The information processing device according to claim 1, wherein the auxiliary information includes topology status information indicating whether the topology is consistent between frames.

3. The information processing device according to claim 1, wherein the auxiliary information includes adaptation behavior information relating to the adaptive control method of a target layer in the hierarchical structure.

4. The information processing device according to claim 3, wherein the adaptation behavior information indicates whether or not the hierarchical layer is to be included as a candidate for acquisition in the adaptive control.

5. The information processing device according to claim 1, further comprising a content file generating unit that generates the content file and stores the adaptive control data of the mesh.

6. The information processing device according to claim 5, wherein said control file generating section further stores, in said control file, usage information indicating a usage of said adaptive control data.

7. The information processing device according to claim 1, wherein the auxiliary information includes information indicating a start position of a target segment.

8. The information processing device according to claim 1, wherein the control file generation unit stores the auxiliary information in an adaptation set of a base mesh in the control file.

9. An information processing device as described in claim 1, further comprising: an encoding unit that generates a bitstream by encoding and multiplexing data of a base mesh, a displacement vector, and attributes corresponding to the mesh; and a content file generation unit that generates the content file and stores the generated bitstream in the content file, wherein the control file generation unit generates the control file corresponding to the generated content file.

10. An information processing method for storing auxiliary information for adaptive control based on the topology of a mesh in a content file that stores mesh data having a hierarchical structure in which the bit rate varies for each layer, and generating a control file that controls distribution of the content file.

11. An information processing device comprising a content acquisition unit that selects and acquires mesh data based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of a content file storing mesh data having a hierarchical structure with different bit rates for each layer.

12. The information processing device described in claim 11, wherein the auxiliary information includes topology status information indicating whether the topology is consistent between frames, and the content acquisition unit, based on the topology status information, lowers the acquisition priority of an adaptation set of a layer in the hierarchical structure in which the topology is not consistent between the frames, and selects data of the mesh.

13. The information processing device of claim 11, wherein the auxiliary information includes adaptation behavior information relating to the adaptive control method of a target layer in the hierarchical structure, and the content acquisition unit selects data of the mesh based on the adaptation behavior information.

14. An information processing device as described in claim 13, wherein the adaptation behavior information indicates whether or not the hierarchy is to be included as a candidate for acquisition in the adaptive control, and the content acquisition unit selects data of the mesh that is indicated by the adaptation behavior information to be included as a candidate for acquisition.

15. The information processing device according to claim 11, wherein when data for the adaptive control of the mesh is present in a layer of the hierarchical structure to be processed, the content acquisition unit prioritizes the selection of the data for the adaptive control over control of a lower layer to be processed.

16. The information processing device of claim 15, wherein the control file further stores usage information indicating a usage of the adaptive control data, and the content acquisition unit lowers the acquisition priority of the adaptive control data corresponding to the usage information among the data of the meshes present in the hierarchical layer to be processed in the hierarchical structure.

17. The information processing device according to claim 11, wherein the auxiliary information includes information indicating a start position of a target segment.

18. The information processing device according to claim 11, wherein the content acquisition unit selects and acquires data of the mesh based on the auxiliary information stored in an adaptation set of a base mesh in the control file.

19. The information processing device of claim 11, further comprising a decoding unit configured to select and acquire a bitstream stored in the content file based on the auxiliary information, and to generate data of the mesh by demultiplexing the acquired bitstream and decoding and reconstructing sub-bitstreams of a base mesh, a displacement vector, and attributes.

20. An information processing method for selecting and acquiring mesh data based on auxiliary information for adaptive control based on the topology of the mesh, which is stored in a control file that controls the distribution of a content file storing mesh data having a hierarchical structure with different bit rates for each layer.

Images