Information processing apparatus and method

Abstract

The present disclosure pertains to an information processing device and method capable of suppressing a decrease in coding efficiency while suppressing a decrease in random accessibility. Generating random access metadata for performing a random access to the grid data based on a current sample as an inter-grid sample; and the random access metadata is stored in a content file that stores the grid data. Further, random access is controlled on the basis of random access metadata for performing random access on the basis of a current sample as an inter-grid sample, the random access metadata being stored in a content file storing the grid data, and reproduction of the grid data is started by the random access. The present disclosure can be applied to, for example, an information processing device, an information processing method, etc.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method capable of suppressing a reduction in random accessibility while suppressing a reduction in coding efficiency. Background Art

[0002] Conventionally, there has been the International Organization for Standardization Base Media File Format (ISOBMFF), which is a file container specification of the international standard technology of Moving Picture Experts Group-4 (MPEG-4) for moving picture compression (for example, see Non-Patent Document 1 and Non-Patent Document 2).

[0003] In addition, as an encoding method of a mesh, which is 3D data representing the three-dimensional structure of an object by connecting vertices, there is already video-based dynamic mesh coding (V-DMC) (for example, see non-patent documents 3 to non-patent documents 6).

[0004] In recent years, there has been a desire to establish a method for storing a V-DMC-encoded bitstream (also referred to as a V-DMC bitstream) in the International Organization for Standardization's Base Media File Format (ISOBMFF) and transmitting the bitstream. For example, it is assumed that ISO / IEC 23090-10 (see, for example, Non-Patent Document 7), which is a standard for V-PCC distribution technology, will be expanded and standardized. In this case, the V-DMC bitstream is stored in a track of the ISOBMFF.

[0005] Reference List

[0006] Non-patent documents

[0007] Non-Patent Document 1: “Information technology - Coding of audio-visual objects - Part 12: ISO base media file format, TECHNICAL CORRIGENDUM 1”, ISO / IEC FDIS14496-12:2020(E), ISO / IEC 14496-12:2015 / Cor.1, ISO / IEC JTC 1 / SC 29 / WG 11, 2016 / 6 / 3, ISO / IEC 14496-12:2022, 7th Edition, 2022-01 Non-Patent Document 2: “Information technology - Coding of audio-visual objects - Part 15: Carriage of network abstraction layer (NAL) unitstructured video in the ISO base media file Information technology — Coding of audiovisual objects — Part 15: Transport of structured video in the ISO base media file format — Network Abstraction Layer (NAL) units,” ISO / IEC FDI S14496-15:2014(E), ISO / IEC JTC 1 / SC 29 / WG 11, January 13, 2014, ISO / IEC 14496-15:2019, Edition 5, September 2019

[0008] Non-Patent Document 3: Khaled Mammou, Jungsun Kim, Alexis Tourapis, DimitriPodborski, Krasimir Kolarov, “Apple’s Dynamic Mesh Coding CfP Response,” ISO / IEC JTC 1 / SC 29 / WG 7m59281, April 2022

[0009] Non-Patent Document 4: Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Khaled Mammou, “VDMC support in the V3C framework,” ISO / IEC JTC1 / SC 29 / WG 7m60748, October 2022

[0010] Non-Patent Document 5: Alexis Tourapis, Jungsun Kim, Dimitri Podborski, Khaled Mammou, “Base mesh data substream format for VDMC”, ISO / IEC JTC 1 / SC 29 / WG 7m60362, July 2022 Non-Patent Document 6: “WD 1.0 of V-DMC”, ISO / IEC JTC 1 / SC 29 / WG 07N0486, MDS22184, November 16, 2022 Non-Patent Document 7: “Text of ISO / IEC FDIS23090-10 Carriage of Visual Volumetric Video-based Coding Data”, ISO / IEC JTC 1 / SC29 / WG 03N00241, August 20, 2021, ISO / IEC 23090-10:2022, Edition 1, May 2022 Summary of the Invention

[0011] Problems to be solved by the present invention

[0012] However, a grid (a base grid in the case of V-DMC) may include one or more subgrids. Due to this subgrid configuration, the frequency of random access points (RAPs) in the entire bitstream may decrease, and random accessibility may decrease. The reduction in RAPs can be suppressed by reducing the number of samples to be inter-coded and increasing the number of samples to be intra-coded, but in this case, there is a possibility of reduced coding efficiency.

[0013] The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to be able to suppress a decrease in random accessibility while suppressing a decrease in coding efficiency.

[0014] Solution to the problem

[0015] An information processing device according to one aspect of the present technology is an information processing device including: a random access metadata generation unit that generates random access metadata for performing random access on mesh data based on a current sample that is an inter-mesh sample; and a storage unit that stores the random access metadata in a content file storing the mesh data.

[0016] An information processing method according to one aspect of the present technology is an information processing method including: generating random access metadata for performing random access on mesh data based on a current sample that is an inter-mesh sample; and storing the random access metadata in a content file storing the mesh data.

[0017] An information processing device according to another aspect of the present technology is an information processing device including: a random access reproduction control unit that controls random access based on random access metadata for performing random access based on a current sample that is an inter-grid sample, the random access metadata being stored in a content file storing grid data; and a reproduction processing unit that starts reproduction of the grid data through random access.

[0018] An information processing method according to another aspect of the present technology is an information processing method including: controlling random access based on random access metadata for performing random access based on a current sample as an inter-grid sample, the random access metadata being stored in a content file storing grid data; and starting reproduction of the grid data through random access.

[0019] In an information processing apparatus and method according to one aspect of the present technology, random access metadata for performing random access on mesh data based on a current sample as an inter-mesh sample is generated; and the random access metadata is stored in a content file storing the mesh data.

[0020] In an information processing device and method according to another aspect of the present technology, random access is controlled based on random access metadata for performing random access based on a current sample as an inter-grid sample, the random access metadata being stored in a content file storing grid data; and reproduction of the grid data is started through random access. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a diagram used to explain the grid.

[0022] Figure 2 It is a diagram for explaining V-DMC.

[0023] Figure 3 is a diagram used to explain the subgrid.

[0024] Figure 4 is a diagram for explaining synchronization samples.

[0025] Figure 5 is a diagram illustrating an example of a method of storing a bit stream obtained by encoding mesh data in a file container and distributing the bit stream.

[0026] Figure 6is a diagram showing a main track configuration example of a content file.

[0027] Figure 7 is a diagram illustrating an example of random access metadata.

[0028] Figure 8 is a diagram illustrating an example of random access control.

[0029] Figure 9 is a diagram illustrating an example of random access control.

[0030] Figure 10 is a diagram illustrating an example of random access control.

[0031] Figure 11 is a diagram illustrating an example of random access metadata.

[0032] Figure 12 is a diagram illustrating an example of random access control.

[0033] Figure 13 is a diagram illustrating an example of random access metadata.

[0034] Figure 14 is a diagram illustrating an example of random access control.

[0035] Figure 15 is a diagram illustrating an example of random access metadata.

[0036] Figure 16 is a diagram illustrating an example of random access control.

[0037] Figure 17 is a diagram illustrating an example of random access metadata.

[0038] Figure 18 is a diagram illustrating an example of random access control.

[0039] Figure 19 is a diagram illustrating an example of random access metadata.

[0040] Figure 20 is a diagram showing a configuration example of a Matroska media container.

[0041] Figure 21 is a block diagram showing a main configuration example of a file generating device.

[0042] Figure 22 is a block diagram showing a main configuration example of a V-DMC encoding unit.

[0043] Figure 23 is a flowchart illustrating an example of the flow of file generation processing.

[0044] Figure 24 is a flowchart illustrating an example of the flow of a V-DMC encoding process.

[0045] Figure 25 It is a block diagram showing a main configuration example of a reproducing apparatus.

[0046] Figure 26 is a block diagram showing a main configuration example of a V-DMC decoding unit.

[0047] Figure 27 is a flowchart showing an example of the flow of the reproduction process.

[0048] Figure 28 : is a flowchart showing an example of the flow of a V-DMC decoding process.

[0049] Figure 29 : is a flowchart showing an example of the flow of random access reproduction processing.

[0050] Figure 30 is a block diagram showing a main configuration example of a computer. DETAILED DESCRIPTION

[0051] Hereinafter, modes for implementing the present disclosure (hereinafter referred to as embodiments) will be described. Note that the description will be given in the following order.

[0052] 1. Documents supporting technical content and technical terminology, etc.

[0053] 2. V-DMC and its distribution

[0054] 3. Transmission and control of random access metadata

[0055] 4. First embodiment (file generation device)

[0056] 5. Second Embodiment (Reproduction Device)

[0057] 6. Additional Notes

[0058] <1. Documents supporting technical content and technical terminology>

[0059] In addition to the contents disclosed in the embodiments, the scope disclosed in the present technology includes contents described in the following non-patent documents and the like known at the time of filing, and contents of other documents cited in the following non-patent documents and the like.

[0060] Non-Patent Document 1: (as described above)

[0061] Non-patent document 2: (as described above)

[0062] Non-Patent Document 3: (as described above)

[0063] Non-Patent Document 4: (as described above)

[0064] Non-Patent Document 5: (as described above)

[0065] Non-Patent Document 6: (as described above)

[0066] Non-Patent Document 7: (as described above)

[0067] Non-patent literature 8: https: / / www.matroska.org / index.html

[0068] That is, the contents described in the above-described non-patent literature, the contents of other documents cited in the above-described non-patent literature, and the like are also the basis for determining the supporting requirements.

[0069] <2. V-DMC and its distribution>

[0070] <grid>

[0071] Conventionally, as 3D data representing the three-dimensional structure of a three-dimensional structural object (an object having a three-dimensional shape), a mesh is known that represents the three-dimensional shape of the object surface by forming polygons with vertices and connections (also called edges).

[0072] like Figure 1 As shown in the upper left side of FIG, in the mesh, vertices 11 and connections 12 connecting vertices 11 form polygonal planes (polygons). In the following description, it is assumed that the polygons have triangular shapes. The surface of an object with a three-dimensional structure, that is, the three-dimensional shape of the object, is represented by polygons (also called faces). Note that a texture 13 can be attached (also called applied) to each face of the mesh.

[0073] Grid data includes, for example, Figure 1 The information shown in the lower part of the Figure 1 The vertex information 14 shown first from the left in the lower portion of is information indicating the three-dimensional positioning (three-dimensional coordinates (X, Y, Z)) of each vertex 11 constituting the mesh. Figure 1 The connection information 15 shown second from the left in the lower portion of is information indicating each connection (edge) 12 constituting the mesh. Figure 1 The texture image 16 shown third from the left in the lower portion is mapping information of the texture 13 attached to each face. Figure 1 The UV map 17 shown fourth from the left in the lower portion of FIG is information indicating the correspondence relationship between the vertices 11 and the texture 13. In the UV map 17, the coordinates (UV coordinates) of each vertex 11 in the texture image 16 are indicated.

[0074] Note that the mesh data may change in the time direction like a moving image of 2D data. That is, the mesh data may have a structure in which frames including mesh data representing a three-dimensional structure at different times are continuous.

[0075] <v-dmc>

[0076] As a method of encoding such a lattice, for example, there is video-based dynamic lattice coding (V-DMC) as disclosed in Non-Patent Documents 3 to 6.

[0077] In V-DMC, the mesh to be encoded (referred to as the original mesh in this specification) is represented by a base mesh with a lower level of detail (i.e., coarser) than the original mesh and displacement vectors of the segmentation points obtained by subdividing the base mesh. The base mesh and displacement vectors are then encoded. For example, a dynamic mesh stream generated by capturing with a camera is encoded. This dynamic mesh stream is (unregistered) mesh data with a structure that changes every frame.

[0078] For example, suppose there is a Figure 2 The original mesh is shown in the top part of . Figure 2 In the figure, black dots indicate vertices, and lines connecting the dots indicate connections (edges). As described above, a mesh originally forms a surface (polygon) by vertices and edges in space, but here, for ease of description, the mesh is described as a group of vertices connected linearly (in series) in a plane.

[0079] By extracting some vertices of the original mesh, we can form Figure 2 The coarse (low-fineness) grid shown second from the top in . This is called the base grid.

[0080] By subdividing each polygon of the base mesh, adding vertices and edges, such as Figure 2 As shown in the third section from the top of the figure, this subdivision allows, for example, the addition of vertices equal to the number obtained by decimating the original mesh. In other words, by subdividing the base mesh, a mesh with the same number of vertices as the original mesh is obtained. In this specification, the vertices added by this subdivision are also referred to as split points.

[0081] However, when the vertices of the original mesh are extracted, the connections have been updated, and the split points are formed on these updated connections (edges). Therefore, even if the number of vertices is the same as the number of vertices of the original mesh by subdivision, the subdivided base mesh ( Figure 2 The positions of the vertices in the third part from the top of the mesh are also consistent with the original mesh ( Figure 2 In other words, if Figure 2 As shown in the bottom part of , ideally, the original mesh can be restored by moving (displacing) the positions of the vertices of the subdivided base mesh to the vertex positions of the original mesh. In this specification, the displacement (movement) of such a vertex expressed as a vector is called a displacement vector.

[0082] That is, ideally, the original mesh can be represented as a base mesh and displacement vectors. By representing the original mesh as a base mesh and displacement vectors in this way, the number of polygons (i.e., the number of vertices and the number of edges) is reduced. Therefore, by encoding the base mesh and displacement vectors instead of the original mesh, a decrease in encoding efficiency (an increase in the amount of code) can be suppressed.

[0083] That is, in V-DMC, if Figure 1 As shown, V-DMC data including a base mesh, displacement vectors, attributes (textures), and atlas information is generated from 3D data such as mesh data, and the V-DMC data is encoded. The atlas information is information required to reconstruct the mesh, and includes, for example, information indicating the correspondence between the base mesh, displacement vectors, and textures. For example, the atlas information may include information associating patches of the mesh with patches of the attributes, such as Figure 1 UV Mapping 17.

[0084] When encoding V-DMC data, the base mesh, displacement vectors, attributes (texture), and atlas information that constitute the V-DMC data are encoded, and a corresponding bitstream (encoded data) is generated. The base mesh stream, displacement vector stream, attribute (texture) stream, and atlas information stream are then set as substreams to generate a single bitstream (also called a V-DMC bitstream).

[0085] As mentioned above, since mesh data can change in the time direction, it has a temporal structure (a continuous frame structure). Therefore, in V-DMC, mesh data is segmented and processed according to this temporal structure. The processing unit in the time direction is called a sample. In other words, encoding and decoding are performed using one frame or multiple consecutive frames of mesh data as a sample. Typically, one frame of mesh data is one sample.

[0086] A sample may be intra-coded independently of other samples, or it may be inter-coded using other samples (other frames) as key samples (key frames).

[0087] Encode the atlas information in a predetermined manner. The displacement vectors are packed into the frame images (also referred to as displacement maps or geometry maps) and encoded as a moving image (also referred to as displacement video or geometry video) using an encoding scheme for moving images. The attributes (textures) are packed into the frame images (also referred to as texture images or attribute maps) and encoded as a moving image (also referred to as texture video or attribute video) by using an encoding scheme for moving images. In the case of intra-frame encoding, the base mesh is encoded independently of other samples by a predetermined encoding scheme (e.g., Draco, etc.), and in the case of inter-frame encoding, the difference from the key sample is obtained and the difference is encoded. Note that the encoding can be skipped when there is no difference between the sample to be processed and the key sample (when the sample to be processed is the same as the key sample).

[0088] When decoding the V-DMC bitstream, each substream of the base mesh, displacement vectors, attributes (textures), and atlas information is decoded by a decoding method corresponding to a corresponding one in the encoding method, and the atlas information, base mesh, displacement vectors, and textures are restored (generated). Then, as described above, the base mesh is subdivided based on the atlas information, the displacement vectors are applied to the vertices (i.e., the vertices are displaced), and the textures are applied to the faces, thereby reconstructing the mesh (generating the restored mesh).

[0089] <Distribution of V-DMC Bitstream>

[0090] In recent years, there has been a desire to establish a method of storing such V-DMC streams in the International Organization for Standardization (ISO) base media file format (ISOBMFF) or the like and distributing the V-DMC bitstream. For example, as described in Non-Patent Document 1 and Non-Patent Document 2, ISOBMFF is a file container specification for the international standard technology of moving image compression, "Moving Picture Experts Group-4 (MPEG-4)". For example, it can be assumed that ISO / IEC 23090-10, which is a distribution technology standard for V-PCC described in Non-Patent Document 7, is extended, and the V-DMC bitstream is stored in ISOBMFF. In this case, the V-DMC bitstream is stored in an ISOBMFF track. For example, it can be conceived to store the substreams of the atlas information, base mesh, displacement vectors, and attributes in corresponding different ISOBMFF tracks.

[0091] <Sub-Mesh>

[0092] Incidentally, the mesh (the base mesh in the case of V-DMC) may include one or more sub-meshes. The sub-mesh is a unit of encoding processing in the spatial direction (region direction). That is, the sub-meshes can be encoded independently of each other. For example, if the mesh 31 shown in A of Figure 3 is divided as shown in Figure 3 If subgrid 31A and subgrid 31B are shown in FIGB , subgrid 31A and subgrid 31B can be encoded independently of each other. Thus, subgrid 31A and subgrid 31B can be decoded independently. In other words, only subgrid 31A can be decoded, or only subgrid 31B can be decoded. In other words, only the desired portion of the grid can be decoded, and unnecessary increases in the decoding processing load can be suppressed. Consequently, increases in the decoding processing load and processing time can be suppressed.

[0093] <Random Access Control>

[0094] However, due to this sub-grid configuration, there is a possibility that the frequency of random access points (RAPs) of the entire bit stream decreases and random accessibility decreases.

[0095] Figure 4 is a diagram showing a configuration example of a bit stream obtained by encoding mesh data and stored in ISOBMFF. Figure 4 The bitstream shown is a bitstream obtained by encoding grid data having two subgrids (subgrid #1, subgrid #2), and has 9 samples from picture order count (POC) 0 to POC 8. Note that POC indicates the decoding order of each sample.

[0096] The samples corresponding to the subgrids are also called subsamples. Figure 4 In the case of the example of , each sample has a subsample corresponding to subgrid #1 and a subsample corresponding to subgrid #2. The square denoted "Draco" indicates an intra-coded subsample, and the square denoted "motion" indicates an inter-coded subsample.

[0097] Among samples encoded intra-frame, only samples can be decoded normally. On the other hand, in order to decode inter-frame encoded samples normally, key samples (e.g., samples immediately preceding them) need to have been decoded normally. However, as described above, in the case of grid data, each sample is encoded for each subgrid. Therefore, in order to decode a certain sample normally, it is only necessary that all subsamples constituting the sample can be decoded normally. For example, due to Figure 4 The subsample of subgrid #1 of POC4 in the bitstream is intra-coded, so it is only necessary to start decoding from POC4 to decode this subsample normally. On the other hand, the subsample of subgrid #2 of POC4 is inter-coded, and it is only necessary to start decoding from POC2 to decode this subsample normally. In other words, in order to decode the sample of POC4 normally, it is only necessary to start decoding from POC2.

[0098] In the case of 2D data, in ISOBMFF, a sample in the middle of a sequence designated as a RAP (also called a synchronization sample) is accessed, and random access reproduction starting from this sample can be performed. Samples that can be designated as synchronization samples are intra-coded samples.

[0099] However, methods for storing mesh data in file containers such as ISOBMFF have just begun to be studied, and there is no method for achieving random access like 2D data. Even if random access is applied, there is no method for controlling random access based on subsamples of 3D data, at least in file containers such as ISOBMFF. That is, in the case of a bit stream of mesh data, for example, as in Figure 4 In the POC2 sample in , each subgrid can be encoded by different methods, but which subgrid is encoded and how it is encoded in the file container (i.e., the encoding structure of each subgrid) is not managed. Figure 4 In POC0, only samples in which all subsamples are intra-coded can be designated as synchronization samples. That is, in this case, decoding needs to start from POC0.

[0100] As described above, due to the configuration of the subgrid, the frequency of RAPs in the entire bitstream may be reduced. Consequently, random accessibility may be reduced. Consequently, the viewing experience quality of trick playback (e.g., skip playback, fast forward, rewind, etc.) may be degraded.

[0101] Note that a method of suppressing the reduction of RAP by reducing the number of samples to be subjected to inter-frame coding and increasing the number of samples to be subjected to intra-frame coding is conceivable, but in this case, there is a possibility that the coding efficiency of the bit stream decreases.

[0102] <3. Transmission and Control of Random Access Metadata>

[0103] <Method 1>

[0104] Therefore, when a bit stream obtained by encoding mesh data is stored in a file container, it is possible to suppress a decrease in random accessibility while suppressing a decrease in encoding efficiency. Figure 5 As shown in the top row of the table, random access metadata can be stored in files (method 1).

[0105] Here, random access metadata is control information for performing random access to grid data. More specifically, random access metadata is control information for performing random access to grid data based on a current sample that is a sample to be processed. For example, random access metadata may be control information for performing random access to grid data based on a current sample that is an inter-grid sample. Here, an inter-grid sample is a sample that includes an inter-grid subsample, and an inter-grid subsample is a subsample of inter-frame encoded grid data. Note that in this specification, a subsample of intra-frame encoded grid data is also referred to as an intra-grid subsample, and a sample that includes only an intra-grid subsample is also referred to as an intra-grid sample.

[0106] For example, a file generating device (a first information processing device) that stores a bit stream obtained by encoding mesh data in a file container may include: a random access metadata generating unit that generates random access metadata for performing random access on the mesh data based on a current sample that is an inter-grid sample; and a storage unit that stores the random access metadata in a content file storing the mesh data.

[0107] In addition, a file generation device (first information processing device) that stores a bit stream obtained by encoding mesh data in a file container can generate random access metadata for performing random access to the mesh data based on a current sample that is an inter-grid sample, and can store the random access metadata in a content file that stores the mesh data.

[0108] Therefore, the first information processing device can also set the inter-grid samples as synchronization samples (i.e., RAP). That is, the first information processing device can also set the samples other than the samples that only include intra-frame coded subsamples as synchronization samples (i.e., RAP). Therefore, the first information processing device can generate a content file that can be randomly accessed by more various samples while suppressing the reduction in coding efficiency. That is, the first information processing device can suppress the reduction in random accessibility while suppressing the reduction in coding efficiency. For example, the first information processing device can suppress the reduction in the viewing experience quality of special effects reproduction (e.g., jump reproduction, fast forward, rewind, etc.) while suppressing the reduction in coding efficiency.

[0109] For example, a reproduction device (second information processing device) that decodes a bit stream stored in a file container and reproduces mesh data may include: a random access reproduction control unit that controls random access based on random access metadata for performing random access based on a current sample that is an inter-grid sample, the random access metadata being stored in a content file storing the mesh data; and a reproduction processing unit that starts reproduction of the mesh data through random access.

[0110] In addition, a reproduction device (second information processing device) that decodes a bit stream stored in a file container and reproduces mesh data can control random access based on random access metadata for performing random access based on a current sample that is an inter-grid sample, which random access metadata is stored in a content file storing the mesh data, and can start reproduction of the mesh data through random access.

[0111] Therefore, the second information processing device can easily (without parsing the bitstream) perform random access to samples other than samples that only include intra-frame encoded subsamples based on the information of the content file (random access metadata). Therefore, the second information processing device can perform random access to a wider variety of samples of the content file while suppressing a decrease in coding efficiency. In other words, the second information processing device can suppress a decrease in random accessibility while suppressing a decrease in coding efficiency. For example, the second information processing device can suppress a decrease in the viewing experience quality of special effects reproduction (e.g., jump reproduction, fast forward, rewind, etc.) while suppressing a decrease in coding efficiency.

[0112] Random access metadata can be stored anywhere in the content file. For example, in the first information processing device, the storage unit can store the random access metadata in a track storing mesh data in the content file. Furthermore, in the second information processing device, the random access metadata can be stored in a track storing mesh data in the content file.

[0113] The configuration of tracks in a content file can be anything. For example, Figure 6 As shown, the substream of atlas information (atlas data substream), the substream of base grid (base grid substream), the substream of displacement vector (geometry video substream) and the substream of attribute (texture) (attribute video substream) can be stored in different tracks. Figure 6 An example of a track configuration for a content file is shown. In this example, the atlas information substream (atlas data substream) is stored in the atlas track (atlas track). The base mesh substream (base mesh substream) is stored in the base mesh track (base mesh track). The displacement vector substream (geometry video substream) is stored in the geometry track (geometry track). The attribute (texture) substream (attribute video substream) is stored in the attribute track (attribute track).

[0114] When a content file has such a track configuration, random access metadata can be stored in the base grid track. For example, the random access metadata can be stored in the area storing metadata for the base grid track. Therefore, a second information processing device can easily perform random access to a wider variety of samples of the base grid by simply referencing the base grid track. In other words, a first information processing device can easily generate a content file that allows random access to a wider variety of samples of the base grid by simply referencing the base grid track.

[0115] The random access metadata may be stored in the content file in any manner. For example, in the first information processing device, the storage unit may store the random access metadata as a parameter of the sample group. Furthermore, in the second information processing device, the random access metadata may be stored in the content file as a parameter of the sample group.

[0116] Figure 7 is a diagram showing an example of random access metadata. Figure 7 As shown, an entry (MeshRollRecoveryEntry) of a sample group may be newly defined as random access metadata, and control information for performing random access to mesh data may be stored in the entry (MeshRollRecoveryEntry).

[0117] When the sample corresponding to the random access metadata is decoded, the random access metadata can be referenced. Random access metadata can be set for each sample. Furthermore, random access metadata can be shared by multiple samples. For example, random access metadata can be set for each predetermined sample section. In this case, the common random access metadata is applied to each sample in the sample section. Furthermore, random access metadata can be set for an entire sequence. In this case, the common random access metadata is applied to all samples in the sequence.

[0118] Furthermore, random access metadata may be set only for some samples. For example, random access metadata may be set only for predetermined samples. For example, in the first information processing device, the random access metadata generation unit may generate random access metadata corresponding to inter-grid samples. Furthermore, in the second information processing device, the random access metadata may correspond to inter-grid samples. Note that random access metadata may be set only for some inter-grid samples.

[0119] Random access metadata can be applied to any mesh data. For example, random access metadata can be applied to a base mesh of V-DMC. For example, in the first information processing device and the second information processing device, the mesh data may be data of a base mesh having a lower fineness than the original mesh, the base mesh being generated by extracting vertices from the original mesh to be encoded, the original mesh to be encoded including vertices and connections representing the three-dimensional structure of the object. In addition, random access metadata can be applied in a case where the original mesh is set as an encoding target without applying V-DMC and the bitstream of the original mesh is stored in a file container such as ISOBMFF for distribution. For example, in the first information processing device and the second information processing device, the mesh data may be data of the original mesh to be encoded including vertices and connections representing the three-dimensional structure of the object.

[0120] Furthermore, the file container storing the bitstream of the mesh data and the random access metadata may be any file container. For example, in the first information processing device and the second information processing device, the content file may be a file container conforming to the International Organization for Standardization Base Media File Format (ISOBMFF).

[0121] <Method 1-1>

[0122] In the case of applying method 1, for example, Figure 5 As shown in the second row from the top of the table, random access metadata including control information for specifying a decoding start sample may be stored in a file (method 1-1). In this specification, a "decoding start sample" is also referred to as a decoding start sample.

[0123] For example, in the first information processing device, the random access metadata may include control information that specifies a decoding start sample.

[0124] For example, Figure 8 As shown, control information designating a decoding start sample for normal decoding of a sample of POC4 may be included as random access metadata corresponding to the sample of POC4. Figure 8 In the example of , for both subgrid #1 and subgrid #2, there are intra-grid subsamples in samples from POC2 to POC4. Therefore, if decoding starts from the sample of POC2, the sample of POC4 can be decoded normally. Control information ("rolling" negative distance) that designates such a sample of POC2 as the decoding start sample can be included in the random access metadata corresponding to the sample of POC4.

[0125] Note that in Figure 8 In the example of , as the decoding start sample for normal decoding of the sample of POC4, the sample of POC2 is closest to the sample of POC4. As described above, the random access metadata may include control information that specifies the sample closest to the sample corresponding to the random access metadata among the decoding start samples that meet the condition (that is, the decoding start sample for normal decoding of the sample corresponding to the random access metadata).

[0126] In this case, the reproduction device (second information processing device) can start decoding from the decoding start sample specified by the control information. For example, in the second information processing device, the random access metadata may include control information specifying the decoding start sample. The random access reproduction control unit can then start decoding the mesh data from the sample specified by the control information.

[0127] That is, therefore, the grid-to-grid samples (in Figure 8 In the case of the example of , the sample of POC2) can also be a synchronous sample (i.e., RAP). Therefore, the first information processing device and the second information processing device can suppress the reduction of random accessibility while suppressing the reduction of coding efficiency. For example, the first information processing device and the second information processing device can suppress the reduction of viewing experience quality of special effects reproduction (e.g., jump reproduction, fast forward, rewind, etc.) while suppressing the reduction of coding efficiency. In particular, in the case of method 1-1, it is possible to achieve the following: Figure 8 In the case of the example, the sample of POC4 is displayed normally. That is, image integrity can be prioritized.

[0128] In addition, the "control information specifying the decoding start sample" included in the random access metadata can have any specifications. For example, in the first information processing device and the second information processing device, the control information can specify the decoding start sample by the difference from the POC of the current sample.

[0129] For example, Figure 7 As shown, the parameter "roll_distance" can be defined in MeshRollRecoveryEntry. The parameter "roll_distance" indicates the difference in POC between the current sample and the designated sample. Here, the current sample indicates the sample to be processed corresponding to the random access metadata. In addition, the designated sample indicates the sample specified by the control information. In the case of method 1-1, the designated sample indicates the decoding start sample.

[0130] Note that the difference value (the difference in POC between the current sample and the decoding start sample) can be a value with a positive or negative sign, or can be an absolute value. For example, in the first information processing device and the second information processing device, the "difference value from the POC of the current sample" in the "control information for specifying the decoding start sample" described above can be a value with a positive or negative sign, or can be an absolute value. Note that when the difference value is indicated by a positive or negative sign, it is indicated by a negative value. Furthermore, when the difference value is indicated by an absolute value, control is performed under the assumption that the specified sample is a sample before the current sample (a sample with a smaller POC).

[0131] <Method 1-2>

[0132] In the case of applying method 1, for example, as from Figure 5 As shown in the third row from the top of the table in , random access metadata including control information specifying samples that are normally decoded can be stored in the file (method 1-2). In this specification, the "normally decoded samples" are also referred to as normal decoding samples.

[0133] For example, in the first information processing device, the random access metadata may include control information that specifies samples that are decoded normally.

[0134] For example, Figure 9 As shown, control information specifying a normally decoded sample that is normally decoded in the case of starting decoding from a sample of POC4 may be included as random access metadata corresponding to the sample of POC4. Figure 9 In the case of the example of , with respect to subgrid #1, the subsample of POC4 is an intra-grid subsample. Furthermore, with respect to the sample following POC4 of subgrid #2 (the sample with a larger POC), the subsample of POC6 is the intra-grid subsample closest to POC4. Therefore, in the case of starting decoding from the sample of POC4, decoding can be performed normally from the sample of POC6. That is, all subsamples can be decoded normally from the sample of POC6. Control information ("rolling" a positive distance) for designating such a sample of POC6 as a normally decoded sample can be included in the random access metadata corresponding to the sample of POC4.

[0135] In this case, the reproduction device (second information processing device) can start decoding from the current sample. For example, in the second information processing device, the random access metadata may include control information that specifies the sample to be decoded normally. The random access reproduction control unit can then start decoding the mesh data from the current sample.

[0136] That is, therefore, the grid-to-grid samples (in Figure 9 In the case of the example of , the sample of POC4) can also be a synchronous sample (i.e., RAP). Therefore, the first information processing device and the second information processing device can suppress the reduction of random accessibility while suppressing the reduction of coding efficiency. For example, the first information processing device and the second information processing device can suppress the reduction of viewing experience quality of special effects reproduction (e.g., jump reproduction, fast forward, rewind, etc.) while suppressing the reduction of coding efficiency. In particular, in the case of method 1-2, it is possible to Figure 8 In the case of the example, the sample of POC4) starts decoding and displaying. Although it is not until the normal decoding sample (in Figure 9 In the example of , normal decoding and display can only be performed until the sample of POC6, but since there is no need to perform decoding from the sample before the current sample, decoding and display can be started at a higher speed. In other words, low latency can be prioritized.

[0137] Note that the "control information specifying the sample to be decoded normally" included in the random access metadata may have any format. For example, in the first information processing device and the second information processing device, the control information may specify the sample to be decoded normally based on the difference from the POC of the current sample.

[0138] For example, Figure 7 As shown, the parameter "roll_distance" can be defined in MeshRollRecoveryEntry. The parameter "roll_distance" indicates the difference in POC between the current sample and the specified sample. In the case of method 1-2, the specified sample indicates a normal decoding sample.

[0139] Note that the difference value (the difference in POC between the current sample and the normally decoded sample) can be a value with a positive or negative sign, or can be an absolute value. For example, in the first information processing device and the second information processing device, the "difference from the POC of the current sample" in the "control information specifying the normally decoded sample" described above can be a value with a positive or negative sign, or can be an absolute value. Note that when the difference value is indicated by a positive or negative sign, it is indicated by a positive value. Furthermore, when the difference value is indicated by an absolute value, control is performed under the assumption that the designated sample is a sample after the current sample (the sample with a larger POC).

[0140] <Methods 1-3>

[0141] In the case of applying method 1, for example, Figure 5 As shown in the fourth row from the top of the table in , random access metadata including control information specifying a decoding start sample and control information specifying a sample to be decoded normally may be stored in the file (methods 1-3).

[0142] For example, in the first information processing apparatus, the random access metadata may include first control information that specifies a decoding start sample and second control information that specifies a sample to be normally decoded.

[0143] For example, Figure 10 As shown, the random access metadata corresponding to the sample of POC4 may include both control information specifying a decoding start sample for normal decoding of the POC4 sample and control information specifying a normal decoding sample that is normally decoded when decoding starts from the sample of POC4. Figure 10 In the example case, control information designating the sample of POC2 as the decoding start sample ("rolling" negative distance) and control information designating the sample of POC6 as the normal decoding sample ("rolling" positive distance) are included in the random access metadata corresponding to the sample of POC4.

[0144] In this case, the reproduction device (second information processing device) can select whether to start decoding from the decoding start sample specified by the control information or from the current sample. For example, in the second information processing device, the random access metadata may include first control information specifying the decoding start sample and second control information specifying the sample to be decoded normally. The random access reproduction control unit can then select whether to control random access based on the first control information or the second control information.

[0145] Note that such selection can be performed based on any information such as the operating mode of the reproduction device (second information processing device) itself, the load status, random access metadata, etc. For example, the reproduction device (second information processing device) can control random access by using control information specified by a user, an application, etc. In addition, when the current load is greater than a predetermined standard, the reproduction device (second information processing device) can prioritize low latency, and in other cases can prioritize image integrity. In addition, the reproduction device (second information processing device) can control random access based on the second control information when the normal decoding sample is closer to the current sample than the decoding start sample, and can control random access based on the first control information when the normal decoding sample is not closer to the current sample than the decoding start sample. Of course, other selection methods can be used.

[0146] That is, the grid samples (in Figure 10 In the case of the example of , the sample of POC2 or the sample of POC4) can also be a synchronous sample (i.e., RAP). Therefore, the first information processing device and the second information processing device can suppress the reduction of random accessibility while suppressing the reduction of coding efficiency. For example, the first information processing device and the second information processing device can suppress the reduction of viewing experience quality of special effects reproduction (e.g., jump reproduction, fast forward, rewind, etc.) while suppressing the reduction of coding efficiency. In particular, in the case of methods 1-3, it is possible to achieve the following: Figure 10 In the case of the example, the normal decoding and display of the POC4 sample) can be realized, and the normal decoding sample (in Figure 10 In the example of , the normal decoding and display of the POC6 sample) is performed. That is, image integrity can be prioritized, and low latency can be prioritized. A more appropriate control method can also be selected according to the situation.

[0147] Note that the first control information and the second control information included in the random access metadata can each have any specifications. For example, in the first information processing device and the second information processing device, the first control information included in the random access metadata can specify a decoding start sample by the difference from the picture order count (POC) of the current sample. Furthermore, the second control information included in the random access metadata can specify a sample that is normally decoded by the difference from the POC of the current sample.

[0148] For example, Figure 11 As shown, the parameters "negative_roll_distance" and "positive_roll_distance" can be defined in MeshRollRecoveryEntry. The parameter "negative_roll_distance" indicates the number of samples that need to be pre-decoded before the current sample in order to correctly decode the samples belonging to the sample group. That is, the parameter "negative_roll_distance" indicates the difference in POC between the current sample and the decoding start sample. In addition, the parameter "positive_roll_distance" indicates the number of samples after the current sample required to correctly complete the decoding of the sample when decoding starts from the sample belonging to the sample group. That is, the parameter "positive_roll_distance" indicates the difference in POC between the current sample and the normally decoded sample.

[0149] Note that these differences (the difference in POC between the current sample and the decoding start sample, and the difference in POC between the current sample and the normally decoded sample) can be values ​​with positive or negative signs. For example, in the first information processing device and the second information processing device, the "difference from the POC of the current sample" in the above-mentioned first control information and the second control information can be values ​​with positive or negative signs. Note that in this case, the difference in POC in the first control information (the difference in POC between the current sample and the decoding start sample) is indicated by a negative value, and the difference in POC in the second control information (the difference in POC between the current sample and the normally decoded sample) is indicated by a positive value. Of course, these differences can be indicated by absolute values. In this case, control is performed under the assumption that the specified sample corresponding to the first control information is the sample before the current sample (the sample with a smaller POC), and the specified sample corresponding to the second control information is the sample after the current sample (the sample with a larger POC).

[0150] <Methods 1-4>

[0151] In the case of applying method 1, for example, Figure 5 As shown in the fifth row from the top of the table in [ ], a subgrid can be a subsample, and random access metadata for each subsample can be stored in the file (methods 1-4). That is, the content file can manage the bitstream of each subgrid as a subsample in a sample. Then, random access metadata can be set for the subsample.

[0152] For example, in the first information processing device, the content file may manage a subgrid of the grid data as a subsample. The random access metadata generation unit may generate random access metadata corresponding to the subsample. The storage unit may store the random access metadata corresponding to the subsample in the content file.

[0153] During reproduction, the bitstream stored in such a content file can be independently decoded for each subsample. For example, in the second information processing device, the content file can manage subgrids of grid data as subsamples and store random access metadata corresponding to the subsamples. The random access reproduction control unit can then control random access to the subsamples based on the random access metadata corresponding to the subsamples. Furthermore, the reproduction processing unit can start reproduction of the grid data for each subsample through random access.

[0154] Therefore, the second information processing device can easily (without parsing the bitstream) control random access reproduction for each subsample. In other words, the first information processing device can easily (without parsing the bitstream) generate a content file that can control random access reproduction for each subsample. In addition, by controlling random access for each subsample, compared with the case of controlling random access reproduction for each sample, the decoding of unnecessary subsamples can be further reduced, and the increase in the load of the decoding process can be suppressed. In addition, a wider range of samples can be randomly accessed. That is, while suppressing the reduction in coding efficiency, the reduction in random accessibility can be suppressed.

[0155] The random access metadata corresponding to the subsample can be stored in any manner and at any location in the content file. For example, the random access metadata can be stored in a subsample information box (SubSampleInformationBox) of the content file. For example, the random access metadata can be stored as codec-specific parameters (codec_specific_parameters) in a subsample information box of the content file. For example, in a first information processing device, the storage unit can store the random access metadata in a subsample information box of the content file. Furthermore, in a second information processing device, the random access metadata can be stored in a subsample information box of the content file. That is, the second information processing device can refer to the random access metadata stored in the subsample information box of the content file during random access reproduction.

[0156] <Method 1-4-1>

[0157] In the case of applying methods 1 to 4, for example, as from Figure 5 As shown in the sixth row from the top of the table in , the submesh coding type information (submesh_coding_type) can also be stored in the file as random access metadata for each subsample (method 1-4-1).

[0158] For example, in the first information processing apparatus and the second information processing apparatus, the random access metadata may include information indicating an encoding type of a sub-grid corresponding to a sub-sample. In addition, in the second information processing apparatus, the random access reproduction control unit may control random access to the sub-sample based on the encoding type.

[0159] The coding type of the sub-grid (sub-sample) may include, for example, intra-frame coding (I), inter-frame coding (P), skip coding (S), etc. Figure 12 As shown, for each sub-sample, the encoding type of the sub-grid corresponding to the sub-sample can be indicated as random access metadata. Figure 12 , “I”, “P”, or “S” appended to each subsample indicates an encoding type of a subgrid corresponding to the subsample, which is indicated as random access metadata.

[0160] Since the encoding type of the sub-mesh corresponding to each sub-sample can be known by referring to the random access metadata in this manner, random access reproduction can be controlled according to the encoding type.

[0161] exist Figure 12 In the example, for example, the sample of POC4 is set as the current sample. By referring to the random access metadata, it can be seen that, for subgrid #1, the encoding type of the current subsample to be processed (i.e., the subsample of POC4) is "I (Intra-frame coding)". In addition, it can be seen that, for subgrid #2, the encoding type of the current subsample is "P (Inter-frame coding)". In addition, it can be seen that the subsample of POC2 is the subsample with the encoding type of "I (Intra-frame coding)" that is before POC4 and closest to POC4. Similarly, it can be seen that the subsample of POC6 is the subsample with the encoding type of "I (Intra-frame coding)" that is after POC4 and closest to POC4.

[0162] That is, if image integrity is prioritized, the reproduction device only needs to start decoding from the current subsample (subsample of POC4) for subgrid #1, and start decoding from the subsample of POC2 for subgrid #2. With this control, the current sample can be decoded and displayed normally. In addition, for example, decoding of unnecessary subsamples with a coding type of "I (Intra-frame Coding)" such as the subsamples of POC0 and POC3 corresponding to subgrid #1 and the subsample of POC0 corresponding to subgrid #2 can be omitted. In addition, for example, decoding of subsamples with a coding type of "P (Inter-frame Coding)" and undecoded reference destination subsamples (subsamples with a coding type of "I (Intra-frame Coding)") such as the subsample of POC2 corresponding to subgrid #1 and the subsample of POC1 corresponding to subgrid #2 can be omitted. Furthermore, for example, decoding of subsamples with a coding type of "S (Skip Coding)" such as the subsamples of POC1 corresponding to subgrid #1 and the subsamples of POC3 corresponding to subgrid #2 can be omitted. Thus, an increase in the decoding processing load can be suppressed. Furthermore, such control can be performed based on random access metadata. In other words, such control can be easily implemented without parsing the bitstream.

[0163] On the other hand, if low latency is prioritized, the reproduction device only needs to start decoding from the current sample. In this case, subgrid #1 can be decoded normally from the subsample of POC4. In addition, subgrid #2 can be decoded normally from the subsample of POC6. Note that in this case, for example, decoding of subsamples with a coding type of "P (Inter-frame coding)", undecoded reference destination subsamples (subsamples with a coding type of "I (Intra-frame coding)"), such as the subsamples of POC4 and POC5 corresponding to subgrid #2, can be omitted. In addition, decoding of subsamples with a coding type of "S (Skip Coding)" can be omitted. Therefore, the increase in the load of the decoding process can be suppressed. In addition, such control can be performed based on random access metadata. That is, such control can be easily implemented without parsing the bitstream.

[0164] Note that in this case, random access metadata can have any size. For example, Figure 13 As shown, "submesh_coding_type" can be defined as a codec-specific parameter in the subsample information box. "submesh_coding_type" is a parameter indicating the coding type of the submesh. For example, in the case of submesh_coding_type=0, it can be indicated that the coding type of the submesh is "P (inter-frame coding)". In addition, in the case of submesh_coding_type=1, it can be indicated that the coding type of the submesh is "I (intra-frame coding)". In addition, in the case of submesh_coding_type=2, it can be indicated that the coding type of the submesh is "S (skip coding)". Of course, this value is just an example and is not limited to this example. In addition, the coding type indicated by this parameter is also optional and is not limited to this example.

[0165] <Method 1-4-2>

[0166] In the case of applying methods 1 to 4, for example, as from Figure 5 As shown in the seventh row from the top of the table in [ ], random access metadata including control information specifying a decoding start subsample can be stored in the file (method 1-4-2). In other words, the control of method 1-1 can be performed for each subsample. In this specification, this "decoding start subsample" is also referred to as a decoding start subsample.

[0167] For example, in the first information processing device, the random access metadata may include control information specifying a decoding start subsample. In this case, the reproduction device (the second information processing device) may start decoding from the decoding start sample specified by the control information. For example, in the second information processing device, the random access metadata may include control information specifying a decoding start subsample. The random access reproduction control unit may then decode the mesh data starting from the subsample specified by the control information.

[0168] Note that the "control information specifying the decoding start subsample" included in the random access metadata can have any format. For example, in the first information processing device and the second information processing device, the control information can specify the decoding start subsample by the difference from the POC of the current subsample.

[0169] Note that the difference value (the difference in POC between the current subsample and the decoding start subsample) can be a positive or negative value, or an absolute value. For example, in the first information processing device and the second information processing device, the "difference value from the POC of the current subsample" in the "control information specifying the decoding start subsample" can be a positive or negative value, or an absolute value. Note that when the difference value is indicated by a positive or negative value, it is indicated by a negative value. Furthermore, when the difference value is indicated by an absolute value, control is performed under the assumption that the decoding start subsample is the subsample before the current subsample (the subsample with a smaller POC).

[0170] For example, Figure 14 As shown, control information specifying a decoding start subsample for normal decoding of the subsample may be included as random access metadata corresponding to each subsample. Figure 14 , the number in the square attached to the left side of each subsample indicates the "difference value (absolute value) of POC between the current subsample and the decoding start subsample". For example, the subsample of POC5 corresponding to subgrid #1 is assigned "1" as the difference value (absolute value). That is, the subsample of POC4 corresponding to subgrid #1 is designated as the decoding start subsample. In addition, the subsample of POC5 corresponding to subgrid #2 is assigned "3" as the difference value (absolute value). That is, the subsample of POC2 corresponding to subgrid #2 is designated as the decoding start subsample. Therefore, in this case, by starting decoding from the subsample of POC4 for subgrid #1 and starting decoding from the subsample of POC2 for subgrid #2, the sample of POC5 can be decoded normally. Similarly, for other samples, the decoding start subsample used for normal decoding of the subsample is designated as the random access metadata.

[0171] Note that in Figure 14 In the example of , as the decoding start subsample for normal decoding of the subsample corresponding to subgrid #1 of POC5, the subsample of POC4 is closest to POC5. Similarly, as the decoding start subsample for normal decoding of the subsample corresponding to subgrid #2 of POC5, the subsample of POC2 is closest to POC5. As described above, the random access metadata may include control information that specifies the subsample closest to the subsample corresponding to the random access metadata among the decoding start subsamples that meet the condition (i.e., the decoding start subsample used for normal decoding of the subsample corresponding to the random access metadata).

[0172] For example, Figure 15 As shown, random access metadata can be stored in the subsample information box. For example, the random access metadata can be set as a codec-specific parameter in the subsample information box. For example, the random access metadata can be defined in the codec-specific parameter "negative_roll_distance" defined in the subsample information box. The parameter "negative_roll_distance" indicates the number of subsamples included in the sample before the current subsample, including subsamples that need to be decoded in advance to correctly decode the subsample. In other words, the parameter "negative_roll_distance" indicates the difference in the POC between the current subsample and the decoding start subsample.

[0173] By controlling random access in this manner, the first information processing device and the second information processing device can obtain an effect similar to the case of applying method 1-1. In addition, by controlling random access to each subsample, the decoding of unnecessary subsamples can be further reduced compared to the case of controlling random access reproduction of each sample. For example, the decoding of unnecessary subsamples with a coding type of "I (intra-frame coding)" can be omitted, or the decoding of subsamples with a coding type of "P (inter-frame coding)" for which the reference target subsample (subsample with a coding type of "I (intra-frame coding)") has not yet been decoded can be omitted. Therefore, the increase in the load of the decoding process can be suppressed. In addition, such control can be performed based on random access metadata. That is, such control can be easily implemented without the need to parse the bitstream.

[0174] <Method 1-4-3>

[0175] In the case of applying methods 1 to 4, for example, Figure 5 As shown in the eighth row from the top of the table, random access metadata including control information specifying subsamples that are decoded normally can be stored in a file (method 1-4-3). In other words, the control of method 1-2 can be performed for each subsample. In this specification, this "subsample that is decoded normally" is also referred to as a normally decoded subsample.

[0176] For example, in the first information processing device, the random access metadata may include control information specifying a subsample to be decoded normally. In this case, the reproduction device (the second information processing device) may start decoding from the current subsample. For example, in the second information processing device, the random access metadata may include control information specifying a subsample to be decoded normally. The random access reproduction control unit may then start decoding the grid data from the current subsample.

[0177] Note that the "control information specifying the subsample to be decoded normally" included in the random access metadata may have any specifications. For example, in the first information processing device and the second information processing device, the control information may specify the subsample to be decoded normally based on the difference from the picture order count (POC) of the current subsample.

[0178] Note that the difference value (the difference in POC between the current subsample and the normally decoded subsample) can be a value with a positive or negative sign, or an absolute value. For example, in the first information processing device and the second information processing device, the "difference from the POC of the current subsample" in the "control information specifying the normally decoded subsample" can be a value with a positive or negative sign, or an absolute value. Note that when the difference value is indicated by a value with a positive or negative sign, the difference value is indicated by a positive value. Furthermore, when the difference value is indicated by an absolute value, control is performed based on the assumption that the normally decoded subsample is the subsample after the current subsample (the subsample with a larger POC).

[0179] For example, Figure 16 As shown, control information specifying a subsample to be normally decoded in the case of starting decoding from a subsample may be included as random access metadata corresponding to each subsample. Figure 16 , the number in the square attached to the left side of each subsample indicates the "difference value (absolute value) of POC between the current subsample and the normally decoded subsample". For example, the subsample corresponding to subgrid #1 of POC5 is assigned "2" as the difference value (absolute value). That is, the subsample corresponding to subgrid #1 of POC7 is designated as the normally decoded subsample. In addition, the subsample corresponding to subgrid #2 of POC5 is assigned "1" as the difference value (absolute value). That is, the subsample corresponding to subgrid #2 of POC6 is designated as the normally decoded sample. Therefore, in this case, by starting decoding from the sample of POC5, subgrid #1 can be normally decoded according to the subsample of POC7, and subgrid #2 can be normally decoded according to the subsample of POC6. Similarly, for other samples, the normally decoded subsample is designated as the random access metadata in the case of starting decoding from the subsample.

[0180] For example, Figure 17 As shown, random access metadata can be stored in the subsample information box. For example, the random access metadata can be set as a codec-specific parameter in the subsample information box. For example, the random access metadata can include the codec-specific parameter "positive_roll_distance" defined in the subsample information box. The parameter "positive_roll_distance" indicates the number of subsamples included in the sample after the current sample, including the subsamples required to correctly complete the decoding of the subsample when decoding starts from the subsample. In other words, the parameter "positive_roll_distance" indicates the difference in the POC between the current subsample and the normally decoded subsample.

[0181] By controlling random access in this manner, the first information processing device and the second information processing device can obtain an effect similar to the case of applying method 1-2. In addition, by controlling random access to each subsample, the decoding of unnecessary subsamples can be further reduced compared to the case of controlling random access reproduction of each sample. For example, the decoding of unnecessary subsamples with a coding type of "I (intra-frame coding)" can be omitted, or the decoding of subsamples with a coding type of "P (inter-frame coding)" for which the reference target subsample (subsample with a coding type of "I (intra-frame coding)") has not yet been decoded can be omitted. Therefore, the increase in the load of the decoding process can be suppressed. In addition, such control can be performed based on random access metadata. That is, such control can be easily implemented without the need to parse the bitstream.

[0182] <Method 1-4-4>

[0183] In the case of applying methods 1 to 4, for example, Figure 5 As shown in the bottom row of the table, random access metadata including control information for specifying a subsample to start decoding and control information for specifying a subsample to be decoded normally can be stored in a file (method 1-4-4). In other words, the control of method 1-3 can be performed for each subsample.

[0184] For example, in the first information processing device, the random access metadata may include first control information specifying a decoding start subsample and second control information specifying a subsample to be decoded normally. In this case, the reproduction device (the second information processing device) can choose whether to start decoding from the decoding start subsample specified by the random access metadata or from the current subsample. For example, in the second information processing device, the random access metadata may include first control information specifying a decoding start sample and second control information specifying a sample to be decoded normally. The random access reproduction control unit can then choose whether to control random access based on the first control information or the second control information.

[0185] Note that such selection can be performed based on any information such as the operating mode, load status, random access metadata, etc. of the reproduction device (second information processing device) itself. For example, the reproduction device (second information processing device) can control random access by using control information specified by a user, an application, etc. In addition, in the case where the current load is greater than a predetermined standard, the reproduction device (second information processing device) can prioritize low latency, and in other cases can prioritize image integrity. In addition, in the case where the normal decoding subsample is closer to the current subsample than the decoding start subsample, the reproduction device (second information processing device) can control random access based on the second control information, and in the case where the normal decoding subsample is not closer to the current subsample than the decoding start subsample, the reproduction device (second information processing device) can control random access based on the first control information. Of course, other selection methods can also be used.

[0186] Note that the first control information and the second control information included in the random access metadata can each have any format. For example, in the first information processing device and the second information processing device, the first control information included in the random access metadata can specify a subsample to start decoding based on the difference between the POC and the current subsample. Furthermore, the second control information included in the random access metadata can specify a subsample to be decoded normally based on the difference between the POC and the current subsample.

[0187] Note that these differences (the difference in POC between the current subsample and the decoding start subsample, and the difference in POC between the current subsample and the normally decoded subsample) can have positive or negative signs. For example, in the first information processing device and the second information processing device, the "difference from the POC of the current subsample" in the first and second control information can have positive or negative signs. Note that in this case, the difference in POC between the current subsample and the decoding start subsample in the first control information is indicated by a negative value, while the difference in POC between the current subsample and the decoding start subsample in the second control information is indicated by a positive value. Of course, these differences can also be indicated by absolute values. In this case, control is performed based on the assumption that the decoding start subsample corresponding to the first control information is the subsample before the current subsample (the subsample with a smaller POC), and the normally decoded subsample corresponding to the second control information is the subsample after the current subsample (the subsample with a larger POC).

[0188] For example, Figure 18 As shown, the random access metadata corresponding to each subsample may include first control information specifying a decoding start subsample for normal decoding of the subsample, and second control information specifying a subsample to be normally decoded when decoding starts from the subsample. Figure 18 , the number in the square attached to the left of each subsample indicates "the difference (absolute value) of POC between the current subsample and the decoding start subsample / the difference (absolute value) of POC between the current subsample and the normally decoded subsample".

[0189] For example, the subsample corresponding to subgrid #1 of POC5 is assigned "1" as the POC difference between the current subsample and the decoding start subsample, and the subsample is assigned "2" as the POC difference between the current subsample and the normal decoding subsample. That is, the subsample corresponding to subgrid #1 of POC4 is designated as the decoding start subsample, and the subsample corresponding to subgrid #1 of POC7 is designated as the normal decoding subsample.

[0190] Furthermore, the subsample corresponding to subgrid #2 of POC5 is assigned "3" as the POC difference between the current subsample and the decoding start subsample, and the subsample is assigned "1" as the POC difference between the current subsample and the normal decoding subsample. That is, the subsample corresponding to subgrid #2 of POC2 is designated as the decoding start subsample, and the subsample corresponding to subgrid #2 of POC6 is designated as the normal decoding subsample.

[0191] Therefore, in this case, by starting decoding from the subsamples of POC4 for subgrid #1 and starting decoding from the subsamples of POC2 for subgrid #2, samples of POC5 can be decoded normally. In addition, by starting decoding from the samples of POC5, subgrid #1 can be decoded normally based on the subsamples of POC7, and subgrid #2 can be decoded normally based on the subsamples of POC6.

[0192] Similarly, for other samples, a decoding start subsample for normal decoding of a subsample and a normally decoded subsample that is normally decoded in the case of starting decoding from the subsample are designated as random access metadata.

[0193] For example, Figure 19 As shown, the random access metadata can be stored in the subsample information box. For example, the random access metadata can be set as codec-specific parameters in the subsample information box. For example, the random access metadata may include codec-specific parameters "negative_roll_distance" and "positive_roll_distance" defined in the subsample information box. The parameter "negative_roll_distance" indicates the number of subsamples included in the sample before the current subsample, including subsamples that need to be decoded in advance to correctly decode the subsample. That is, the parameter "negative_roll_distance" indicates the difference in POC between the current subsample and the decoding start subsample. In addition, the parameter "positive_roll_distance" indicates the number of subsamples included in the sample after the current sample, including subsamples necessary to correctly complete the decoding of the subsample when decoding starts from the subsample. That is, the parameter "positive_roll_distance" indicates the difference in POC between the current subsample and the normally decoded subsample.

[0194] By controlling random access in this manner, the first information processing device and the second information processing device can obtain an effect similar to the case of applying methods 1-3. In addition, by controlling random access to each subsample, the decoding of unnecessary subsamples can be further reduced compared to the case of controlling random access reproduction of each sample. For example, the decoding of unnecessary subsamples with a coding type of "I (intra-frame coding)" can be omitted, or the decoding of subsamples with a coding type of "P (inter-frame coding)" for which the reference target subsample (subsample with a coding type of "I (intra-frame coding)") has not yet been decoded can be omitted. Therefore, the increase in the load of the decoding process can be suppressed. In addition, such control can be performed based on random access metadata. That is, such control can be easily implemented without the need to parse the bitstream.

[0195] <Other application examples>

[0196] The present technology is not limited to V-DMC and can be applied to any encoding scheme. In addition, the present technology is not limited to the base mesh and can be applied to any mesh data. For example, the present technology can be applied to a file container that stores a bit stream obtained by encoding the original mesh using an encoding scheme that encodes the mesh without decimation (simplification), such as Geometry-Based Point Cloud Compression (G-PCC). Note that mesh data can be generated by converting 3D data other than mesh data, such as point cloud data.

[0197] In addition, the file container storing the bitstream is not limited to the file container compliant with ISOBMFF, and may have any specifications. For example, in the first information processing apparatus and the second information processing apparatus, the content file may be a Matroska media container. Figure 20 is a diagram showing a main configuration example of a Matroska media container. Of course, the file can be a file in another format. Furthermore, the track configuration of the file container can be any configuration. For example, a multi-track consisting of multiple tracks can be used, or a single track consisting of one track can be used.

[0198] <Combination>

[0199] As long as there is no contradiction, each of the above methods can be applied in combination with any other method. Three or more methods can be applied in combination. For example, any two or more of methods 1-1 to 1-4 can be applied in combination. In addition, the combinable methods can include not only Figure 5 Those shown as "methods" in the table of FIG. 3 may also include all of the above elements. In addition, each of the above methods may be applied in combination with other methods not described above.

[0200] Note that in this specification, as long as there is no contradiction, the description of a superordinate method also applies to the subordinate methods belonging to that method. For example, when describing "Method 1 can be applied", any one or more of Methods 1-1 to 1-4 can be applied. In addition, any one or more of Methods 1-4-1 to 1-4-4 can be applied.

[0201] <4. First embodiment>

[0202] <File Generation Device>

[0203] The present technology described above can be applied to any device. Figure 21 : is a block diagram showing a configuration example of a file generating apparatus to which one aspect of the information processing apparatus of the present technology is applied. Figure 21 The file generating apparatus 300 shown in FIG. 1 is an apparatus that generates a content file by converting mesh data into V-DMC data, encoding the data, and storing the data in a file container such as ISOBMFF.

[0204] Note that in Figure 21 , showing the main processing unit, data flow, etc., and Figure 21 That is, in the file generating device 300, there may be Figure 21 Processing units may be shown as boxes, or there may be processing units not shown in Figure 21 The process or data flow is shown as arrows or the like.

[0205] like Figure 21 As shown, the file generating apparatus 300 includes a V-DMC data generating unit 311 , a V-DMC encoding unit 312 , a content file generating unit 313 , and a random access metadata generating unit 314 .

[0206] The V-DMC data generation unit 311 performs processing related to the generation of V-DMC data. For example, the V-DMC data generation unit 311 may acquire mesh data (original mesh) input from outside the file generation device 300. The original mesh may include not only geometry (information about vertex positions, connections, etc.) but also attributes (e.g., textures applied to faces).

[0207] The V-DMC data generation unit 311 can convert the original mesh to generate V-DMC data (base mesh, displacement vectors, attributes (texture), and atlas information). Therefore, the V-DMC data generation unit 311 can also be referred to as a conversion unit. For example, the V-DMC data generation unit 311 can generate the base mesh by extracting the geometry of the original mesh. Extraction is the process of reducing the number of faces (polygons) by extracting some vertices (and connections) of the mesh. Therefore, the V-DMC data generation unit 311 can also be referred to as an extraction unit that performs extraction. In addition, the V-DMC data generation unit 311 can generate displacement vectors corresponding to the base mesh by subdividing the base mesh. Therefore, the V-DMC data generation unit 311 can also be referred to as a displacement vector generation unit that generates displacement vectors. In addition, the V-DMC data generation unit 311 can generate atlas information corresponding to the base mesh, displacement vectors, and attributes (texture). Therefore, the V-DMC data generation unit 311 can also be referred to as an atlas information generation unit that generates atlas information. The V-DMC data generating unit 311 may supply the generated V-DMC data to the V-DMC encoding unit 312. Therefore, the V-DMC data generating unit 311 may also be referred to as a V-DMC data supplying unit that supplies V-DMC data.

[0208] The V-DMC encoding unit 312 performs processing related to encoding V-DMC data. For example, the V-DMC encoding unit 312 may obtain V-DMC data supplied from the V-DMC data generation unit 311. Furthermore, the V-DMC encoding unit 312 may encode the obtained V-DMC data. Specifically, the V-DMC encoding unit 312 may encode each of the atlas information, base grid, displacement vectors, and attributes included in the V-DMC data, generating encoded data for the atlas information, base grid, displacement vectors, and attributes. The V-DMC encoding unit 312 may arrange the generated encoded data for the atlas information, base grid, displacement vectors, and attributes into separate substreams and combine (multiplex) the substreams to generate a single V-DMC bitstream. Therefore, the V-DMC encoding unit 312 may also be referred to as a V-DMC bitstream generation unit. The V-DMC encoding unit 312 may supply the generated V-DMC bitstream to the content file generation unit 313.

[0209] The content file generation unit 313 performs processing related to content file generation. For example, the content file generation unit 313 may acquire the V-DMC bitstream supplied by the V-DMC encoding unit 312. Furthermore, the content file generation unit 313 may generate a content file, which serves as a file container for storing the bitstream. The content file may be, for example, a file compliant with ISOBMFF. The content file generation unit 313 may store the acquired V-DMC bitstream in the content file. Therefore, the content file generation unit 313 may also be referred to as a storage unit that stores the V-DMC bitstream in the content file.

[0210] Furthermore, the content file generating unit 313 may supply the V-DMC bitstream to the random access metadata generating unit 314. The content file generating unit 313 may supply the random access metadata generating unit 314 with a content file storing the V-DMC bitstream instead of the V-DMC bitstream.

[0211] Furthermore, the content file generation unit 313 may acquire random access metadata supplied from the random access metadata generation unit 314. The content file generation unit 313 may store the acquired random access metadata in a content file storing a V-DMC bitstream corresponding to the random access metadata. Therefore, the content file generation unit 313 may also be referred to as a storage unit that stores random access metadata in a content file.

[0212] The content file generation unit 313 can output the generated content file to the outside of the file generation device 300 (eg, a distribution server, a reproduction device, etc.) Therefore, the content file generation unit 313 can also be called a content file supply unit (providing unit).

[0213] Figure 22 312 is a block diagram showing a main configuration example of the V-DMC encoding unit 312. Figure 22 In the figure, the main parts of the processing unit, data flow, etc. are shown, and Figure 22 That is, in the V-DMC encoding unit 312, there may be Figure 22 Processing units may be shown as boxes, or there may be processing units not shown in Figure 22 The process or data flow is shown as arrows or the like.

[0214] like Figure 22 As shown, the V-DMC encoding unit 312 includes an atlas information encoding unit 351, a base grid encoding unit 352, a displacement vector correction unit 353, a displacement video encoding unit 354, a grid reconstruction unit 355, an attribute map conversion unit 356, an attribute video encoding unit 357 and a multiplexing unit 358.

[0215] The atlas information encoding unit 351 performs processing related to encoding atlas information. For example, the atlas information encoding unit 351 may acquire atlas information supplied from the V-DMC data generating unit 311. Furthermore, the atlas information encoding unit 351 may encode the acquired atlas information using a predetermined encoding scheme to generate encoded data of the atlas information. The atlas information encoding unit 351 may supply the generated encoded data of the atlas information to the multiplexing unit 358.

[0216] The base grid encoding unit 352 performs processing related to encoding the base grid. For example, the base grid encoding unit 352 may obtain the base grid supplied from the V-DMC data generation unit 311. Furthermore, the base grid encoding unit 352 may quantize the obtained base grid, encode the quantized base grid using a predetermined encoding scheme (e.g., Draco), and generate encoded data of the base grid. In this case, the base grid encoding unit 352 may encode the base grid based on the atlas information. The base grid encoding unit 352 may supply the generated encoded data of the base grid to the displacement vector correction unit 353. Furthermore, the base grid encoding unit 352 may supply the generated encoded data of the base grid to the multiplexing unit 358.

[0217] The displacement vector correction unit 353 performs processing related to displacement vector correction. For example, the displacement vector correction unit 353 may obtain the base grid and displacement vectors supplied from the V-DMC data generation unit 311. Furthermore, the displacement vector correction unit 353 may obtain the encoded data of the base grid supplied from the base grid encoding unit 352. The displacement vector correction unit 353 may correct the displacement vectors based on this information. For example, the displacement vector correction unit 353 may decode and inverse quantize the obtained encoded data of the base grid and restore the base grid (or generate a restored base grid). The displacement vector correction unit 353 may subdivide each of the pre-encoding base grid and the restored base grid. The displacement vector correction unit 353 may compare the subdivided pre-encoding base grid with the subdivided restored base grid to obtain coding distortion. The displacement vector correction unit 353 may correct the displacement vectors based on the coding distortion. The displacement vector correction unit 353 may supply the corrected displacement vectors to the displacement video encoding unit 354. Furthermore, the displacement vector correction unit 353 may supply the subdivided restored base grid to the grid reconstruction unit 355.

[0218] The displacement video encoding unit 354 performs processing related to encoding displacement video. Displacement video is a moving image, in which a displacement map is a frame image, and the displacement map is a two-dimensional area in which displacement vectors are packed. For example, the displacement video encoding unit 354 can obtain the displacement vectors supplied from the displacement vector correction unit 353. Furthermore, the displacement video encoding unit 354 can generate a displacement map by performing a wavelet transform on the displacement vectors, quantizing the displacement vectors, and packing the displacement vectors into a two-dimensional area. Furthermore, the displacement video encoding unit 354 can generate a displacement video using the displacement map as a frame image. Furthermore, the displacement video encoding unit 354 can encode the generated displacement video using a predetermined encoding scheme for 2D moving images to generate coded data for the displacement video. The displacement video encoding unit 354 can supply the coded data for the generated displacement video to the multiplexing unit 358. Furthermore, the displacement video encoding unit 354 can decode the coded data for the generated displacement video and restore the displacement video (generating a restored displacement video). The displacement video encoding unit 354 can extract the displacement map from the restored displacement video. The displacement video encoding unit 354 may unpack the displacement vector from the displacement map, inverse quantize the displacement vector, and supply the inverse quantized displacement vector to the grid reconstruction unit 355.

[0219] The mesh reconstruction unit 355 performs processing related to mesh reconstruction. For example, the mesh reconstruction unit 355 may obtain the base mesh (subdivided restored base mesh) supplied from the displacement vector correction unit 353. Furthermore, the mesh reconstruction unit 355 may obtain the displacement vectors supplied from the displacement video encoding unit 354. The mesh reconstruction unit 355 may reconstruct a restored mesh using the base mesh and the displacement vectors. The mesh reconstruction unit 355 may supply the restored mesh to the attribute map conversion unit 356.

[0220] The attribute map conversion unit 356 performs processing related to attribute map conversion. For example, the attribute map conversion unit 356 may obtain the restored mesh supplied by the mesh reconstruction unit 355. Furthermore, the attribute map conversion unit 356 may obtain atlas information supplied by the V-DMC data generation unit 311. The attribute map conversion unit 356 may obtain the original mesh input to the file generation device 300. The attribute map conversion unit 356 may obtain the attribute map supplied by the V-DMC data generation unit 311. The attribute map conversion unit 356 may convert the obtained attribute map based on other information obtained. For example, the attribute map conversion unit 356 may convert the attribute map based on atlas information, the original mesh, etc., so that it corresponds to the restored mesh. In other words, the attribute map conversion unit 356 generates a converted attribute map. Therefore, the attribute map conversion unit 356 may also be referred to as an attribute map generation unit. The attribute map conversion unit 356 may supply the converted attribute map to the attribute video encoding unit 357.

[0221] The attribute video encoding unit 357 performs processing related to encoding the attribute video. For example, the attribute video encoding unit 357 may obtain the attribute map supplied from the attribute map conversion unit 356. Furthermore, the attribute video encoding unit 357 may use the obtained attribute map as a frame image to generate the attribute video. Furthermore, the attribute video encoding unit 357 may encode the generated attribute video using a predetermined encoding scheme for 2D moving images to generate encoded data for the attribute video. The attribute video encoding unit 357 may supply the generated encoded data for the attribute video to the multiplexing unit 358.

[0222] The multiplexing unit 358 performs processing related to multiplexing of coded data (substreams). For example, the multiplexing unit 358 can obtain the coded data of the atlas information supplied from the atlas information encoding unit 351. In addition, the multiplexing unit 358 can obtain the coded data of the base grid supplied from the base grid encoding unit 352. In addition, the multiplexing unit 358 can obtain the coded data of the displacement video supplied from the displacement video encoding unit 354. In addition, the multiplexing unit 358 can obtain the coded data of the attribute video supplied from the attribute video encoding unit 357. The multiplexing unit 358 can multiplex these coded data into substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 358 can also be called a bitstream generation unit. The multiplexing unit 358 can supply the generated V-DMC bitstream to the content file generation unit 313.

[0223] The file generation device 300 having the above configuration may be the first information processing device, and the various methods (present technology) described above may be applied in <3. Transmission and control of random access metadata>.

[0224] For example, in the file generation device 300 (first information processing device), the random access metadata generation unit 314 can generate random access metadata for performing random access on mesh data based on the current sample as an inter-mesh sample, and the content file generation unit 313 can store the random access metadata in the content file storing the mesh data.

[0225] Therefore, the file generation device 300 can achieve an effect similar to that described above in <3. Transmission and Control of Random Access Metadata>. That is, when the bitstream obtained by encoding the mesh data is stored in the file container, the file generation device 300 can suppress the reduction in random accessibility while suppressing the reduction in encoding efficiency.

[0226] <File Generation Processing Flow>

[0227] Next, we will refer to Figure 23 The flowchart of FIG. 1 describes an example of the flow of the file generation process executed by the file generation apparatus 300 .

[0228] In step S301 , when the file generation process starts, the V-DMC data generation unit 311 of the file generation apparatus 300 generates V-DMC data by extracting mesh data and the like.

[0229] In step S302 , the V-DMC encoding unit 312 performs a V-DMC data encoding process, and encodes the V-DMC data to generate a V-DMC bit stream.

[0230] In step S303, the content file generation unit 313 generates a content file and stores the V-DMC bitstream in the content file.

[0231] In step S304, the random access metadata generation unit 314 generates random access metadata for performing random access on the mesh data based on the current sample as an inter-mesh sample. In addition, the content file generation unit 313 stores the random access metadata in the content file storing the mesh data.

[0232] When the processing of step S304 ends, the file generation process ends.

[0233] <V-DMC Encoding Processing Flow>

[0234] Next, a flowchart of Figure 24 will be referred to describe an example of the V-DMC encoding processing flow executed in step S302 of Figure 23

[0235] In step S321, when the V-DMC encoding processing starts, the atlas information encoding unit 351 encodes the atlas information and generates encoded data of the atlas information.

[0236] In step S322, the base mesh encoding unit 352 encodes the base mesh and generates encoded data of the base mesh.

[0237] In step S323, the displacement vector correction unit 353 corrects the displacement vector.

[0238] In step S324, the displacement video encoding unit 354 encodes the displacement video having the displacement map with the corrected displacement vector packed as a frame image and generates encoded data of the displacement video.

[0239] In step S325, the mesh reconstruction unit 355 reconstructs the mesh (generates a restored mesh).

[0240] In step S326, the attribute map conversion unit 356 converts the attribute map.

[0241] In step S327, the attribute video encoding unit 357 encodes the attribute video having the attribute map as a frame image and generates encoded data of the attribute video.

[0242] In step S328, the multiplexing unit 358 multiplexes each of the encoded data of the atlas information, the encoded data of the bitstream, the encoded data of the displacement video, and the encoded data of the attribute video as substreams and generates a V-DMC bitstream.

[0243] ​When the processing in step S328 ends, the V-DMC encoding process ends, and the processing returns to Figure 23 .

[0244] By performing each of the processes described above, the file generation device 300 can achieve an effect similar to that described above in <3. Transmission and Control of Random Access Metadata>. That is, when the bitstream obtained by encoding the mesh data is stored in a file container, the file generation device 300 can suppress a decrease in random accessibility while suppressing a decrease in encoding efficiency.

[0245] <5. Second embodiment>

[0246] <Reproduction Device>

[0247] The present technology described above can be applied to any device. Figure 25 : is a block diagram showing a configuration example of a reproduction device as one aspect of an information processing device to which the present technology is applied. Figure 25 The reproduction device 400 shown is a reproduction device that performs a reproduction process for mesh data (V-DMC data). For example, the reproduction device 400 obtains the content file generated by the file generation device 300, decodes the V-DMC bitstream stored in the content file, and reproduces the V-DMC data. In other words, the reproduction device 400 reconstructs the mesh using the V-DMC data, performs rendering to generate a display image, and displays the display image.

[0248] Note that in Figure 25 , showing the main processing unit, data flow, etc., and Figure 25 That is, in the reproduction device 400, there may be Figure 25 Processing units may be shown as boxes, or there may be processing units not shown in Figure 25 The process or data flow is shown as arrows or the like.

[0249] like Figure 25 As shown, the reproduction apparatus 400 includes a content file acquisition unit 411, a reproduction processing unit 412, a random access operation reception unit 421, and a random access reproduction control unit 422. In addition, the reproduction processing unit 412 includes a V-DMC decoding unit 431, a mesh reconstruction unit 432, and a display processing unit 433.

[0250] The content file acquisition unit 411 performs processing related to the acquisition of content files. For example, the content file acquisition unit 411 can acquire a content file. For example, the content file can be generated by applying the present technology using the file generation device 300. In addition, the content file acquisition unit 411 can extract the V-DMC bitstream stored in the acquired content file from the acquired content file and supply the V-DMC bitstream to the reproduction processing unit 412 (V-DMC decoding unit 431). At this time, the content file acquisition unit 411 can extract and supply the V-DMC bitstream under the control of the reproduction processing unit 412. In addition, at this time, the content file acquisition unit 411 can perform random access to the V-DMC bitstream stored in the content file under the control of the random access reproduction control unit 422 (via the reproduction processing unit 412).

[0251] The reproduction processing unit 412 performs processing related to the reproduction of mesh data. For example, the reproduction processing unit 412 may control the content file acquisition unit 411 to extract and acquire a V-DMC bitstream from a content file. In this case, the reproduction processing unit 412 may perform random access control under the control of the random access reproduction control unit 422. Furthermore, the reproduction processing unit 412 may decode the acquired V-DMC bitstream, reconstruct the mesh, render the mesh, and display the display image.

[0252] The random access operation receiving unit 421 performs processing related to receiving random access operations. Random access operations indicate operations (instructions) related to random access to the V-DMC bitstream stored in the content file. For example, random access operations may include instructions for special effects reproduction (e.g., jump reproduction, fast forward, rewind, etc.). For example, the random access operation receiving unit 421 may have an input device operated by a user or the like, and receive random access operations input via the input device. In addition, the random access operation receiving unit 421 may include an application program interface (API) that receives instructions supplied from an application or the like, and may receive random access operations input via the API. The application may perform any processing. The random access operation receiving unit 421 may supply the received random access operations to the random access reproduction control unit 422.

[0253] The random access reproduction control unit 422 performs processing related to the control of random access reproduction. Random access reproduction indicates a reproduction process involving random access of the V-DMC bitstream stored in the content file. Reproduction processes involving random access may include, for example, special effects reproduction (e.g., jump reproduction, fast forward, rewind, etc.). For example, the random access reproduction control unit 422 can control the reproduction processing unit 412 to perform random access reproduction. That is, the random access reproduction control unit 422 can control the reproduction processing unit 412 to randomly access the V-DMC bitstream stored in the content file and start reproduction from a sample in the middle of the sequence (jump reproduction). In addition, the random access reproduction control unit 422 can control the reproduction processing unit 412 to perform fast forward or rewind by applying such a reproduction method.

[0254] The V-DMC decoding unit 431 of the reproduction processing unit 412 performs processing related to decoding of the V-DMC bitstream. For example, the V-DMC decoding unit 431 may obtain a V-DMC bitstream extracted from a content file in the content file acquisition unit 411. The V-DMC decoding unit 431 may decode the V-DMC bitstream and restore V-DMC data (generate restored V-DMC data). The V-DMC decoding unit 431 may supply the restored V-DMC data (restored V-DMC data) to the mesh reconstruction unit 432.

[0255] The mesh reconstruction unit 432 performs processing related to the reconstruction of the mesh (3D data). For example, the mesh reconstruction unit 432 may obtain the restored V-DMC data supplied from the V-DMC decoding unit 431. The mesh reconstruction unit 432 may convert the restored V-DMC data into mesh data. The mesh reconstruction unit 432 may reconstruct a mesh (restored mesh) using the mesh data. The mesh reconstruction unit 432 may supply the restored mesh (3D data) to the display processing unit 433.

[0256] The display processing unit 433 performs processing related to the display of the restored mesh. For example, the display processing unit 433 can obtain the restored mesh (3D data) supplied from the mesh reconstruction unit 432. The display processing unit 433 can render the restored mesh and generate a display image for displaying the restored mesh. The display processing unit 433 may include a display device such as a display and display the display image using the display device. Therefore, the display processing unit 433 may also be referred to as a display unit. In addition, the display processing unit 433 can supply the display image to the outside of the reproduction device 400 (for example, a recording medium, another display device, etc.). Therefore, the display processing unit 433 may also be referred to as an output unit.

[0257] Figure 26 4 is a block diagram showing a main configuration example of the V-DMC decoding unit 431. Note that Figure 26 shows the main configuration including processing units, data flow, etc., and Figure 26 The processing units and data streams shown in FIG are not necessarily all. That is, in the V-DMC decoding unit 431, there may be Figure 26 Processing units may be shown as boxes, or there may be processing units not shown in Figure 26 The process or data flow is shown as arrows or the like.

[0258] like Figure 26 As shown, the V-DMC decoding unit 431 includes a demultiplexing unit 451, an atlas information decoding unit 452, a base grid decoding unit 453, a base grid reconstruction unit 454, a subdivision unit 455, a displacement video decoding unit 456, an unpacking unit 457, a displacement vector application unit 458 and an attribute video decoding unit 459.

[0259] The demultiplexing unit 451 performs demultiplexing-related processing. For example, the demultiplexing unit 451 can obtain the V-DMC bitstream supplied from the content file acquisition unit 411. Furthermore, the demultiplexing unit 451 can demultiplex the obtained V-DMC bitstream and extract the encoded data of the atlas information, the encoded data of the base grid, the encoded data of the displacement video, and the encoded data of the attribute video. Therefore, the demultiplexing unit 451 can also be referred to as an acquisition unit for the various types of information included in the V-DMC bitstream. The demultiplexing unit 451 can supply the extracted encoded data of the atlas information to the atlas information decoding unit 452. Furthermore, the demultiplexing unit 451 can supply the extracted encoded data of the base grid to the base grid decoding unit 453. Furthermore, the demultiplexing unit 451 can supply the extracted encoded data of the displacement video to the displacement video decoding unit 456. Furthermore, the demultiplexing unit 451 can supply the extracted encoded data of the attribute video to the attribute video decoding unit 459.

[0260] The atlas information decoding unit 452 performs processing related to decoding the encoded data of the atlas information. For example, the atlas information decoding unit 452 can obtain the encoded data of the atlas information supplied from the demultiplexing unit 451. In addition, the atlas information decoding unit 452 can decode the obtained encoded data of the atlas information and generate (restore) the atlas information. The atlas information decoding unit 452 can supply the atlas information to the base grid decoding unit 453 and the subdivision unit 455. In addition, the atlas information decoding unit 452 can supply the atlas information as restored V-DMC data to the grid reconstruction unit 432.

[0261] The base mesh decoding unit 453 performs processing related to decoding the encoded data of the base mesh. For example, the base mesh decoding unit 453 can obtain the encoded data of the base mesh supplied from the demultiplexing unit 451. In addition, the base mesh decoding unit 453 can decode the obtained encoded data of the base mesh using a predetermined decoding scheme (e.g., Draco, etc.) and generate (restore) information about the base mesh (e.g., vertex list, triangle list, etc.). For example, the base mesh decoding unit 453 can decode the encoded data of the base mesh based on the atlas information supplied from the atlas information decoding unit 452. The base mesh decoding unit 453 can supply the atlas information and information about the base mesh to the base mesh reconstruction unit 454.

[0262] The base grid reconstruction unit 454 performs processing related to base grid reconstruction. For example, the base grid reconstruction unit 454 may obtain the atlas information and base grid-related information supplied from the base grid decoding unit 453. The base grid reconstruction unit 454 may reconstruct the base grid (generate a restored base grid) using the obtained atlas information and base grid-related information. The base grid reconstruction unit 454 may supply the base grid (restored base grid) to the subdivision unit 455.

[0263] The subdivision unit 455 performs processing related to subdividing the triangles of the base mesh (restored base mesh). For example, the subdivision unit 455 may obtain the base mesh (restored base mesh) supplied from the base mesh reconstruction unit 454. In addition, the subdivision unit 455 may obtain atlas information supplied from the atlas information decoding unit 452. The subdivision unit 455 may subdivide the triangles of the base mesh (restored base mesh) based on the atlas information and generate segmentation points. The subdivision unit 455 may supply the subdivided base mesh (subdivided restored base mesh) to the displacement vector application unit 458.

[0264] The displacement video decoding unit 456 performs processing related to decoding the coded data of the displacement video. For example, the displacement video decoding unit 456 can obtain the coded data of the displacement video supplied from the demultiplexing unit 451. In addition, the displacement video decoding unit 456 can decode the coded data of the displacement video using a predetermined decoding scheme for 2D moving images and generate (restore) the displacement video. The displacement video decoding unit 456 can supply the generated displacement video to the depacketizing unit 457.

[0265] The unpacking unit 457 performs processing related to unpacking displacement vectors. For example, the unpacking unit 457 may obtain the displacement video supplied from the displacement video decoding unit 456. Furthermore, the unpacking unit 457 may unpack the displacement map of the frame image as the displacement video and extract the displacement vectors. The unpacking unit 457 may supply the displacement vectors to the displacement vector application unit 458.

[0266] The displacement vector application unit 458 performs processing related to applying displacement vectors to the subdivided base mesh (subdivided restored base mesh). For example, the displacement vector application unit 458 may obtain the subdivided base mesh (subdivided restored base mesh) supplied from the subdividing unit 455. In addition, the displacement vector application unit 458 may obtain the displacement vectors supplied from the unpacking unit 457. In addition, the displacement vector application unit 458 may apply the displacement vectors to the vertices of the subdivided base mesh (subdivided restored base mesh). In other words, the displacement vector application unit 458 reconstructs a mesh (geometry) corresponding to the original mesh. In this specification, the mesh (geometry) corresponding to the original mesh is also referred to as a decoded mesh. The displacement vector application unit 458 may supply the generated decoded mesh to the mesh reconstruction unit 432 as restored V-DMC data.

[0267] The attribute video decoding unit 459 performs processing related to decoding the encoded data of the attribute video. For example, the attribute video decoding unit 459 can obtain the encoded data of the attribute video supplied from the demultiplexing unit 451. In addition, the attribute video decoding unit 459 can decode the encoded data of the attribute video using a predetermined decoding scheme for 2D moving images and generate (restore) the attribute video. The attribute video decoding unit 459 can supply the attribute map, which is the frame image of the attribute video, as the restored V-DMC data to the mesh reconstruction unit 432.

[0268] The reproducing device 400 having the above configuration may be the above-described second information processing device, and the various methods described above in <3. Transmission and control of random access metadata> may be applied.

[0269] For example, in the reproduction device 400 (second information processing device), the random access reproduction control unit 422 can control random access based on random access metadata for performing random access based on the current sample as an inter-grid sample, which random access metadata is stored in the content file storing the grid data, and the reproduction processing unit 412 can start reproduction of the grid data through random access.

[0270] That is, the random access reproduction control unit 422 can perform control related to random access reproduction based on the random access metadata stored in the content file through the reproduction processing unit 412. For example, the random access reproduction control unit 422 can refer to the random access metadata stored in the content file via the reproduction processing unit 412 and control random access reproduction based on the random access metadata through the reproduction processing unit 412.

[0271] Therefore, the reproduction device 400 can achieve an effect similar to that described above in <3. Transmission and Control of Random Access Metadata>. That is, when the bit stream obtained by encoding the mesh data is stored in a file container, the reproduction device 400 can suppress a decrease in random accessibility while suppressing a decrease in encoding efficiency.

[0272] Note that the random access reproduction control unit 422 can choose whether to decode the grid data from the sample specified by the control information (first control information) based on the control information specifying the decoding start sample, or whether to decode the grid data from the current sample based on the control information specifying the sample to be decoded normally (second control information).

[0273] Such selection can be performed based on any information such as the operating mode of the reproduction device 400, the load status, random access metadata, etc. For example, the random access reproduction control unit 422 can control random access by using control information specified by a user, an application, etc. For example, the specification can be supplied from the random access operation receiving unit 421. In addition, in the case where the current load of the reproduction device 400 is greater than a predetermined standard, the random access reproduction control unit 422 can prioritize low latency, and in other cases can prioritize image integrity. In addition, in the case where the normally decoded sample is closer to the current sample than the decoding start sample, the random access reproduction control unit 422 can control random access based on control information (second control information) that specifies the sample to be decoded normally, and in other cases, can control random access based on control information (first control information) that specifies the decoding start sample. Of course, other selection methods can be applied.

[0274] Using this control method, the reproduction device 400 can achieve normal decoding and display from the current sample, and can also achieve normal decoding and display from the normally decoded sample. In other words, the reproduction device 400 can prioritize image integrity or low latency. In addition, the reproduction device 400 can also select a more appropriate control method according to the situation.

[0275] <Flow of playback processing>

[0276] Next, we will refer to Figure 27 The flowchart describes an example of the flow of the reproduction process performed by the reproduction device 400.

[0277] When the reproduction process starts, in step S401, the random access reproduction control unit 422 of the reproduction device 400 sets a random access mode, which is an operation mode related to the control of random access. For example, the random access reproduction control unit 422 selects whether to decode the mesh data starting from the sample specified by the control information (first control information) that specifies the decoding start sample, or whether to decode the mesh data starting from the current sample based on the control information (second control information) that specifies the sample to be normally decoded.

[0278] In step S402, the content file acquisition unit 411 acquires the content file and extracts the V-DMC bitstream stored in the content file.

[0279] In step S403, the V-DMC decoding unit 431 performs V-DMC decoding processing to decode the V-DMC bitstream and recover the V-DMC data (generate the recovered V-DMC data).

[0280] In step S404, the mesh reconstruction unit 432 converts the recovered V-DMC data into mesh data and reconstructs the mesh (the recovered mesh) using the mesh data.

[0281] In step S405, the display processing unit 433 renders the recovered mesh to generate display information and displays the display information.

[0282] When the processing in step S405 ends, the reproduction process ends.

[0283] <Flowchart of V-DMC decoding processing>

[0284] Reference will be made to Figure 28 the flowchart in Figure 27 to describe an example of the flow of the V-DMC decoding processing executed in step S403.

[0285] When the V-DMC decoding processing starts, in step S421, the demultiplexing unit 451 demultiplexes the V-DMC bitstream.

[0286] In step S422, the atlas information decoding unit 452 decodes the encoded data of the atlas information as the atlas information and generates (recovers) the atlas information.

[0287] In step S423, the base mesh decoding unit 453 decodes the encoded data of the base mesh according to the atlas information and generates (recovers) the information about the base mesh.

[0288] In step S424, the base mesh reconstruction unit 454 reconstructs the base mesh (generates the recovered base mesh) using the atlas information and the information about the base mesh.

[0289] In step S425 , the subdivision unit 455 subdivides the reconstructed base mesh (restored base mesh) and generates segmentation points.

[0290] In step S426 , the displacement video decoding unit 456 decodes the encoded data of the displacement video and generates (restores) the displacement video.

[0291] In step S427 , the unpacking unit 457 unpacks the displacement map of the frame image as the displacement video and extracts the displacement vector.

[0292] In step S428 , the displacement vector application unit 458 applies the displacement vectors to the vertices of the subdivided restored base mesh and generates a decoded mesh.

[0293] In step S429, the attribute video decoding unit 459 decodes the encoded attribute video data and generates (restores) the attribute video. The atlas information generated (restored) in step S422, the decoded grid generated in step S428, and the attribute video generated (restored) in step S429 are supplied to the grid reconstruction unit 432 as restored V-DMC data.

[0294] When the process of step S429 ends, the V-DMC decoding process ends, and the process returns to Figure 27 .

[0295] <Flow of Random Access Reproduction Processing>

[0296] Next, we will refer to Figure 29 The flowchart of FIG. 1 describes an example of the flow of a random access reproduction process for randomly accessing and reproducing a V-DMC bit stream stored in a content file.

[0297] When the random access reproduction process starts, in step S451 , the random access operation receiving unit 421 receives a random access operation.

[0298] In step S452, the random access reproduction control unit 422 performs the operation in step S401 ( Figure 27 ) determines whether the negative control mode or the positive control mode is set as the random access mode. The negative control mode is a mode in which mesh data is decoded starting from the sample specified by the control information based on control information (first control information) that specifies a decoding start sample. The positive control mode is a mode in which mesh data is decoded starting from the current sample based on control information (second control information) that specifies a sample to be decoded normally.

[0299] In a case where it is determined that the negative control mode is set, the process proceeds to step S453.

[0300] In step S453, the random access playback control unit 422 randomly accesses a designated sample before the current sample based on the random access metadata of the content file. The playback processing unit 412 controls the content file acquisition unit 411 in response to this control and begins extracting from the designated sample. In other words, the V-DMC decoding unit 431 decodes the base grid starting from the designated sample.

[0301] In step S454 , under the control of the random access reproduction control unit 422 , the V-DMC decoding unit 431 reconstructs and subdivides the base grid starting from the current sample.

[0302] In step S455 , under the control of the random access reproduction control unit 422 , the V-DMC decoding unit 431 decodes the displacement video according to the decoding of the base mesh, and applies the displacement vector to the vertex from the current sample.

[0303] In step S456, the V-DMC decoding unit 431 decodes the attribute video according to the decoded base mesh under the control of the random access reproduction control unit 422. In addition, the mesh reconstruction unit 432 applies texture to the face of the decoded mesh from the current sample under the control of the random access reproduction control unit 422.

[0304] In step S457, the display processing unit 433 renders the decoded grid from the current sample to generate display information, and displays the display information. When the process of step S457 ends, the random access reproduction process ends.

[0305] Furthermore, in the event that determination is made in step S452 that the positive control mode is set, the process proceeds to step S458 .

[0306] In step S458, the random access playback control unit 422 randomly accesses the current sample based on the random access metadata of the content file. The playback processing unit 412 controls the content file acquisition unit 411 accordingly and starts extracting from the current sample. In other words, the V-DMC decoding unit 431 decodes the base grid starting from the current sample.

[0307] In step S459 , under the control of the random access reproduction control unit 422 , the V-DMC decoding unit 431 reconstructs and subdivides the base mesh starting from a designated sample after the current sample.

[0308] In step S460 , under the control of the random access reproduction control unit 422 , the V-DMC decoding unit 431 decodes the displacement video according to the decoding of the base mesh, and applies the displacement vector to the vertex from the designated sample.

[0309] In step S461, the V-DMC decoding unit 431 decodes the attribute video based on the decoded base mesh under the control of the random access reproduction control unit 422. In addition, the mesh reconstruction unit 432 applies texture to the face of the decoded mesh from the specified sample under the control of the random access reproduction control unit 422.

[0310] In step S462, the display processing unit 433 renders the decoded grid from the designated sample to generate display information, and displays the display information. When the process of step S462 ends, the random access reproduction process ends.

[0311] That is, the random access reproduction control unit 422 controls random access based on random access metadata for performing random access based on the current sample as an inter-grid sample, which is stored in the content file storing the grid data. Then, the reproduction processing unit 412 starts reproduction of the grid data through random access.

[0312] By performing each of the processes described above, the reproduction device 400 can achieve an effect similar to that described above in <3. Transmission and Control of Random Access Metadata>. That is, when the bit stream obtained by encoding the mesh data is stored in a file container, the reproduction device 400 can suppress a decrease in random accessibility while suppressing a decrease in encoding efficiency.

[0313] <6. Supplementary Notes>

[0314] <Computer>

[0315] The series of processes described above can be performed by hardware or software. In the case of performing the series of processes by software, the program included in the software is installed in a computer. Here, the computer includes, for example, a computer incorporated in dedicated hardware, a general-purpose personal computer capable of performing various functions by installing various programs, and the like.

[0316] Figure 30 : is a block diagram showing a configuration example of hardware of a computer that executes the series of processes described above by a program.

[0317] exist Figure 30 In a computer 900 shown in FIG, a central processing unit (CPU) 901 , a read only memory (ROM) 902 , and a random access memory (RAM) 903 are connected to one another via a bus 904 .

[0318] Furthermore, an input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

[0319] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touchpad, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a nonvolatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives the removable medium 921, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0320] In the computer configured as described above, for example, the CPU 901 loads a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes the program, thereby performing the series of processes described above. The RAM 903 also appropriately stores data and the like necessary for the CPU 901 to perform various types of processes.

[0321] For example, the program executed by the computer can be applied by being recorded on the removable medium 921 as a package medium, etc. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by attaching the removable medium 921 to the drive 915 .

[0322] In addition, the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0323] In addition, the program may be installed in the ROM 902 or the storage unit 913 in advance.

[0324] <Target audiences of this technology>

[0325] This technique can be applied to any encoding / decoding scheme.

[0326] Furthermore, the present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0327] In addition, for example, the present technology can also be implemented as a partial configuration of a device, such as a processor as a system large-scale integrated circuit (LSI) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a collection obtained by further adding other functions to the unit (e.g., a video collection), etc.

[0328] Furthermore, for example, the present technology can also be applied to a network system including multiple devices. For example, the present technology can be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology can be implemented in a cloud service that provides image (moving image) related services to any terminal such as a computer, audio-visual (AV) equipment, portable information processing terminal, or Internet of Things (IoT) device.

[0329] Note that in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), and it does not matter whether all components are in the same housing. Therefore, multiple devices stored in different housings and connected via a network, and a single device in which multiple modules are stored in a single housing, are both systems.

[0330] <Fields and applications of this technology>

[0331] The systems, devices, processing units, etc. to which this technology is applied can be used in any field such as transportation, medical treatment, crime prevention, agriculture, animal husbandry, mining, beauty care, factories, home appliances, weather and nature monitoring. In addition, its application is also arbitrary.

[0332] For example, the present technology can be applied to systems and devices for providing content for enjoyment, etc. Furthermore, for example, the present technology can also be applied to systems and devices for transportation, such as traffic condition management and autonomous driving control. Furthermore, for example, the present technology can also be applied to systems and devices for security. Furthermore, for example, the present technology can also be applied to systems and devices for automatic control of machines, etc. Furthermore, for example, the present technology can also be applied to systems and devices for providing agriculture and animal husbandry. Furthermore, for example, the present technology can also be applied to systems and devices for monitoring the state of nature, such as volcanoes, forests, oceans, wild animals, etc. Furthermore, for example, the present technology can also be applied to systems and devices for sports.

[0333] <Other>

[0334] Note that in this specification, a "flag" is information for identifying a plurality of states, and includes not only information for identifying two states of true (1) and false (0), but also information capable of identifying three or more states. Therefore, the value that a "flag" can take can be binary or ternary or more, such as 1 / 0. That is, the number of bits forming the "flag" is any number, and can be one bit or a plurality of bits. Furthermore, it is assumed that identification information (including a flag) includes not only its identification information in a bit stream, but also difference information of identification information with respect to certain reference information in the bit stream, and therefore, in this specification, "flag" and "identification information" include not only its information but also difference information with respect to reference information.

[0335] In addition, various types of information (such as metadata) about the encoded data (bitstream) can be transmitted or recorded in any form as long as the information is associated with the encoded data. Here, the term "association" means that, for example, when processing one data, another data can be used (linked). That is, data associated with each other can be collected as one data, or can be separate data. For example, information associated with the encoded data (image) can be transmitted on a transmission path different from the transmission path of the encoded data (image). In addition, for example, information associated with the encoded data (image) can be recorded in a recording medium different from the recording medium of the encoded data (image) (or in another recording area of ​​the same recording medium). Note that this "association" may not be the entire data, but a part of the data. For example, an image and the information corresponding to the image can be associated with each other in any unit such as multiple frames, a frame, or a part within a frame.

[0336] Note that in this specification, for example, terms such as "combine", "multiplex", "add", "merge", "include", "store", "put in", "introduce" and "insert" refer to combining multiple objects into one, such as combining encoded data and metadata into one data, and refer to a method of "association" described above.

[0337] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0338] For example, a configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, a configuration described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Furthermore, a configuration other than the configuration described above may be added to the configuration of each device (or each processing unit). Furthermore, when the configuration and operation of the entire system are substantially the same, a portion of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or another processing unit).

[0339] In addition, for example, the program described above can be executed in any device. In this case, the device only needs to have necessary functions (functional blocks, etc.) and obtain necessary information.

[0340] Furthermore, for example, each step in a flowchart may be performed by a single device, or may be shared and performed by multiple devices. Furthermore, when a single step includes multiple processing elements, the multiple processing elements may be performed by a single device, or may be shared and performed by multiple devices. In other words, the multiple processing elements included in a single step may be performed as a single process. Conversely, a process described as multiple steps may also be performed collectively as a single step.

[0341] Furthermore, for example, in a program executed by a computer, the processing of the steps describing the program may be performed in a time-series order according to the order described in this specification, or may be performed in parallel or individually at a desired time, such as when a call is made. That is, the processing of each step may be performed in a different order from the order described above, as long as there is no contradiction. Furthermore, the processing of the steps describing the program may be performed in parallel with the processing of another program, or may be performed in conjunction with the processing of another program.

[0342] Furthermore, for example, various technologies related to the present technology may be independently implemented as individual independent entities, as long as there is no contradiction. Of course, any of the various technologies may be implemented in combination. For example, part or all of the technology described in any embodiment may be implemented in combination with part or all of the technology described in other embodiments. Furthermore, part or all of any of the technologies described above may be implemented together with another technology not described above.

[0343] Note that the present technology can also have the following configurations.

[0344] (1) An information processing device comprising:

[0345] a random access metadata generating unit that generates random access metadata for performing random access on mesh data based on a current sample that is an inter-mesh sample; and

[0346] A storage unit stores the random access metadata in a content file storing the mesh data.

[0347] (2) The information processing device according to (1), wherein

[0348] The random access metadata includes control information specifying a decoding start sample.

[0349] (3) The information processing device according to (2), wherein

[0350] The control information specifies the decoding start sample by a difference from a picture order count (POC) of the current sample.

[0351] (4) The information processing device according to (3), wherein

[0352] The difference is a value having a positive or negative sign.

[0353] (5) The information processing device according to (3), wherein

[0354] The differences are absolute values.

[0355] (6) The information processing device according to (1), wherein

[0356] The random access metadata includes control information specifying samples that are decoded normally.

[0357] (7) The information processing device according to (6), wherein

[0358] The control information specifies the normally decoded sample by a difference from a picture order count (POC) of the current sample.

[0359] (8) The information processing device according to (7), wherein

[0360] The difference is a value with a positive or negative sign.

[0361] (9) The information processing device according to (7), wherein

[0362] The difference is the absolute value.

[0363] (10) The information processing device according to (1), wherein

[0364] The random access metadata includes first control information specifying a decoding start sample and second control information specifying a sample to be normally decoded.

[0365] (11) The information processing device according to (10), wherein

[0366] The first control information specifies the decoding start sample by a difference from a picture order count (POC) of the current sample, and

[0367] The second control information specifies the normally decoded sample by using the POC difference between the second control information and the current sample.

[0368] (12) The information processing device according to (11), wherein

[0369] The difference is a value having a positive or negative sign.

[0370] (13) The information processing device according to (1), wherein

[0371] The content file manages the subgrids of the grid data as subsamples.

[0372] The random access metadata generation unit is configured to generate the random access metadata corresponding to the subsample, and

[0373] The storage unit is configured to store the random access metadata corresponding to the subsample in the content file.

[0374] (14) The information processing device according to (13), wherein

[0375] The storage unit is configured to store the random access metadata in a subsample information box of the content file.

[0376] (15) The information processing device according to (13), wherein

[0377] The random access metadata includes information indicating an encoding type of the sub-grid corresponding to the sub-sample.

[0378] (16) The information processing device according to (13), wherein

[0379] The random access metadata includes control information specifying a decoding start subsample.

[0380] (17) The information processing device according to (16), wherein

[0381] The control information specifies the decoding start subsample by a difference from a picture order count (POC) of a current subsample.

[0382] (18) The information processing device according to (17), wherein

[0383] The difference is a value having a positive or negative sign.

[0384] (19) The information processing device according to (17), wherein

[0385] The differences are absolute values.

[0386] (20) The information processing device according to (13), wherein

[0387] The random access metadata includes control information specifying subsamples to be decoded normally.

[0388] (21) The information processing device according to (20), wherein

[0389] The control information specifies the normally decoded subsample by a difference from a picture order count (POC) of a current subsample.

[0390] (22) The information processing device according to (21), wherein

[0391] The difference is a value having a positive or negative sign.

[0392] (23) The information processing device according to (21), wherein

[0393] The differences are absolute values.

[0394] (24) The information processing device according to (13), wherein

[0395] The random access metadata includes first control information designating a decoding start subsample and second control information designating a subsample to be normally decoded.

[0396] (25) The information processing device according to (24), wherein

[0397] The first control information specifies the decoding start subsample by a difference from a picture order count (POC) of a current subsample, and

[0398] The second control information specifies the normally decoded subsample by a difference between the second control information and the POC of the current subsample.

[0399] (26) The information processing device according to (25), wherein

[0400] The difference is a value having a positive or negative sign.

[0401] (27) The information processing device according to (1), wherein

[0402] The storage unit is configured to store random access metadata as a parameter of the sample group.

[0403] (28) The information processing device according to (1), wherein

[0404] The random access metadata generation unit is configured to generate the random access metadata corresponding to the inter-grid sample.

[0405] (29) The information processing device according to (1), wherein

[0406] The mesh data is data of a base mesh having a lower degree of fineness than an original mesh, the base mesh being generated by extracting vertices from the original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object.

[0407] (30) The information processing device according to (1), wherein

[0408] The mesh data is data of an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object.

[0409] (31) The information processing device according to (1), wherein

[0410] The content file is a file container compliant with the International Organization for Standardization Base Media File Format (ISOBMFF).

[0411] (32) The information processing device according to (1), wherein

[0412] The content file is a Matroska media container.

[0413] (33) The information processing device according to (1), wherein

[0414] The storage unit is configured to store the random access metadata in a track of the content file storing the mesh data.

[0415] (34) An information processing method comprising:

[0416] generating random access metadata for performing random access on mesh data based on a current sample being an inter-mesh sample; and

[0417] The random access metadata is stored in a content file storing the mesh data.

[0418] (41) An information processing device comprising:

[0419] a random access reproduction control unit that controls random access based on random access metadata for performing random access based on a current sample that is an inter-grid sample, the random access metadata being stored in a content file storing the grid data; and

[0420] A reproduction processing unit starts reproduction of the mesh data through the random access.

[0421] (42) The information processing device according to (41), wherein

[0422] The random access metadata includes control information specifying a decoding starting sample, and

[0423] The random access reproduction control unit is configured to decode the mesh data starting from the sample specified by the control information.

[0424] (43) The information processing device according to (42), wherein

[0425] The control information specifies the decoding start sample by a difference from a picture order count (POC) of the current sample.

[0426] (44) The information processing device according to (43), wherein

[0427] The difference is a value having a positive or negative sign.

[0428] (45) The information processing device according to (43), wherein

[0429] The differences are absolute values.

[0430] (46) The information processing device according to (41), wherein

[0431] The random access metadata includes control information specifying samples that are decoded normally, and

[0432] The random access reproduction control unit is configured to decode the mesh data starting from the current sample.

[0433] (47) The information processing device according to (46), wherein

[0434] The control information specifies the normally decoded sample by a difference from a picture order count (POC) of the current sample.

[0435] (48) The information processing device according to (47), wherein

[0436] The difference is a value having a positive or negative sign.

[0437] (49) The information processing device according to (47), wherein

[0438] The differences are absolute values.

[0439] (50) The information processing device according to (41), wherein

[0440] The random access metadata includes first control information specifying a decoding start sample and second control information specifying a sample to be decoded normally, and

[0441] The random access reproduction control unit is configured to select whether to control the random access based on the first control information or to control the random access based on the second control information.

[0442] (51) The information processing device according to (50), wherein

[0443] The first control information specifies the decoding start sample by a difference from a picture order count (POC) of the current sample, and

[0444] The second control information specifies the normally decoded sample by a difference between the second control information and the POC of the current sample.

[0445] (52) The information processing device according to (51), wherein

[0446] The difference is a value having a positive or negative sign.

[0447] (53) The information processing device according to (41), wherein

[0448] The content file manages a sub-grid of the grid data as a sub-sample and stores the random access metadata corresponding to the sub-sample.

[0449] The random access reproduction control unit is configured to control the random access to the subsample based on the random access metadata corresponding to the subsample, and

[0450] The reproduction processing unit is configured to start reproduction of the mesh data for each sub-sample through the random access.

[0451] (54) The information processing device according to (53), wherein

[0452] The content file stores the random access metadata in a subsample information box.

[0453] (55) The information processing device according to (53), wherein

[0454] The random access metadata includes information indicating an encoding type of the sub-grid corresponding to the sub-sample, and

[0455] The random access reproduction control unit is configured to control the random access to the subsample based on the encoding type.

[0456] (56) The information processing device according to (53), wherein

[0457] The random access metadata includes control information specifying a decoding start subsample, and

[0458] The random access reproduction control unit is configured to decode the mesh data starting from the subsample specified by the control information.

[0459] (57) The information processing device according to (56), wherein

[0460] The control information specifies the decoding start subsample by a difference from a picture order count (POC) of the current subsample.

[0461] (58) The information processing device according to (57), wherein

[0462] The difference is a value having a positive or negative sign.

[0463] (59) The information processing device according to (57), wherein

[0464] The differences are absolute values.

[0465] (60) The information processing device according to (53), wherein

[0466] The random access metadata includes control information specifying subsamples to be decoded normally, and

[0467] The random access reproduction control unit is configured to decode the grid data starting from the current subsample.

[0468] (61) The information processing device according to (60), wherein

[0469] The control information specifies the normally decoded subsample by a difference from a picture order count (POC) of the current subsample.

[0470] (62) The information processing device according to (61), wherein

[0471] The difference is a value having a positive or negative sign.

[0472] (63) The information processing device according to (61), wherein

[0473] The differences are absolute values.

[0474] (64) The information processing device according to (53), wherein

[0475] The random access metadata includes first control information specifying a decoding start subsample and second control information specifying a subsample to be normally decoded, and

[0476] The random access reproduction control unit is configured to select whether to control the random access based on the first control information or to control the random access based on the second control information.

[0477] (65) The information processing device according to (64), wherein

[0478] The first control information specifies the decoding start subsample by a difference from a picture order count (POC) of a current subsample, and

[0479] The second control information specifies the normally decoded subsample by a difference between the second control information and the POC of the current subsample.

[0480] (66) The information processing device according to (65), wherein

[0481] The difference is a value having a positive or negative sign.

[0482] (67) The information processing device according to (41), wherein

[0483] The content file stores the random access metadata as a parameter of a sample group.

[0484] (68) The information processing device according to (41), wherein

[0485] The random access metadata corresponds to the inter-grid samples.

[0486] (69) The information processing device according to (41), wherein

[0487] The mesh data is data of a base mesh having a lower degree of detail than an original mesh, the base mesh being generated by extracting vertices from the original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object.

[0488] (70) The information processing device according to (41), wherein

[0489] The mesh data is data of an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object.

[0490] (71) The information processing device according to (41), wherein

[0491] The content file is a file container compliant with the International Organization for Standardization Base Media File Format (ISOBMFF).

[0492] (72) The information processing device according to (41), wherein

[0493] The content file is a Matroska media container.

[0494] (73) The information processing device according to (41), wherein

[0495] The content file stores the random access metadata in a track storing the mesh data.

[0496] (74) An information processing method comprising:

[0497] controlling random access based on random access metadata for performing random access based on a current sample that is an inter-mesh sample, the random access metadata being stored in a content file storing mesh data; and

[0498] The reproduction of the mesh data is started by the random access.

[0499] Reference Signs List

[0500] 300 file generation device

[0501] 311V-DMC data generation unit

[0502] 312V-DMC encoder unit

[0503] 313 Content file generation unit

[0504] 314 Random Access Metadata Generation Unit

[0505] 351 Atlas Information Coding Unit

[0506] 352-based grid coding unit

[0507] 353 displacement vector correction unit

[0508] 354 displacement video coding unit

[0509] 355 grid reconstruction units

[0510] 356 Attribute Mapping Conversion Unit

[0511] 357 Attribute Video Coding Unit

[0512] 358 multiplexing unit

[0513] 400 reproduction device

[0514] 411 content file acquisition unit

[0515] 412 reproduction processing unit

[0516] 421 Random Access Operation Receiving Unit

[0517] 422 Random access reproduction control unit

[0518] 431V-DMC decoding unit

[0519] 432 grid reconstruction units

[0520] 433 display processing unit

[0521] 451 demultiplexing unit

[0522] 452 Atlas Information Decoding Unit

[0523] 453-based grid decoding unit

[0524] 454-based grid reconstruction unit

[0525] 455 subdivision unit

[0526] 456 bit shift video decoding unit

[0527] 457 Unpacking Unit

[0528] 458 Displacement Vector Application Unit

[0529] 459 Attribute Video Decoding Unit

[0530] 900 computers

Claims

1. An information processing device, comprising: a random access metadata generating unit that generates random access metadata for performing random access on mesh data based on a current sample that is an inter-mesh sample; as well as A storage unit stores the random access metadata in a content file storing the mesh data.

2. The information processing device according to claim 1, wherein The random access metadata includes control information specifying a decoding start sample.

3. The information processing device according to claim 2, wherein: The control information specifies the decoding start sample by a difference from a picture order count (POC) of the current sample.

4. The information processing device according to claim 1, wherein: The random access metadata includes control information specifying samples that are decoded normally. The information processing apparatus according to claim 4 , wherein The control information specifies the normally decoded sample by a difference from a picture order count (POC) of the current sample. The information processing apparatus according to claim 1 , wherein: The random access metadata includes first control information specifying a decoding start sample and second control information specifying a sample to be normally decoded.

7. The information processing apparatus according to claim 6, wherein: The first control information specifies the decoding start sample by a difference from a picture order count (POC) of the current sample, and The second control information specifies the normally decoded sample by a difference between the second control information and the POC of the current sample.

8. The information processing apparatus according to claim 1, wherein: The content file manages the subgrids of the grid data as subsamples. The random access metadata generation unit is configured to generate the random access metadata corresponding to the subsample, and The storage unit is configured to store the random access metadata corresponding to the subsample in the content file.

9. The information processing apparatus according to claim 8, wherein: The storage unit is configured to store the random access metadata in a subsample information box of the content file.

10. The information processing apparatus according to claim 8, wherein: The random access metadata includes information indicating an encoding type of the sub-grid corresponding to the sub-sample.

11. The information processing apparatus according to claim 8, wherein: The random access metadata includes control information specifying a decoding start subsample.

12. The information processing apparatus according to claim 11, wherein: The control information specifies the decoding start subsample by a difference from a picture order count (POC) of a current subsample.

13. The information processing apparatus according to claim 8, wherein: The random access metadata includes control information specifying subsamples to be decoded normally. The information processing apparatus according to claim 13 , wherein: The control information specifies the normally decoded subsample by a difference from a picture order count (POC) of a current subsample.

15. The information processing apparatus according to claim 8, wherein: The random access metadata includes first control information designating a decoding start subsample and second control information designating a subsample to be normally decoded.

16. The information processing apparatus according to claim 15, wherein: The first control information specifies the decoding start subsample by a difference from a picture order count (POC) of a current subsample, and The second control information specifies the normally decoded subsample by a difference between the second control information and the POC of the current subsample.

17. The information processing apparatus according to claim 1, wherein: The storage unit is configured to store the random access metadata in a track of the content file storing the mesh data.

18. An information processing method, comprising: generating random access metadata for performing random access on mesh data based on a current sample being an inter-mesh sample; as well as The random access metadata is stored in a content file storing the mesh data.

19. An information processing device comprising: a random access reproduction control unit that controls the random access based on random access metadata for performing random access based on a current sample that is an inter-grid sample, the random access metadata being stored in a content file storing the grid data; as well as A reproduction processing unit starts reproduction of the mesh data through the random access.

20. An information processing method, comprising: controlling the random access based on random access metadata for performing random access based on a current sample that is an inter-grid sample, the random access metadata being stored in a content file storing grid data; as well as The reproduction of the mesh data is started by the random access.

Images