Information processing apparatus and method
By storing the display timing information of subframes in the G-PCC file, the problem that the decoder is difficult to determine the display timing of subframes is solved, and high-quality reproduction and display of 3D data is realized, thereby avoiding delay and frame loss.
Patent Information
- Application Number
- CN202380026550.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-17
- Filing Date
- 2023-02-28
- Publication Date
- 2024-11-01
AI Technical Summary
During the decoding process of the G-PCC bitstream, it is difficult for the decoder to determine the display timing of each subframe, resulting in a decrease in the 3D data reproduction and display quality, and delays or frame loss may occur.
By storing the display timing information of the subframes as metadata in the file, the extraction unit extracts subframe samples from the bit stream and decodes these samples in the decoding unit to satisfy the display timing of the subframes, and constructs a decoded point cloud.
Ensure that each subframe is decoded and displayed at the correct timing, avoiding reproduction delays and frame loss, and improving the reproduction and display quality of 3D data.
Smart Images

Figure CN118891884A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method capable of preventing a reduction in the quality of reproduction and display of 3D data. Background Art
[0002] In the related art, in MPEG-I Part 9 (ISO / IEC (International Organization for Standardization / International Electrotechnical Commission) 23090-9), G-PCC (Geometry-based Point Cloud Compression) is standardized (for example, see non-patent document 1), wherein G-PCC is a coding technology for encoding a point cloud which is a collection of points having position information and attribute information (for example, color, reflectivity, etc.) in a three-dimensional space, wherein geometric information representing a three-dimensional shape and attributes representing the attribute information are divided.
[0003] In addition, there is ISOBMFF (International Organization for Standardization Base Media File Format) which is a file container specification of MPEG-4 (Moving Picture Experts Group-4), an international standard technology for moving picture compression (for example, see Non-Patent Document 2).
[0004] In addition, in order to efficiently perform reproduction processing and network distribution from local storage of a G-PCC bit stream which is a bit stream of encoded data encoded using G-PCC, a method for storing a G-PCC bit stream in an ISOBMFF file is standardized in MPEG-I Part 18 (ISO / IEC 23090-18) (for example, see non-patent document 3).
[0005] By the way, in order to encode a point cloud, there is a method of encoding a combined frame in which a plurality of frames of a point cloud are combined as subframes and using the resulting combined frame as a sample (for example, Non-Patent Document 4).
[0006] [Citation List]
[0007] [Non-patent literature]
[0008] [Non-patent document 1]
[0009] "Information technology-MPEG-I(Coded Representation of ImmersiveMedia)-Part 9:Geometry-based Point Cloud Compression" ("Text of ISO / IECFDIS23090-9Geometry-based Point Cloud Compression(draft 20)"), ISO / IEC 23090-9:2022(E), MDS19617, March 13, 2022
[0010] [Non-patent document 2]
[0011] "Information technology-Coding of audio-visual objects-Part 12: ISObase media file format, TECHNICAL CORRIGENDUM 1", ISO / IEC FDIS14496-12:2020(E), ISO / IEC 14496-12:2015 / Cor.1, ISO / IEC JTC 1 / SC 29 / WG 11, June 3, 2016
[0012] [Non-patent document 3]
[0013] Sejin Oh, Ryohei Takahashi, Youngkwon Lim, "Draft text of ISO / IECFDIS23090-18Carriage of Geometry-based Point Cloud Compression Data", ISO / IECJTC 1 / SC 29 / WG 03N0311, MDS20565, October 2, 2021
[0014] [Non-patent document 4]
[0015] “G-PCC codec description”, ISO / IEC JTC 1 / SC 29 / WG 7N 0215, MDS20983, December 29, 2021 Summary of the invention
[0016] [Technical issues]
[0017] However, in the method described in Non-Patent Document 4, the decoder does not know at what timing each sample should be acquired and decoded before decoding the bit stream. This may result in a reduction in the quality of 3D data reproduction and display due to, for example, a delay in the reproduction process or frame loss during display.
[0018] The present disclosure has been made in view of such circumstances, and aims to prevent degradation in the quality of reproduction and display of 3D data.
[0019] [Solution to the problem]
[0020] An information processing device according to one aspect of the present technology is an information processing device, which includes: an extraction unit, which extracts samples including subframes from a bitstream stored in a file based on first information about the display timing of the subframes stored in the file as metadata to meet the display timing of the subframes; a decoding unit, which decodes the extracted samples; and a construction unit, which constructs a point cloud of the subframes included in the decoded samples based on the first information to meet the display timing of the subframes.
[0021] An information processing method according to one aspect of the present technology is an information processing method, which includes: based on first information about display timing of subframes stored in a file as metadata, extracting samples including subframes from a bitstream stored in a file to meet the display timing of the subframes; decoding the extracted samples; and based on the first information, constructing a point cloud of the subframes included in the decoded samples to meet the display timing of the subframes.
[0022] An information processing device according to another aspect of the present technology is an information processing device including: an encoding unit that encodes a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data; a bitstream generating unit that generates a bitstream including the encoded data and generates first information about display timing of the subframes based on the bitstream; and a file generating unit that generates a file for storing the bitstream and stores the first information as metadata in the file.
[0023] An information processing method according to another aspect of the present technology is an information processing method, which includes: encoding a combined frame in which multiple frames are combined as subframes into samples to generate encoded data; generating a bit stream including the encoded data; generating first information about display timing of the subframes based on the bit stream; generating a file for storing the bit stream; and storing the first information as metadata in the file.
[0024] In an information processing device and method according to one aspect of the present technology, based on first information about display timing of subframes stored in a file as metadata, samples including subframes are extracted from a bitstream stored in a file to meet the display timing of the subframes; the extracted samples are decoded; and based on the first information, a point cloud of the subframes included in the decoded samples is constructed to meet the display timing of the subframes.
[0025] In an information processing device and method according to another aspect of the present technology, a combined frame in which multiple frames are combined as subframes is encoded as samples to generate encoded data; a bit stream including the encoded data is generated; first information about display timing of the subframes is generated based on the bit stream; a file for storing the bit stream is generated; and the first information is stored as metadata in the file. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] [ Figure 1 ]
[0027] Figure 1 An example of the structure of a G-PCC bitstream is shown.
[0028] [ Figure 2 ]
[0029] Figure 2 A combined frame is shown.
[0030] [ Figure 3 ]
[0031] Figure 3 The frame number attribute is shown.
[0032] [ Figure 4 ]
[0033] Figure 4 The frame index attribute is shown.
[0034] [ Figure 5 ]
[0035] Figure 5 An example of the structure of a G-PCC file is shown.
[0036] [ Figure 6 ]
[0037] Figure 6 An example of a SubSampleInformationBox is shown.
[0038] [ Figure 7 ]
[0039] Figure 7 An example of codec_specific_parameters is shown.
[0040] [ Figure 8 ]
[0041] Figure 8 An example of the encoding / decoding method is shown.
[0042] [ Fig. 9 ]
[0043] Fig. 9 An example of codec_specific_parameters is shown.
[0044] [ Fig.10 ]
[0045] Fig.10 An example of application of subsample_delta is shown.
[0046] [ Fig.11 ]
[0047] Fig.11 An example of codec_specific_parameters is shown.
[0048] [ Fig.12 ]
[0049] Fig.12 An example of application of duration is shown.
[0050] [ Fig.13 ]
[0051] Fig.13 A configuration example of a Matroska media container is shown.
[0052] [ Fig.14 ]
[0053] Fig.14 is a block diagram showing a main configuration example of a file generating device.
[0054] [ Fig.15 ]
[0055] Fig.15 is a flowchart showing an example of the flow of file generation processing.
[0056] [ Fig.16 ]
[0057] Fig.16 is a block diagram showing a main configuration example of a reproducing device.
[0058] [ Fig.17 ]
[0059] Fig.17 is a flowchart showing an example of the flow of the reproduction process.
[0060] [ Fig.18 ]
[0061] Fig.18 is a block diagram showing a main configuration example of a computer. DETAILED DESCRIPTION
[0062] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
[0063] 1. Documents supporting technical content and technical terminology, etc.
[0064] 2. Reproduction of combined frames
[0065] 3. Transmit information about display timing
[0066] 4. First Embodiment (File Generating Device, Reproducing Device)
[0067] 5. Supplement
[0068] <1. Documents supporting technical content and technical terminology, etc.>
[0069] The scope disclosed in the present technology is not limited to the contents described in the embodiments, and also includes the contents described in the following non-patent documents, patent documents, etc. known at the time of filing, and the contents of other documents cited in the following non-patent documents, patent documents, etc.
[0070] [Non-patent document 1] (as mentioned above)
[0071] [Non-patent document 2] (as mentioned above)
[0072] [Non-patent document 3] (as mentioned above)
[0073] [Non-patent document 4] (as mentioned above)
[0074] [Non-patent document 5] https: / / www.matroska.org / index.html
[0075] In other words, the contents of the non-patent documents, patent documents, etc. listed above, as well as the contents of other documents cited in the patent documents listed above, are also the basis for determining the support requirements.
[0076] <2. Reproduction of combined frames>
[0077] <Point Cloud>
[0078] In the related art, there is 3D data (for example, point cloud) expressing a three-dimensional structure by point position information, attribute information, and the like.
[0079] For example, in the case of a point cloud, a three-dimensional structure (an object having a three-dimensional shape) is expressed as a set of multiple points. The point cloud consists of the position information (also referred to as geometric information) of each point and the attribute information (also referred to as attributes). The attributes can include any information. For example, color information, reflectivity information, normal information, etc. of each point can be included in the attributes. In this way, according to the point cloud, the data structure is relatively simple, and any three-dimensional structure can be expressed with sufficient accuracy by using a sufficiently large number of points.
[0080] <Overview of G-PCC>
[0081] In Non-Patent Document 1, an encoding technique called geometry-based point cloud compression (G-PCC) is disclosed, which encodes a point cloud by dividing the point cloud into geometric information and attributes. G-PCC is standardized in MPEG-I Part 9 (ISO / IEC 23090-9).
[0082] For example, the geometric information is compressed by octree encoding. Octree encoding is an encoding system that performs encoding by converting the data structure of geometric information into an octree.
[0083] In this case, voxels are used to express the geometric information, and the voxel is a three-dimensional region having a hierarchical structure in the shape of a rectangular parallelepiped (including a cube shape). In other words, it is expressed whether there is a point for each voxel. The voxels in which there are points are divided, and the presence / absence of points in the divided small regions (lower-level voxels) is expressed. For the voxels in which there are points, such division is recursively repeated until the lowest level, thereby forming a hierarchical structure of voxels.
[0084] The octree expresses such a voxel representation (the presence / absence of points for each voxel) as an octree. Each node of the octree corresponds to each voxel of the voxel representation. For example, a node corresponding to a voxel in which there is a point is represented as a value "1", and a node corresponding to a voxel in which there is no point is represented as a value "0". In a node corresponding to a voxel in which there is a point (a node with a value "1"), child nodes (nodes at a lower level) are formed.
[0085] The bitstream of the encoded data generated by encoding the geometric information as described above is also referred to as a geometric bitstream.
[0086] Techniques such as prediction weight lifting, region adaptive hierarchical transform (RAHT), fixed weight lifting, etc. are applied to the compression of attributes. The bitstream of the encoded data generated by encoding the attributes is also referred to as an attribute bitstream. The bitstream obtained by combining the geometric bitstream and the attribute bitstream into one will also be referred to as a G-PCC bitstream.
[0087] <Bitstream Structure>
[0088] An example of the structure of such a G-PCC bitstream is Figure 1 As shown in Figure 1 In the example, each rectangular area represents a type-length-value encapsulation structure (tlv_encapsulation()).
[0089] SPS (Sequence Parameter Set) indicates a field in which a sequence parameter set is stored. GPS (Geometry Parameter Set) indicates an area in which a geometry parameter set is stored. APS (Attribute Parameter Set) indicates a field in which an attribute parameter set is stored. GPS and APS may be given for each slice. GPS and APS are associated with SPS using sps_id.
[0090] The geometry information data unit indicates a field in which geometry information data of one slice is stored. The geometry information data unit is associated with GPS using gps_id. The attribute data unit indicates a field in which attribute data of one slice is stored. For one geometry information data unit, multiple attribute data units may be given. The attribute data unit is associated with APS using aps_id.
[0091] The geometry information data unit and attribute data unit configuring the same slice are associated with each other through slice_id. The same point can belong to different slices. Slices are classified into independent slices and dependent slices. Independent slices do not depend on other slices when decoding, while dependent slices depend on other slices when decoding.
[0092] <Combined Frame>
[0093] By the way, point clouds can change in the time direction like a moving image (also called dynamic). G-PCC can encode such dynamic point clouds within a certain period of time. Such point clouds are composed of multiple frames (also called point cloud frames) at different times, just like a moving image. For example, for geometric information, an octree of each frame (for each frame) is encoded.
[0094] Non-patent document 4 discloses a method for encoding multiple frames into one frame. Specifically, Figure 2 As shown in , the encoder generates a combined frame by combining the octrees of multiple frames as subframes, encodes the combined frame like a normal frame, and sets the resulting combined frame as a sample. Therefore, the point cloud of the combined frame is a combination of the point clouds of the respective subframes and includes the points of the subframes. In other words, the octree of the combined frame is a combination of the octrees of the respective subframes and includes nodes whose values of the octrees of the respective subframes are "1".
[0095] exist Figure 2In the example of , two frames (frame 1 and frame 2) are combined, but any number of frames can be combined (including in a combined frame). Each frame in the combined frame ( Figure 2 Frame 1 and frame 2 in the example of ) are also called subframes.
[0096] This approach makes it possible to reduce the number of decoder initializations performed for each frame, and thus prevent an increase in the load of the decoding process. For example, this is useful for decoding processing in a client device or the like having relatively low processing performance. There may be redundancy between octrees of subframes. In particular, there may be redundancy between octrees in an upper layer. Such redundancy of octrees may reduce coding efficiency. Combining subframes makes it possible to reduce such redundancy of octrees, and thus prevent a reduction in coding efficiency.
[0097] Subframes may be combined in any manner. For example, any number of subframes may be combined. The order in which the subframes to be combined are arranged does not have to be the order in which the subframes are displayed. For example, non-contiguous subframes may be combined. For 3D data, the length of the display period of a frame (subframe) during reproduction depends on the application and does not have to be constant (it may vary depending on the frame (subframe)).
[0098] <Reproduction of Subframes>
[0099] The reproduction device divides such a combined frame into subframes and reproduces each subframe individually (reproducing each subframe as a normal frame). In other words, the reproduction device needs to know the order in which the subframes are reproduced. In this regard, non-patent document 4 discloses a method of transmitting information indicating the order in which subframes are displayed (frame number attribute and frame index attribute).
[0100] The frame number attribute is a method of assigning a frame number indicating the display order in the entire sequence as an attribute to each subframe. Figure 3 In the example of , a frame number (FN#x (x is a natural number)) is assigned to an attribute data unit (ADU) of each slice (corresponding to a frame). The reproduction device decodes the combined frame and reproduces each subframe so that the images of the point clouds of the respective subframes are displayed in the order indicated by the frame number (i.e., in the order of FN#0, FN#1, FN#2, FN#3, FN#4, FN#5, FN#6, FN#7, etc.).
[0101] like Figure 3 As shown in , with this frame number attribute, the order of subframes can change not only within a combined frame, but also across combined frames. Figure 3In the combined frame, the order of subframe FN#1 and subframe FN#2 is reversed, and the order of subframe FN#4 and subframe FN#5 is reversed. In other words, subframe FN#1 and subframe FN#2 are combined in an order different from the display order to form a combined frame of sample a. Therefore, the order of subframe FN#1 and subframe FN#2 is reversed in the combined frame.
[0102] Therefore, in the case of the frame number attribute, the reproduction device repeats the following processing: decoding the sample to be processed according to the decoding time, displaying the subframe according to the frame number, and if the next subframe to be displayed does not exist in the already decoded sample, decoding the previous sample or the next sample.
[0103] On the other hand, the order of subframe FN#4 and subframe FN#5 is reversed so that the partner (subframe) to be combined with each of the subframes is changed. For example, subframe FN#4 is combined with subframe FN#3 to form a combined frame of sample b. Subframe FN#5 is combined with subframe FN#6 and subframe FN#7 to form a combined frame of sample c. In other words, the order is reversed across the combined frame. Changing the subframes to be combined in this way can prevent a decrease in coding efficiency.
[0104] The frame index attribute is a method of assigning a frame index indicating the display order in a combined frame as an attribute to each subframe. Figure 4 In the example of , a frame index (FI#x (x is a natural number)) is assigned to an attribute data unit (ADU) of each slice (corresponding to a frame). The reproducing device decodes the combined frame and reproduces each subframe so that images of point clouds of the respective subframes are displayed in the order indicated by the frame index.
[0105] like Figure 4 As shown in , in the case of this frame index attribute, the order of subframes can be changed in the combined frame. Figure 4 In the combined frame, the order of subframe FI#1 and subframe FI#2 is reversed. In other words, subframe FN#1 and subframe FN#2 are combined in an order different from the display order to form a combined frame of sample a. Therefore, the order of subframe FN#1 and subframe FN#2 is reversed in the combined frame.
[0106] In this case, the reproduction device displays subframes in the order of FN#0, FN#1, FN#2 of sample a, FN#0, FN#1 of sample b, FN#0, FN#1, FN#2 of sample c, and so on.
[0107] Therefore, in the case of the frame index attribute, the reproducing apparatus repeats the process of decoding a sample to be processed according to the decoding time and appropriately rearranging and displaying subframes in the sample according to the frame index.
[0108] <isobmff>
[0109] In Non-Patent Document 2, the ISOBMFF (International Organization for Standardization Base Media File Format), which is a specification of a file container for the international standard technology of moving image compression of MPEG-4 (Moving Picture Experts Group-4), is disclosed.
[0110] <Storage of G-PCC Bitstream in ISOBMFF>
[0111] In Non-Patent Document 3, for the purpose of the efficiency of reproduction processing and network distribution of the locally stored bitstream encoded in the G-PCC, a method for storing the G-PCC bitstream in the ISOBMFF is disclosed. This method is standardized in MPEG-I Part 18 (ISO / IEC 23090-18). Hereinafter, the G-PCC bitstream stored in the ISOBMFF will be referred to as a G-PCC file.
[0112] Figure 5 An example of the G-PCC file structure is shown. The GPCCSampleEntry in the TrackBox is a field for storing metadata in the G-PCC file, and this field includes the GPCCDecoderConfigurationRecord. For example, the sequence parameter set (SPS) is stored in the GPCCDecoderConfigurationRecord. The GPCCDecoderConfigurationRecord can also store the geometry parameter set (GPS), the attribute parameter set (APS), and the tile inventory according to the sample entry type.
[0113] Each sample in the MediaDataBox can store the geometry information and attributes corresponding to a point cloud frame. In addition, this sample can store the geometry parameter set, the attribute parameter set, and the tile inventory according to the sample entry type.
[0114] For the G-PCC file, two methods are specified: a single-track encapsulation method, in which the correlated geometry information and attributes are stored in one track; and a multi-track method, in which the correlated geometry information and attributes are stored in different tracks.
[0115] <subsampleinformationbox>
[0116] The field used for specific consecutive bytes in a sample is also called a subsample. The definition of the subsample depends on the encoding codec specified. For example, in the case of HEVC (High Efficiency Video Coding), the NAL unit is a subsample. In the case of the method described in non-patent document 4, the combined frame is a sample, and each point cloud frame in the point cloud frame (subframe) that forms the combined frame is a subsample. SubSampleInformationBox is set in the field for storing metadata of the G-PCC file, where information can be added for each subsample (information is added to the subsample).
[0117] Figure 6 An example of the syntax of the SubSampleInformationBox is shown. Figure 6 In the example of , for SubSampleInformationBox, sample_delta, subsample_count, subsample_size, codec_specific_parameters, etc. are defined. sample_delta is a parameter for specifying a sample with subsamples. subsample_count is a parameter indicating the number of subsamples. subsample_size is a parameter indicating the size of a subsample. codec_specific_parameters defines additional subsample information determined depending on the codec.
[0118] Figure 7 An example of the syntax of codec_specific_parameters is shown. Figure 7 As shown in, for example, for a flag value of "0" (flag==0), one subsample (based on a subsample of a G-PCC unit, 1 subsample) includes one GPCC unit. In codec_specific_parameters, the payloadType (e.g., geometry information, attributes, etc.) of the GPCC unit is stored. For example, when the payload is attribute data (payloadType==4), the attribute identifier attrIdx is stored.
[0119] For a flag value of "1" (flag==1), a subsample (a tile-based subsample, 1 subsample) includes one or more consecutive GPCC units corresponding to a GPCC tile, or includes one or more consecutive GPCC units, the one or more consecutive GPCC units including a parameter set, a tile library, and a frame boundary marker. In codec_specific_parameters, when the subsample is a GPCC tile (tile_data==1), the tile identifier tile_id is stored.
[0120] <Display Timing>
[0121] However, in the method described in Non-Patent Document 4, the display timing of each subframe is not explicitly described as metadata in the G-PCC file. This makes it difficult for a decoder to grasp the display timing of each subframe before decoding a bit stream.
[0122] As described above, the display period of each point cloud frame can be defined to have any length. When the combined frame described in non-patent document 4 is applied, any number of subframes can be stored in one sample. Therefore, the length of the display period of each combined frame is not necessarily constant. In addition, since the order of the subframes can be changed for combination as described above, the first subframe of the sample is not necessarily displayed first. In particular, in the case of the frame number attribute, since the order of the subframes can be reversed across samples, the subframes of other samples can be displayed before all the subframes of a certain sample are displayed. Therefore, the display timing of each point cloud frame is uncertain.
[0123] The decoder needs to decode each sample in the bitstream before the first display timing of the subframe included in the sample. However, it is difficult for the decoder to obtain information indicating the display timing of each subframe from the G-PCC file, and as mentioned above, the timing is uncertain. In other words, the decoder does not know at what timing (when) to obtain and decode each sample. This may cause, for example, a delay in the reproduction process, which may cause frame loss during display. A method of decoding all samples before the start of reproduction can also be considered, but in this case, there is a risk of increased delay before the start of display. In addition, it is also difficult for the decoder to determine whether the file can be reproduced before the decoder decodes the bitstream. Therefore, there is a risk that the quality of reproduction and display of 3D data may be reduced.
[0124] <3. Transmission of Information Regarding Display Timing>
[0125] <Method 1>
[0126] Therefore, if Figure 8 As shown in the top row of the table in , information about the display timing of the subframe is stored as metadata of the file (method 1).
[0127] For example, an information processing device (also referred to as a first information processing device) may include: an extraction unit that extracts samples including subframes from a bitstream stored in a file based on first information about display timing of subframes stored in a file as metadata to satisfy display timing of the subframes; a decoding unit that decodes the extracted samples; and a construction unit that constructs a point cloud of subframes included in the decoded samples based on the first information to satisfy display timing of the subframes. In addition, an information processing method (also referred to as a first information processing method) may include: extracting samples including subframes from a bitstream stored in a file based on first information about display timing of subframes stored in a file as metadata to satisfy display timing of subframes; decoding the extracted samples; and constructing a point cloud of subframes included in the decoded samples based on the first information to satisfy display timing of the subframes.
[0128] Therefore, the first information processing device can more easily grasp the display timing of each subframe. Therefore, the first information processing device can decode each sample to meet the display timing of each subframe. Therefore, the first information processing device can prevent the quality of reproduction and display of 3D data from being reduced.
[0129] Furthermore, for example, an information processing device (also referred to as a second information processing device) may include: an encoding unit that encodes a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data; a bitstream generating unit that generates a bitstream including the encoded data, and generates first information about display timing of the subframe based on the bitstream; and a file generating unit that generates a file for storing the bitstream, and stores the first information in the file as metadata. Furthermore, an information processing method (also referred to as a second information processing method) may include: encoding a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data; generating a bitstream including the encoded data; generating first information about display timing of the subframe based on the bitstream; generating a file for storing the bitstream; and storing the first information in the file as metadata.
[0130] By the second information processing device performing processing in this manner, the first information processing device can more easily grasp the display timing of each subframe. In other words, the second information processing device can cause the first information processing device to decode each sample to meet the display timing of each subframe. Therefore, the second information processing device can prevent the first information processing device from reducing the quality of reproduction and display of 3D data.
[0131] In the first information processing device and the second information processing device, the 3D data may be any data as long as it indicates the three-dimensional shape of an object in a three-dimensional space, and may be, for example, a point cloud. In this case, the subframe may be, for example, a point cloud frame. For example, in the case of a combined frame, a sample may correspond to the combined frame, and a subsample may correspond to a subframe. The method of encoding the 3D data (geometric information) may be, for example, G-PCC. Therefore, the bitstream may be, for example, a G-PCC bitstream, and the file may be, for example, a G-PCC file.
[0132] The information about the display timing of the subframe may be stored, for example, in a meta information storage area of the file. For example, the file may be ISOBMFF, and the first information may be stored in a corresponding SubSampleInformationBox. For example, codec_specific_parameters of SubSampleInformationBox may be expanded to store the first information.
[0133] The file may be ISOBMFF, and the first information may be stored in the corresponding SampleGroupBox. SampleGroupBox is a field for storing information assigned to a sample group consisting of a plurality of samples. Therefore, in this case, the first information (information about display timing) of the subsamples (subframes) in each sample constituting the sample group is stored in the SampleGroupBox. For example, the first information may be stored in the SubFrameTimingGroupEntry of the VolumetricVisualSampleGroupEntry. In other words, the first information may be assigned to each sample group.
[0134] <Method 1-1>
[0135] As from Figure 8 As shown in the second row from the top of the table in , the information about the display timing of the subframe (first information) may include an offset from the display timing of the sample (method 1-1).
[0136] Fig. 9 An example of the syntax of codec_specific_parameters of SubSampleInformationBox in this case is shown. Fig. 9 In the example of , for a flag value of "2" (flag == 2), subsample_delta is stored. Subsample_delta is a parameter indicating the difference between the decoding time of the sample to which the subsample (i.e., subframe) belongs and the decoding time of the subsample. In G-PCC, the decoding time and the display time (presentation time) are equivalent. In other words, subsample_delta indicates an offset from the sample display timing.
[0137] The first information may include a sign (positive or negative). Fig. 9 In the example of , for a flag value of "2" (flag == 2), a sign is stored together with subsample_delta. In this case, subsample_delta indicates the absolute value of the difference between the decoding timing of the sample to which the subsample belongs and the decoding timing of the subsample. The sign is a parameter indicating the sign (positive or negative) of the corresponding subsample_delta. For example, the sign may have a value of 0 to indicate positive, and a value of 1 to indicate negative. Instead of storing the sign, subsample_delta may have a negative value as well as 0 or a positive value. In this case, the sign may be eliminated.
[0138] subsample_delta may be subframe_time_offset, which is a parameter indicating the difference between the decoding timing of the sample to which the subframe belongs and the decoding timing of the subframe. In this case, a symbol may also be transmitted, and subframe_time_offset may have a negative value as well as 0 or a positive value. In other words, subframe_time_offset may be equal to subsample_delta, or may be equal to subsample_delta and a symbol.
[0139] Subframe_time_offset having a negative value (or a combination of subframe_time_offset and a sign) enables the subframe to be processed to be displayed before the display timing of the sample to which the subframe belongs. In other words, the order of subframes can be changed across samples.
[0140] The first information processing device acquires the sample decoding timing DT[n] and the first information from the G-PCC file, and derives the display timing of each subframe by adding the first information to the sample decoding timing DT[n]. Fig.10 An example of this derivation is shown. Fig.10 In the example of , the sample decoding timing DT[n] of sample a, sample b, and sample c are 0, 3, and 5, respectively. Here, it is assumed that subsample_delta(subframe_time_offset) can have a negative value. Fig.10 As shown in , the subsample_delta of the subsample assigned to the frame number FN#0 is 0. The subsample_delta of the subsample assigned to the frame number FN#1 is 1. The subsample_delta of the subsample assigned to the frame number FN#2 is 2. The subsample_delta of the subsample assigned to the frame number FN#3 is 0. The subsample_delta of the subsample assigned to the frame number FN#4 is -1. The subsample_delta of the subsample assigned to the frame number FN#5 is 2. The subsample_delta of the subsample assigned to the frame number FN#6 is 1. The subsample_delta of the subsample assigned to the frame number FN#7 is 2.
[0141] For example, the first information processing device derives the display timing of the subframe (decoding timing of the subframe) according to the following equation (1).
[0142] Subframe decoding time (= presentation time) = DT(n) + subsample_delta(1)
[0143] Therefore, the display timing (subsample decoding time) of the subsample assigned with the frame number FN#0 is DT(n)+subsample_delta=0+0=0. The display timing (subsample decoding time) of the subsample assigned with the frame number FN#1 is DT(n)+subsample_delta=0+1=1. The display timing (subsample decoding time) of the subsample assigned with the frame number FN#2 is DT(n)+subsample_delta=0+2=2. The display timing (subsample decoding time) of the subsample assigned with the frame number FN#3 is DT(n)+subsample_delta=3+0=3. The display timing (subsample decoding time) of the subsample assigned with the frame number FN#4 is DT(n)+subsample_delta=5-1=4. The display timing (subsample decoding time) of the subsample assigned with the frame number FN#5 is DT(n)+subsample_delta=3+2=5. The display timing (subsample decoding time) of the subsample assigned with frame number FN#6 is DT(n)+subsample_delta=5+1=6. The display timing (subsample decoding time) of the subsample assigned with frame number FN#7 is DT(n)+subsample_delta=5+2=7.
[0144] The first information processing device displays each sub-sample according to the display timing of the sub-sample derived in this way. Therefore, the first information processing device can present each sub-frame at the specified timing. Specifically, the first information processing device acquires and decodes each sample based on the first information about the display timing of each sub-frame stored as metadata in the G-PCC file to meet the display timing of the sub-frame, and constructs a point cloud for each sub-frame. This makes it possible to suppress the occurrence of reproduction delays and frame losses, and thus prevent the quality of reproduction and display of 3D data from being reduced.
[0145] In other words, the second information processing device stores the first information about the display timing of each subframe as metadata in the G-PCC file, thereby enabling the first information processing device to suppress the occurrence of reproduction delays and frame losses and thereby prevent a reduction in the quality of reproduction and display of 3D data.
[0146] In addition to the first information, second information about the display order of each subframe can also be stored in the file as metadata. Therefore, the first information processing device can easily grasp the display order of each subframe by simply referring to the second information without performing the above calculation. The second information may include a frame number indicating the display order in the entire sequence, or a frame index indicating the display order in a sample.
[0147] <Method 1-2>
[0148] like Figure 8 As shown in the bottom row of the table in , the information about the display timing of the subframe (first information) may include second information indicating the display order of the subframe and a duration indicating the length of the display period of the subframe (method 1-2). The second information may, for example, include a frame number indicating the display order of the subframe in the entire sequence.
[0149] Fig.11 An example of the syntax of codec_specific_parameters of SubSampleInformationBox in this case is shown. Fig.11 In the example, for the flag value "2" (flag == 2), the frame number and duration are stored. The frame number is as shown in Figure 3 The duration indicates the length of the display period of the subframe.
[0150] The first information processing device acquires the duration and frame number of each sub-frame from the G-PCC file, and displays each sub-sample (sub-frame) in sequence according to the frame number within a display period indicated by the duration. Fig.12 An example of this state is shown. Fig.12 In the example of , the display period of each subframe is "1", so that the display timings of the subsample assigned with the frame number FN#0 to the subsample assigned with the frame number FN#7 are 0 to 7, respectively.
[0151] Therefore, the first information processing device can display each subframe at the specified timing. Specifically, the first information processing device acquires and decodes each sample based on the first information about the display timing of each subframe stored as metadata in the G-PCC file to meet the display timing of the subframe, and constructs a point cloud for each subframe. This makes it possible to suppress the occurrence of reproduction delay and frame loss, and thus prevent the quality of reproduction and display of 3D data from being reduced.
[0152] In other words, the second information processing device stores the first information about the display timing of each subframe as metadata in the G-PCC file, thereby enabling the first information processing device to suppress the occurrence of reproduction delays and frame losses and thereby prevent a reduction in the quality of reproduction and display of 3D data.
[0153] The second information may include, for example, a frame index indicating a display order of subframes in the sample.
[0154] <Other methods>
[0155] The first information may be information directly indicating the display timing of each subframe. The first information may include only the subframe display order, without including information indicating an offset from the sample display timing or the length of the display period of the subframe.
[0156] The case where a file is configured by a single track, in which interrelated geometric information and attributes are stored in one track, has been described above. However, the present technology can also be applied to a case where a file is configured by multiple tracks, in which ISOBMFF stores interrelated geometric information and attributes in different tracks. In this case, the first information may be stored only in an attribute track in which, for example, a frame number attribute or a frame index attribute is stored (the first information may not be stored in a geometry track). The first information may be stored only in a geometry track in which geometric information is stored (the first information may not be stored in an attribute track in which attributes are stored). The first information may be stored in both a geometry track and an attribute track (the same information may be stored in both a geometry track and an attribute track).
[0157] In the description presented above, although an example in which ISOBMFF is applied as a file format has been described, the G-PCC bitstream may be stored in any type of file, which may be a format other than ISOBMFF. For example, the G-PCC bitstream may be configured to be stored in a Matroska media container. Fig.13 The main configuration examples of the Matroska media container are shown in .
[0158] In this case, for example, first information on display timing of each subframe may be stored as a newly defined element under the track entry element.
[0159] Also in this case, the first information about the display timing of each subframe is stored in the system layer of the Matroska media container. Therefore, the first information processing device can easily obtain the first information by referring to the system layer (without parsing the G-PCC bit stream). Therefore, the first information processing device can easily grasp the display timing of each subframe by referring to the system layer of the Matroska media container. Therefore, the first information processing device acquires and decodes each sample based on the first information to meet the display timing of the subframe, and constructs a point cloud for each subframe. This makes it possible to suppress the occurrence of reproduction delays and frame losses, and thus prevent a reduction in the quality of reproduction and display of 3D data.
[0160] In other words, the second information processing device stores the first information about the display timing of each subframe as metadata in the G-PCC file, thereby enabling the first information processing device to suppress the occurrence of reproduction delays and frame losses and thereby prevent a reduction in the quality of reproduction and display of 3D data.
[0161] <4. First Embodiment>
[0162] <File creation device>
[0163] The present technology described above can be applied to any device. Fig.14 : is a block diagram showing an example of the configuration of a file generating device as one type of information processing device to which the present technology is applied. Fig.14 The file generation device 300 shown in FIG. 1 is a device that encodes point cloud data by applying G-PCC and stores a G-PCC bit stream generated according to the encoding in ISOBMFF (G-PCC file).
[0164] In this case, by applying the above-mentioned technology, the file generating device 300 encodes a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data; generates a bit stream including the encoded data; generates first information about display timing of the subframes based on the bit stream; generates a file for storing the bit stream; and stores the first information as metadata in the file.
[0165] exist Fig.14 , showing the main components such as processing units or data flows, and Fig.14 The components shown in the figure are not necessarily all. In other words, in the file generating device 300, there may be Fig.14 The processing units are shown as blocks and may exist in Fig.14 Processing and data flow are shown using arrows etc.
[0166] like Fig.14 As shown in , the file generating device 300 includes a combined frame generating unit 311, an extracting unit 312, an encoding unit 313, a bit stream generating unit 314, and a file generating unit 315. The encoding unit 313 includes a geometry encoding unit 321, an attribute encoding unit 322, and a metadata generating unit 323.
[0167] The combined frame generation unit 311 combines a plurality of frames of the point cloud input to the file generation device 300 as subframes to generate a combined frame. The combined frame generation unit 311 provides the point cloud including the combined frame to the extraction unit 312.
[0168] The extraction unit 312 extracts geometric information and attributes from the point cloud provided by the combined frame generation unit 311. The extraction unit 312 provides the data of the extracted geometric information to the geometry encoding unit 321 of the encoding unit 313. The extraction unit 312 provides the extracted attributes to the attribute encoding unit 322 of the encoding unit 313.
[0169] The encoding unit 313 encodes the data of the point cloud. The geometry encoding unit 321 encodes the geometry information provided from the extraction unit 312 to generate a geometry bitstream. The geometry encoding unit 321 provides the generated geometry bitstream to the metadata generation unit 323. The geometry encoding unit 321 also provides the generated geometry bitstream to the attribute encoding unit 322. The attribute encoding unit 322 encodes the attributes provided from the extraction unit 312 to generate an attribute bitstream. The attribute encoding unit 322 provides the generated attribute bitstream to the metadata generation unit 323. The metadata generation unit 323 generates metadata to be stored in the G-PCC bitstream by referring to the provided geometry bitstream and attribute bitstream. The metadata generation unit 323 provides the generated metadata together with the geometry bitstream and the attribute bitstream to the bitstream generation unit 314. At this time, by applying the present technology described above in <3. Transmission of information on display timing>, the encoding unit 313 encodes a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data.
[0170] The bitstream generation unit 314 multiplexes the provided geometry bitstream, attribute bitstream, and metadata to generate a G-PCC bitstream. In other words, the bitstream generation unit 314 generates a bitstream (G-PCC bitstream) including the encoded data (geometry bitstream, attribute bitstream, and metadata). At this time, by applying the present technology described above in <3. Transmission of information on display timing>, the bitstream generation unit 314 generates first information on display timing of each subframe based on the bitstream.
[0171] For example, the first information may include a subframe_time_offset indicating an offset from the display timing of the sample. The subframe_time_offset may include a symbol. The first information may include second information indicating the display order of each subframe, and a duration indicating the length of the display period of the subframe. The second information may include a frame number indicating the display order in the entire sequence. The second information may include a frame index indicating the display order in the sample. The bitstream generation unit 314 provides the generated G-PCC bitstream and the first information to the file generation unit 315.
[0172] By applying the present technology described above in <3. Transmission of information about display timing>, the file generation unit 315 generates a G-PCC file (also referred to as a file or content file) for storing the provided G-PCC bitstream and metadata (first information). At this time, the file generation unit 315 can also store the second information about the display order of each subframe as metadata in the file. For example, the second information may include a frame number indicating the display order in the entire sequence. The second information may include a frame index indicating the display order in the sample. The file generation unit 315 outputs the G-PCC file thus generated to the outside of the file generation device 300.
[0173] With this configuration, the file generating device 300 can obtain the same effect as the case of <3. Transmission of information on display timing> Therefore, the file generating device 300 can prevent the quality of reproduction and display of 3D data from being reduced.
[0174] <Flow of file generation processing>
[0175] Will refer to Fig.15 The flowchart shown in describes an example of the flow of the file generation process executed by the file generation apparatus 300.
[0176] When the file generation process starts, in step S301 , the combined frame generation unit 311 of the file generation device 300 combines a plurality of frames as sub-frames for a point cloud to be encoded to generate a combined frame.
[0177] In step S302 , the extraction unit 312 extracts geometric information and attributes from the point cloud.
[0178] In step S303, the encoding unit 313 encodes the geometry information and attributes extracted in step S302 to generate a geometry bitstream and an attribute bitstream. The encoding unit 313 also generates metadata thereof. At this time, by applying the present technology described above in <3. Transmission of information on display timing>, the encoding unit 313 encodes a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data.
[0179] In step S304, the bitstream generation unit 314 multiplexes the geometry bitstream, attribute bitstream, and metadata generated in step S303 to generate a G-PCC bitstream. By applying the present technology described above in <3. Transmission of information on display timing>, the bitstream generation unit 314 also generates first information on display timing of each subframe based on the G-PCC bitstream.
[0180] For example, the first information may include a subframe_time_offset indicating an offset from the display timing of the sample. The subframe_time_offset may include a symbol. The first information may include second information indicating a display order of each subframe, and a duration indicating a length of a display period of the subframe. The second information may include a frame number indicating a display order in the entire sequence. The second information may include a frame index indicating a display order in the sample.
[0181] In step S305, by applying the present technology described above in <3. Transmission of information about display timing>, the file generation unit 315 generates a G-PCC file (i.e., a content file) for storing the G-PCC bitstream generated in step S304. Then, by applying the present technology described above in <3. Transmission of information about display timing>, the file generation unit 315 stores the first information as metadata in the G-PCC file (in the meta information storage area). At this time, the file generation unit 315 may also store the second information about the display order of each subframe as metadata in the file. For example, the second information may include a frame number indicating the display order in the entire sequence. The second information may include a frame index indicating the display order in the sample.
[0182] In step S306 , the file generating unit 315 outputs the G-PCC file (ie, content file) generated in step S305 to the outside of the file generating device 300 .
[0183] When the process of step S306 ends, the file generation process ends.
[0184] As described above, by applying the present technology to the file generation process, the file generation device 300 generates metadata including the spatial configuration information of the subgroup and the dependency information between the subgroup and the layer group, and stores the metadata in the system layer of the G-PCC file. Therefore, the file generation device 300 can obtain the same effect as the case of <3. Transmission of information on display timing>. Therefore, the file generation device 300 can prevent the quality of reproduction and display of 3D data from being reduced.
[0185] <Reproduction Device>
[0186] Fig.16 : is a block diagram showing an example of the configuration of a reproduction device as one aspect of an information processing device to which the present technology is applied. Fig.16 The reproduction device 400 shown in FIG. 4 is a device that decodes a G-PCC bit stream stored in an ISOBMFF (G-PCC file), reconfigures a point cloud, and performs rendering to generate presentation information.
[0187] By applying the present technology described above in <3. Transmission of information on display timing>, the reproduction device 400 extracts a G-PCC bit stream of a point cloud from a G-PCC file, and decodes the extracted G-PCC bit stream to construct a point cloud.
[0188] Fig.16 shows the main components such as processing units and data flows, and Fig.16 Not all components are shown. Fig.16 The processing units shown as blocks and not in Fig.16 The processes and data flows shown as arrows in FIG. 4 and the like may exist in the reproduction device 400 .
[0189] like Fig.16 As shown, the reproduction device 400 includes a file processing unit 411, a decoding unit 412, and a presentation information generating unit 413.
[0190] The file processing unit 411 acquires the G-PCC file input to the reproduction device 400, and performs processing on the G-PCC file. For example, the file processing unit 411 includes an extraction unit 421. The extraction unit 421 extracts a G-PCC bit stream from the provided G-PCC file. At this time, by applying the present technology described above in <3. Transmission of information about display timing>, the extraction unit 421 extracts samples including subframes from the G-PCC bit stream stored in the G-PCC file at a certain timing based on the first information about the display timing of the subframe stored in the G-PCC file as metadata to meet the display timing of the subframe. The extraction unit 421 provides the extracted samples and the first information to the decoding unit 412.
[0191] For example, the first information may include a subframe_time_offset indicating an offset from the display timing of the sample. The subframe_time_offset may include a symbol. The G-PCC file may also store second information about the display order of each subframe as metadata. The extraction unit 421 may extract the sample according to the display order of each subframe indicated in the second information. The second information may include a frame number indicating the display order in the entire sequence. The second information may include a frame index indicating the display order in the sample. The first information may include second information indicating the display order of each subframe, and a duration indicating the length of the display period of the subframe. The second information may include a frame number indicating the display order in the entire sequence. The second information may include a frame index indicating the display order in the sample.
[0192] By applying the present technology described above in <3. Transmission of information about display timing>, the decoding unit 412 decodes the sample (G-PCC bitstream). The decoding unit 412 provides the obtained sample and the first information to the presentation information generation unit 413 (the point cloud construction unit 441). For example, the decoding unit 412 includes a geometry decoding unit 431 and an attribute decoding unit 432. The geometry decoding unit 431 decodes the geometry bitstream included in the sample provided from the extraction unit 421 to obtain geometry information of the sample. The geometry decoding unit 431 provides the geometry information of the sample to the attribute decoding unit 432. The geometry decoding unit 431 also provides the geometry information and the first information of the sample to the presentation information generation unit 413 (the point cloud construction unit 441). The attribute decoding unit 432 also decodes the bitstream of the attribute included in the sample provided from the extraction unit 421 by using the geometry information provided from the geometry decoding unit 431 to obtain the attribute of the sample. The attribute decoding unit 432 supplies the attribute of the sample to the presentation information generating unit 413 (point cloud constructing unit 441 ).
[0193] The presentation information generation unit 413 constructs a point cloud by using the supplied samples (geometric information and attributes), and generates presentation information as information for presenting (eg, displaying) the point cloud. The presentation information generation unit 413 includes a point cloud construction unit 441 and a presentation processing unit 442.
[0194] By applying the present technology described above in <3. Transmission of information on display timing>, the point cloud construction unit 441 constructs a point cloud of a subframe included in a sample provided from the decoding unit 412 at a timing to satisfy the display timing of each subframe based on the first information provided from the decoding unit 412. The point cloud construction unit 441 provides the constructed point cloud to the rendering processing unit 442.
[0195] The rendering processing unit 442 generates rendering information by using the supplied point cloud. The rendering processing unit 442 outputs the generated rendering information to the outside of the reproduction device 400. For example, the rendering information is displayed on a monitor.
[0196] With this configuration, the reproducing device 400 can obtain the same effect as the case of <3. Transmission of information on display timing> Therefore, the reproducing device 400 can prevent a decrease in the quality of reproduction and display of 3D data.
[0197] <Flow of playback processing>
[0198] Will refer to Fig.17 The flowchart shown in describes an example of the reproduction process performed by the reproduction device 400.
[0199] When the reproduction process starts, in step S401, by applying the present technology described above in <3. Transmission of information about display timing>, the file processing unit 411 (extraction unit 421) of the reproduction device 400 obtains information about display timing from the track to be reproduced in the G-PCC file (i.e., content file).
[0200] In step S402, by applying the present technology described above in <3. Transmission of information about display timing>, the extraction unit 421 of the file processing unit 411 extracts samples including subframes from the G-PCC bit stream stored in the G-PCC file at a certain timing based on the first information about the display timing of the subframe stored as metadata in the G-PCC file to meet the display timing of the subframe.
[0201] For example, the first information may include a subframe_time_offset indicating an offset from the display timing of the sample. The subframe_time_offset may include a symbol. The G-PCC file may also store second information about the display order of each subframe as metadata. The extraction unit 421 may extract the sample according to the display order of each subframe indicated in the second information. The second information may include a frame number indicating the display order in the entire sequence. The second information may include a frame index indicating the display order in the sample. The first information may include second information indicating the display order of each subframe, and a duration indicating the length of the display period of the subframe. The second information may include a frame number indicating the display order in the entire sequence. The second information may include a frame index indicating the display order in the sample.
[0202] In step S403, the decoding unit 422 decodes the samples extracted in step S402 by applying the present technique described above in <3. Transmission of information on display timing> In step S404, the decoding unit 422 also extracts subsamples from the decoded samples.
[0203] In step S405, by applying the present technology described above in <3. Transmission of information about display timing>, the point cloud construction unit 441 constructs the point cloud of the subframe included in the sample decoded in step S403 at a certain timing based on the first information to meet the display timing of each subframe.
[0204] In step S406, the rendering processing unit 442 performs rendering or the like using the point cloud constructed in step S405 to generate rendering information. In step S407, the rendering processing unit 442 provides the rendering information to the outside of the reproduction device 400 for rendering.
[0205] When the process of step S407 ends, the reproduction process ends.
[0206] By performing the processing as described above, the reproduction device 400 can obtain the same effect as the case of <3. Transmission of information on display timing> Therefore, the reproduction device 400 can prevent the quality of reproduction and display of 3D data from being reduced.
[0207] <5.Supplement>
[0208] <Computer>
[0209] The above series of processes can be performed by hardware or software. When the series of processes are performed by software, the program constituting the software is installed on the computer. Here, the computer includes, for example, a computer built in special hardware and a general-purpose personal computer on which various programs are installed to be able to perform various functions.
[0210] Fig.18 : is a block diagram showing a configuration example of computer hardware that executes the above-described series of processes using a program.
[0211] exist Fig.18 In the illustrated computer 900 , a central processing unit (CPU) 901 , a read only memory (ROM) 902 , and a random access memory (RAM) 903 are connected to one another via a bus 904 .
[0212] An input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.
[0213] The input unit 911 is, for example, a keyboard, a mouse, a microphone, a touch panel, or an input terminal. The output unit 912 is, for example, a display, a speaker, or an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a nonvolatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0214] In the computer configured as described above, the CPU 901 loads the program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes the program, so that a series of processes are performed. Data and the like necessary for the CPU 901 to perform various processes are also appropriately stored in the RAM 903.
[0215] The program executed by the computer may be recorded in the removable medium 921 as a package medium, etc. and provided in such a form. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915 .
[0216] The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.
[0217] In addition, the program may be installed in advance in the ROM 902, the storage unit 913, or the like.
[0218] <Application target of this technology>
[0219] In the description presented above, although the case in which the present technology is mainly applied to a method in which a G-PCC bitstream is stored in ISOBMFF has been described, the present technology can be applied to any target, which is not limited to the above-mentioned example. In other words, the file used to store the G-PCC bitstream can use any format, which is not limited to the above-mentioned ISOBMFF and Matroska media containers. In addition, any encoding / decoding system can be used for 3D data (point cloud), which is not limited to G-PCC. Any format can be used for 3D data, which is a format other than point cloud. In other words, some or all of the specifications such as files, encoding / decoding systems, 3D data and generation / construction methods thereof may be different from the specifications of the above-mentioned examples, as long as they do not conflict with the characteristics of the present technology described above. Some of the above-mentioned processes and specifications may be omitted.
[0220] The present technology can be applied to any desired configuration. For example, the present technology can be applied to various electronic devices.
[0221] For example, the present technology may be implemented as a configuration of a part of a device, such as a processor of a system large-scale integration (LSI) circuit (e.g., a video processor), a module using multiple processors, etc. (e.g., a video module), a unit using multiple modules, etc. (e.g., a video unit), or a device having other functions added to the unit (e.g., a video device).
[0222] For example, the present technology can also be applied to a network system configured with multiple devices. The present technology can be implemented as, for example, cloud computing for sharing processing between multiple devices via a network. For example, the present technology can be implemented in a cloud service that provides services related to images (moving images) to any terminal such as a computer, an audio-visual (AV) device, a mobile information processing terminal, and an Internet of Things device.
[0223] In this specification, a system means a group of multiple components (devices, modules (components), etc.), and all the components may not be in the same housing. Therefore, multiple devices housed in separate housings and connected via a network and a single device housing multiple modules in a single housing are both systems.
[0224] <Fields and applications of this technology>
[0225] For example, systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical treatment, security, agriculture, animal husbandry, mining, beauty, factories, home appliances, weather and nature monitoring. Any purpose can be set.
[0226] For example, the present technology can be applied to systems and devices for providing viewing content, etc. For example, the present technology can be applied to systems and devices that can be used for transportation such as traffic condition monitoring and autonomous driving control. In addition, for example, the present technology can be applied to systems or devices that can be used for safety. For example, the present technology can be applied to systems and devices that can be used for automatic control of machines, etc. In addition, for example, the present technology can be applied to systems or devices that can be used for agriculture or animal husbandry. The present technology can also be applied to systems and devices that are used to monitor natural conditions such as volcanoes, forests, and oceans and wild animals. In addition, for example, the present technology can be applied to systems or devices that can be used for sports.
[0227] <Others>
[0228] As used herein, a "flag" is information for identifying multiple states, and includes not only information for identifying the two states of true (1) and false (0), but also information that allows identification of three or more states. Therefore, the value indicated by a "flag" can be, for example, a binary value of 1 or 0, or can be ternary or greater. In other words, the number of bits that make up the "flag" can be any number, such as 1 bit or multiple bits. It is also assumed that identification information (also including a flag) is included in the bit stream, or that difference information of the identification information relative to some reference information is included in the bit stream. Therefore, the "flag" and "identification information" in this article include not only information, but also difference information relative to the reference information.
[0229] In addition, various types of information (metadata, etc.) about the coded data (bitstream) can be transmitted or recorded in any form, as long as the information is associated with the coded data. Here, the term "association" means, for example, that another data can be used (linked) when processing one data. In other words, data associated with each other can be collected as one data, or can be separate data. For example, information associated with the coded data (image) can be transmitted on a transmission path different from the transmission path used for the coded data (image). For example, information associated with the coded data (image) can be recorded on a recording medium different from the recording medium used for the coded data (image) (or a recording area of the same recording medium). The "association" can be a part of the data rather than the entire data. For example, an image and information corresponding to an image can be associated with multiple frames, a frame, or any unit such as a part of a frame.
[0230] As used herein, terms such as "synthesize", "multiplex", "add", "integrate", "include", "store", "put into", "encapsulate" and "insert" may mean, for example, combining multiple objects into one object, such as combining encoded data and metadata into one data, and represent a method of the above-mentioned "association".
[0231] The embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made within the scope of the present technology without departing from the spirit and scope of the present technology.
[0232] For example, a configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, a configuration described as a plurality of devices (or processing units) may be integrated and configured as one device (or processing unit). Of course, configurations other than the configurations mentioned above may be added to the configuration of each device (or each processing unit). In addition, as long as the configuration and operation of the entire system are substantially the same as each other, some configurations of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or another processing unit).
[0233] For example, the above-mentioned program can be executed by any device. In this case, the device only needs to have necessary functions (eg, functional blocks) to obtain necessary information.
[0234] For example, each step in a flowchart may be performed by one device, or may be shared and performed by multiple devices. In addition, in the case where multiple processes are included in one step, one device may perform multiple processes, or multiple devices may share and perform multiple processes. In other words, multiple processes included in one step may also be performed as processes of multiple steps. On the other hand, processes described as multiple steps may also be performed collectively as one step.
[0235] For example, in a program executed by a computer, the processing of the steps describing the program may be performed in time series in the order described in this specification, or may be performed in parallel or individually at a desired timing such as when a call is made. That is, the processing of the individual steps may be performed in an order different from the above order as long as no contradiction occurs. In addition, the processing of the steps describing the program may be performed in parallel with the processing of another program, or may be performed in combination with the processing of another program.
[0236] For example, multiple technologies related to the present technology can be independently implemented as a single subject as long as there is no contradiction. Of course, any multiple present technologies can also be performed in combination. For example, some or all of the technologies described in other embodiments can also be combined to implement some or all of the technologies described in any embodiment. It is also possible to combine other technologies not described above to implement some or all of the technologies in any of the above technologies.
[0237] This technology can also be configured as follows:
[0238] (1) An information processing device comprising:
[0239] an extracting unit that extracts samples including the subframe from a bitstream stored in the file based on first information on display timing of the subframe stored in the file as metadata to satisfy display timing of the subframe;
[0240] a decoding unit, the decoding unit decoding the extracted samples; and
[0241] A construction unit is configured to construct a point cloud of the subframe included in the decoded sample based on the first information to meet a display timing of the subframe.
[0242] (2) The information processing device according to (1), wherein the first information includes subframe_time_offset indicating an offset from a display timing of the sample.
[0243] (3) The information processing device according to (2), wherein the subframe_time_offset includes a sign.
[0244] (4) The information processing device according to (2) or (3), wherein:
[0245] The file further stores second information on the display order of the sub-frames as the metadata, and
[0246] The extracting unit extracts the samples according to a display order of the subframes indicated by the second information.
[0247] (5) The information processing device according to (4), wherein the second information includes a frame number indicating a display order in the entire sequence.
[0248] (6) The information processing device according to (4), wherein the second information includes a frame index indicating a display order in the samples.
[0249] (7) The information processing device according to any one of (1) to (6), wherein the first information includes second information indicating a display order of the subframes and a duration indicating a length of a display period of the subframes.
[0250] (8) The information processing device according to (7), wherein the second information includes a frame number indicating a display order in the entire sequence.
[0251] (9) The information processing device according to (7), wherein the second information includes a frame index indicating a display order in the samples.
[0252] (10) The information processing device according to any one of (1) to (9), wherein the file is an International Organization for Standardization Base Media File Format (ISOBMFF), and the first information is stored in a SubSampleInformationBox.
[0253] (11) The information processing device according to any one of (1) to (9), wherein the file is an International Organization for Standardization Base Media File Format (ISOBMFF), and the first information is stored in a SampleGroupBox.
[0254] (12) An information processing method comprising:
[0255] extracting samples including the subframe from a bitstream stored in the file to meet the display timing of the subframe based on first information about the display timing of the subframe stored in the file as metadata;
[0256] decoding the extracted samples; and
[0257] Based on the first information, a point cloud of the subframe included in the decoded sample is constructed to meet the display timing of the subframe.
[0258] (21) An information processing device comprising:
[0259] an encoding unit that encodes a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data;
[0260] a bit stream generating unit that generates a bit stream including the encoded data and generates first information on display timing of the subframe based on the bit stream; and
[0261] A file generating unit is configured to generate a file for storing the bit stream and store the first information as metadata in the file.
[0262] (22) The information processing device according to (21), wherein the first information includes subframe_time_offset indicating an offset from a display timing of the sample.
[0263] (23) The information processing device according to (22), wherein the subframe_time_offset includes a sign.
[0264] (24) The information processing device according to (22) or (23), wherein the file generation unit further stores second information on a display order of the sub-frames in the file as the metadata.
[0265] (25) The information processing device according to (24), wherein the second information includes a frame number indicating a display order in the entire sequence.
[0266] (26) The information processing device according to (24), wherein the second information includes a frame index indicating a display order in the samples.
[0267] (27) The information processing device according to any one of (21) to (26), wherein the first information includes second information indicating a display order of the subframes and a duration indicating a length of a display period of each subframe.
[0268] (28) The information processing device according to (27), wherein the second information includes a frame number indicating a display order in the entire sequence.
[0269] (29) The information processing device according to (27), wherein the second information includes a frame index indicating a display order in the samples.
[0270] (30) An information processing device according to any one of (21) to (29), wherein the file generation unit is International Organization for Standardization Base Media File Format (ISOBMFF), and the first information is stored in a SubSampleInformationBox of the file.
[0271] (31) The information processing device according to any one of (21) to (29), wherein the file generation unit is International Organization for Standardization Base Media File Format (ISOBMFF), and the first information is stored in a SampleGroupBox of the file.
[0272] (32) An information processing method comprising:
[0273] encoding a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data;
[0274] generating a bit stream comprising the encoded data;
[0275] generating first information about display timing of the subframe based on the bit stream;
[0276] generating a file for storing the bitstream; and
[0277] The first information is stored in the file as metadata.
[0278] [reference numerals list]
[0279] 300 file generation device
[0280] 311 combined frame generation unit
[0281] 312 Extraction Unit
[0282] 313 coding units
[0283] 314 Bitstream generation unit
[0284] 315 Document Generation Unit
[0285] 321 Geometry Coding Unit
[0286] 322 attribute coding unit
[0287] 323 Metadata Generation Unit
[0288] 400 Reproduction Device
[0289] 411 File Processing Unit
[0290] 412 decoding unit
[0291] 413 Presentation information generation unit
[0292] 421 Extraction Unit
[0293] 431 Geometry Decoding Unit
[0294] 432 attribute decoding unit
[0295] 441 Point Cloud Construction Unit
[0296] 442 Presentation Processing Unit
[0297] 900 Computer< / subsampleinformationbox> < / isobmff>
Claims
1. An information processing device, comprising: an extracting unit that extracts samples including the subframe from a bitstream stored in the file based on first information on display timing of the subframe stored in the file as metadata to satisfy display timing of the subframe; A decoding unit, wherein the decoding unit decodes the extracted samples; as well as A construction unit is configured to construct a point cloud of the subframe included in the decoded sample based on the first information to meet a display timing of the subframe.
2. The information processing device according to claim 1, wherein: The first information includes subframe_time_offset indicating an offset from a display timing of the sample.
3. The information processing device according to claim 2, wherein: The subframe_time_offset includes a sign.
4. The information processing device according to claim 2, wherein: The file further stores second information on the display order of the sub-frames as the metadata, and The extracting unit extracts the samples according to a display order of the subframes indicated by the second information.
5. The information processing device according to claim 1, wherein: The first information includes second information indicating a display order of the sub-frames and a duration indicating a length of a display period of the sub-frames.
6. The information processing device according to claim 5, wherein: The second information includes frame numbers indicating a display order in the entire sequence.
7. The information processing device according to claim 5, wherein: The second information includes a frame index indicating a display order in the samples.
8. The information processing device according to claim 1, wherein: The file is an International Organization for Standardization Base Media File Format (ISOBMFF), and the first information is stored in a SubSampleInformationBox.
9. The information processing device according to claim 1, wherein: The file is an International Organization for Standardization Base Media File Format (ISOBMFF), and the first information is stored in a SampleGroupBox.
10. An information processing method, comprising: extracting samples including the subframe from a bitstream stored in the file to meet the display timing of the subframe based on first information about the display timing of the subframe stored in the file as metadata; Decoding the extracted samples; as well as A point cloud of the subframe included in the decoded sample is constructed based on the first information to meet display timing of the subframe.
11. An information processing device, comprising: an encoding unit that encodes a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data; a bit stream generating unit that generates a bit stream including the encoded data and generates first information on display timing of the subframe based on the bit stream; as well as A file generating unit is configured to generate a file for storing the bit stream and store the first information as metadata in the file.
12. The information processing device according to claim 11, wherein: The first information includes subframe_time_offset indicating an offset from a display timing of the sample.
13. The information processing device according to claim 12, wherein: The subframe_time_offset includes a sign.
14. The information processing device according to claim 12, wherein: The file generation unit further stores second information on the display order of the sub-frames in the file as the metadata.
15. The information processing device according to claim 11, wherein: The first information includes second information indicating a display order of the subframes and a duration indicating a length of a display period of each subframe.
16. The information processing device according to claim 15, wherein: The second information includes frame numbers indicating a display order in the entire sequence.
17. The information processing device according to claim 15, wherein: The second information includes a frame index indicating a display order in the samples.
18. The information processing device according to claim 11, wherein: The file generation unit is International Organization for Standardization Base Media File Format (ISOBMFF), and the first information is stored in a SubSampleInformationBox of the file.
19. The information processing device according to claim 11, wherein: The file generation unit is International Organization for Standardization Base Media File Format (ISOBMFF), and the first information is stored in a SampleGroupBox of the file.
20. An information processing method, comprising: encoding a combined frame in which a plurality of frames are combined as subframes as samples to generate encoded data; generating a bit stream comprising the encoded data; generating first information about display timing of the subframe based on the bit stream; generating a file for storing the bit stream; as well as The first information is stored in the file as metadata.
