Three-dimensional data storage method, three-dimensional data acquisition method, three-dimensional data storage device, and three-dimensional data acquisition device

Abstract

The invention provides a three-dimensional data storage method, a three-dimensional data acquisition method, a three-dimensional data storage device, and a three-dimensional data acquisition device. A three-dimensional data storage method for obtaining one or more units in which encoding information generated by encoding three-dimensional data is stored, the one or more units having a format corresponding to an encoding method used for encoding the three-dimensional data among a plurality of encoding methods; and saving the one or more units in a file, where the file follows a predetermined file format, each of the one or more units including a header and a payload, the payload including encoded data or metadata, the header includes information indicating a type of data included in the payload.

Description

[0001] This application is a division of an invention patent application with an application date of August 6, 2019, application number 201980051434.5, and invention name “Three-dimensional data storage method, three-dimensional data acquisition method, three-dimensional data storage device, and three-dimensional data acquisition device”. Technical Field

[0002] The present disclosure relates to a three-dimensional data storage method, a three-dimensional data acquisition method, a three-dimensional data storage device, and a three-dimensional data acquisition device. Background Art

[0003] In the future, devices and services that make use of 3D data will become more common in large fields such as computer vision, map information, monitoring, infrastructure inspection, or image distribution for autonomous operation of cars or robots. 3D data is obtained by various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple single-lens cameras.

[0004] As a method of expressing three-dimensional data, there is a method called point cloud, which expresses the shape of a three-dimensional structure through a group of points in a three-dimensional space. The position and color of the point group are stored in the point cloud. Although point cloud is expected to become the mainstream method of expressing three-dimensional data, the amount of point group data is very large. Therefore, in the accumulation or transmission of three-dimensional data, it is necessary to compress the data volume through encoding, just like two-dimensional dynamic images (as an example, there are MPEG-4AVC or HEVC standardized by MPEG).

[0005] Furthermore, compression of point clouds is partially supported by a public library (PointCloud Library) that performs point cloud association processing.

[0006] Furthermore, there is a known technique for searching for facilities around a vehicle using three-dimensional map data and displaying the facilities (for example, refer to Patent Document 1).

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1 International Publication No. 2014 / 020663 Summary of the invention

[0010] Problems to be solved by the invention

[0011] In an apparatus for processing three-dimensional data, it is possible to reduce the amount of processing or increase the speed of processing.

[0012] An object of the present disclosure is to provide a three-dimensional data storage method, a three-dimensional data acquisition method, a three-dimensional data storage device, or a three-dimensional data acquisition device that can reduce the processing amount or speed up the processing in a device for processing three-dimensional data.

[0013] Means for Solving the Problem

[0014] A three-dimensional data storage method according to one aspect of the present disclosure acquires one or more units storing encoded information generated by encoding three-dimensional data, the one or more units having a format corresponding to an encoding method used in the encoding of the three-dimensional data among a plurality of encoding methods; and stores the one or more units in a file, wherein the file follows a predetermined file format, each of the one or more units includes a header and a payload, the payload includes encoded data or metadata, and the header includes information indicating the type of data included in the payload.

[0015] A three-dimensional data acquisition method according to one aspect of the present disclosure acquires one or more units storing encoded information generated by encoding three-dimensional data, the one or more units having a format corresponding to an encoding method used in the encoding of the three-dimensional data among a plurality of encoding methods; and acquires the one or more units from the file, wherein the file follows a predetermined file format, each of the one or more units includes a header and a payload, the payload includes encoded data or metadata, and the header includes information indicating the type of data included in the payload.

[0016] Advantageous Effects of the Invention

[0017] The present disclosure can provide a three-dimensional data storage method, a three-dimensional data acquisition method, a three-dimensional data storage device, or a three-dimensional data acquisition device that can reduce the processing amount or speed up the processing in a device for processing three-dimensional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 FIG. is a diagram showing the configuration of a three-dimensional data encoding / decoding system according to Embodiment 1.

[0019] Figure 2 FIG. is a diagram showing a structural example of point cloud data according to Embodiment 1.

[0020] Figure 3 FIG. is a diagram showing a structural example of a data file describing point cloud data information according to Embodiment 1.

[0021] Figure 4 FIG. is a diagram showing the types of point cloud data according to Embodiment 1.

[0022] Figure 5 This is a diagram showing the structure of the first encoding unit according to Embodiment 1.

[0023] Figure 6 This is a block diagram of the first encoding unit according to Embodiment 1.

[0024] Figure 7 This is a diagram showing the structure of the first decoding unit according to Embodiment 1.

[0025] Figure 8 This is a block diagram of the first decoding unit according to Embodiment 1.

[0026] Figure 9 This is a diagram showing the structure of the second encoding unit according to Embodiment 1.

[0027] Figure 10 This is a block diagram of the second encoding unit according to Embodiment 1.

[0028] Figure 11 This is a diagram showing the structure of the second decoding unit according to Embodiment 1.

[0029] Figure 12 This is a block diagram of the second decoding unit according to Embodiment 1.

[0030] Figure 13 This is a diagram showing the protocol stack related to PCC encoded data according to Embodiment 1.

[0031] Figure 14 This is a diagram showing the protocol stack according to Embodiment 1.

[0032] Figure 15 This is a diagram showing an example of the syntax of the NAL unit according to Embodiment 1.

[0033] Figure 16 This is a diagram showing an example of the syntax of the NAL unit header according to Embodiment 1.

[0034] Figure 17 This is a diagram showing an example of the semantics of pcc_codec_type according to Embodiment 1.

[0035] Figure 18 This is a diagram showing an example of the semantics of pcc_nal_unit_type according to Embodiment 1.

[0036] Figure 19 This is a flowchart of the encoding process according to Embodiment 1.

[0037] Figure 20 This is a flowchart of the decoding process of the second decoding unit according to Embodiment 1.

[0038] Figure 21 This is a flowchart of the decoding process of the first decoding unit according to Embodiment 1.

[0039] Figure 22 It is a diagram showing the protocol stack of Embodiment 2.

[0040] Figure 23 It is a diagram showing a syntax example of NAL units for Codec 2 of Embodiment 2.

[0041] Figure 24 It is a diagram showing a syntax example of the NAL unit header for Codec 2 of Embodiment 2.

[0042] Figure 25 It is a diagram showing a semantic example of codec2_nal_unit_type of Embodiment 2.

[0043] Figure 26 It is a diagram showing a syntax example of NAL units for Codec 1 of Embodiment 2.

[0044] Figure 27 It is a diagram showing a syntax example of the NAL unit header for Codec 1 of Embodiment 2.

[0045] Figure 28 It is a diagram showing a semantic example of codec1_nal_unit_type of Embodiment 2.

[0046] Figure 29 It is a flowchart of the encoding process of Embodiment 2.

[0047] Figure 30 It is a flowchart of the decoding process of Embodiment 2.

[0048] Figure 31 It is a diagram showing the protocol stack of Embodiment 3.

[0049] Figure 32 It is a diagram showing a syntax example of NAL units of Embodiment 3.

[0050] Figure 33 It is a diagram showing a syntax example of the NAL unit header of Embodiment 3.

[0051] Figure 34 It is a diagram showing a semantic example of pcc_nal_unit_type of Embodiment 3.

[0052] Figure 35 It is a flowchart of the encoding process of Embodiment 3.

[0053] Figure 36 It is a flowchart of the decoding process of Embodiment 3.

[0054] Figure 37 It is a flowchart of the encoding process of a modification example of the embodiment.

[0055] Figure 38 Flowchart of the decoding process, which is a variant of the embodiment.

[0056] Figure 39 Block diagram of the encoding section of Embodiment 4.

[0057] Figure 40 Block diagram of the decoding section of Embodiment 4.

[0058] Figure 41 Flowchart of the encoding process of Embodiment 4.

[0059] Figure 42 Flowchart of the decoding process of Embodiment 4.

[0060] Figure 43 Diagram showing the basic structure of ISOBMFF related to Embodiment 5.

[0061] Figure 44 Diagram showing the protocol stack related to Embodiment 5.

[0062] Figure 45 Diagram showing an example of storing NAL units in a file for Codec 1 related to Embodiment 5.

[0063] Figure 46 Diagram showing an example of storing NAL units in a file for Codec 2 related to Embodiment 5.

[0064] Figure 47 Diagram showing the configuration of the first multiplexing section related to Embodiment 5.

[0065] Figure 48 Diagram showing the configuration of the first demultiplexing section related to Embodiment 5.

[0066] Figure 49 Diagram showing the configuration of the second multiplexing section related to Embodiment 5.

[0067] Figure 50 Diagram showing the configuration of the second demultiplexing section related to Embodiment 5.

[0068] Figure 51 Flowchart of the process performed by the first multiplexing section related to Embodiment 5.

[0069] Figure 52 Flowchart of the process performed by the second multiplexing section related to Embodiment 5.

[0070] Figure 53 Flowchart of the process performed by the first demultiplexing section and the first decoding section related to Embodiment 5.

[0071] Figure 54 It is a flowchart showing the processing performed by the second demultiplexing unit and the second decoding unit regarding Embodiment 5.

[0072] Figure 55 It is a diagram showing the configuration of the encoding unit and the third multiplexing unit regarding Embodiment 6.

[0073] Figure 56 It is a diagram showing the configuration of the third demultiplexing unit and the decoding unit regarding Embodiment 6.

[0074] Figure 57 It is a flowchart showing the processing performed by the third multiplexing unit regarding Embodiment 6.

[0075] Figure 58 It is a flowchart showing the processing performed by the third demultiplexing unit and the decoding unit regarding Embodiment 6.

[0076] Figure 59 It is a flowchart showing the processing performed by the three-dimensional data storage device regarding Embodiment 6.

[0077] Figure 60 It is a flowchart showing the processing performed by the three-dimensional data acquisition device regarding Embodiment 6. Detailed Embodiment

[0078] A three-dimensional data storage method according to an aspect of the present disclosure obtains one or more units storing an encoded stream obtained by encoding point cloud data, stores the one or more units in a file, and during the storage, stores information indicating that the data stored in the file is data obtained by encoding point cloud data in the control information of the file.

[0079] Thereby, in a device that processes a file generated by this three-dimensional data storage method, it is possible to refer to the control information of the file and determine at an early stage whether the data stored in the file is encoded data of point cloud data. Thereby, it is possible to reduce the processing amount of the device or speed up the processing.

[0080] For example, the information may further indicate an encoding method used for encoding the point cloud data among a first encoding method and a second encoding method.

[0081] Thereby, in a device that processes a file generated by this three-dimensional data storage method, it is possible to refer to the control information of the file and determine at an early stage the codec used for the data stored in the file. Thereby, it is possible to reduce the processing amount of the device or speed up the processing.

[0082] For example, it may also be that the first encoding method encodes the position information indicating the positions of point cloud data represented by an N-ary tree (where N is an integer of 2 or more), and encodes the attribute information using the position information, and the second encoding method generates a two-dimensional image from the point cloud data and encodes the two-dimensional image using an image encoding method.

[0083] For example, it may also be that the file complies with ISOBMFF (ISO based media file format).

[0084] A three-dimensional data acquisition method according to one aspect of the present disclosure acquires a file including one or more units storing an encoded stream obtained by encoding point cloud data, obtains the one or more units from the file, and control information of the file includes information indicating that the data stored in the file is data obtained by encoding point cloud data.

[0085] Thereby, this three-dimensional data acquisition method can early determine whether the data stored in the file is encoded data of point cloud data. Thereby, it is possible to reduce the processing amount or speed up the processing of the apparatus performing this three-dimensional data acquisition method or a subsequent-stage apparatus.

[0086] For example, it may also be that the information further indicates the encoding method used for the encoding in the first encoding method and the second encoding method.

[0087] Thereby, this three-dimensional data acquisition method can refer to the control information of the file and early determine the codec used for the data stored in the file. Thereby, it is possible to reduce the processing amount or speed up the processing of the apparatus performing this three-dimensional data acquisition method or a subsequent-stage apparatus.

[0088] For example, it may also be that based on the information, the data encoded by any one of the encoding methods is obtained from the encoded point cloud data including the data encoded by the first encoding method and the data encoded by the second encoding method.

[0089] For example, it may also be that the first encoding method encodes the position information indicating the positions of point cloud data represented by an N-ary tree (where N is an integer of 2 or more), and encodes the attribute information using the position information, and the second encoding method generates a two-dimensional image from the point cloud data and encodes the two-dimensional image using an image encoding method.

[0090] For example, it may also be that the file complies with ISOBMFF (ISO based media file format).

[0091] In addition, a three-dimensional data storage device according to an aspect of the present disclosure includes a processor and a memory. The processor uses the memory to obtain one or more units that store an encoded stream obtained by encoding point cloud data, saves the one or more units in a file, and during the saving, saves information indicating that the data saved in the file is data obtained by encoding point cloud data in control information of the file.

[0092] Thereby, in a device that processes a file generated by this three-dimensional data storage method, it is possible to refer to the control information of the file and determine at an early stage whether the data saved in the file is encoded data of point cloud data. Thereby, it is possible to reduce the processing amount of the device or speed up the processing.

[0093] In addition, a three-dimensional data acquisition device according to an aspect of the present disclosure includes a processor and a memory. The processor uses the memory to obtain a file that stores one or more units that store an encoded stream obtained by encoding point cloud data, and obtains the one or more units from the file. The control information of the file includes information indicating that the data saved in the file is data obtained by encoding point cloud data.

[0094] Thereby, this three-dimensional data acquisition device can determine at an early stage whether the data saved in the file is encoded data of point cloud data. Thereby, it is possible to reduce the processing amount of this three-dimensional data acquisition device or a subsequent device or speed up the processing.

[0095] In addition, these general or specific aspects can be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and can be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0096] Hereinafter, embodiments will be specifically described with reference to the drawings. In addition, all of the embodiments to be described below are specific examples showing the present disclosure. The numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of the constituent elements, steps, and the order of the steps shown in the following embodiments are all examples, and the gist thereof is not to limit the present disclosure. In addition, among the constituent elements of the following embodiments, the constituent elements not described in the technical solution showing the most general concept are described as optional constituent elements.

[0097] (Embodiment 1)

[0098] When using the encoded data of point clouds for actual devices or services, in order to suppress network bandwidth, it is desired to transmit and receive the required information according to the usage. However, such a function does not exist in the encoding structures of three-dimensional data to date, and thus there is no corresponding encoding method either.

[0099] In the present embodiment, a three-dimensional data encoding method and a three-dimensional data encoding device for providing a function of transmitting and receiving the required information according to the usage in the encoded data of three-dimensional point clouds, a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data, a three-dimensional data multiplexing method for multiplexing the encoded data, and a three-dimensional data transmission method for transmitting the encoded data will be described.

[0100] In particular, currently, as encoding methods (encoding formats) for point cloud data, a first encoding method and a second encoding method have been studied, but a method for defining the structure of the encoded data and saving the encoded data to a system format does not exist. In this case, there are problems such as being unable to directly perform MUX processing (multiplexing) in the encoding section, or transmitting or storing.

[0101] In addition, a method for supporting a format in which two codecs, namely the first encoding method and the second encoding method, coexist, such as PCC (Point Cloud Compression), does not exist to date.

[0102] In the present embodiment, the structure of PCC encoded data in which two codecs, namely the first encoding method and the second encoding method, coexist and a method for saving the encoded data to a system format will be described.

[0103] First, the structure of the three-dimensional data (point cloud data) encoding and decoding system of the present embodiment will be described. Figure 1 It is a diagram showing a structural example of the three-dimensional data encoding and decoding system of the present embodiment. As Figure 1 shown, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection section 4604.

[0104] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data as three-dimensional data. In addition, the three-dimensional data encoding system 4601 can be a three-dimensional data encoding device implemented by a single device, or a system implemented by multiple devices. Further, the three-dimensional data encoding device may also be included in a part of the multiple processing sections included in the three-dimensional data encoding system 4601.

[0105] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a prompting unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.

[0106] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603 and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data based on the sensor information and outputs the point cloud data to the encoding unit 4613.

[0107] The prompting unit 4612 prompts the user with the sensor information or the point cloud data. For example, the prompting unit 4612 displays information or an image based on the sensor information or the point cloud data.

[0108] The encoding unit 4613 encodes (compresses) the point cloud data and outputs the obtained encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, the sensor information.

[0109] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data, control information, and additional information input from the encoding unit 4613. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0110] The input / output unit 4615 (for example, a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 4616 (or an application program execution unit) controls each processing unit. That is, the control unit 4616 performs controls such as encoding and multiplexing.

[0111] In addition, the sensor information can be input to the encoding unit 4613 or the multiplexing unit 4614. Additionally, the input / output unit 4615 can also directly output the point cloud data or the encoded data to the outside.

[0112] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604.

[0113] The three-dimensional data decoding system 4602 generates point cloud data as three-dimensional data by decoding the encoded data or the multiplexed data. In addition, the three-dimensional data decoding system 4602 can be a three-dimensional data decoding device implemented by a single device or a system implemented by multiple devices. Additionally, the three-dimensional data decoding device can also include a part of the multiple processing units included in the three-dimensional data decoding system 4602.

[0114] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621, an input / output unit 4622, a demultiplexing unit 4623, a decoding unit 4624, a presentation unit 4625, a user interface 4626, and a control unit 4627.

[0115] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603.

[0116] The input / output unit 4622 acquires a transmission signal, decodes the multiplexed data (file format or packet) according to the transmission signal, and outputs the multiplexed data to the demultiplexing unit 4623.

[0117] The demultiplexing unit 4623 acquires encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 4624.

[0118] The decoding unit 4624 reconstructs point cloud data by decoding the encoded data.

[0119] The presentation unit 4625 presents the point cloud data to the user. For example, the presentation unit 4625 displays information or an image based on the point cloud data. The user interface 4626 acquires an instruction based on the user's operation. The control unit 4627 (or application execution unit) controls each processing unit. That is, the control unit 4627 controls demultiplexing, decoding, presentation, etc.

[0120] In addition, the input / output unit 4622 may also directly acquire point cloud data or encoded data from the outside. Additionally, the presentation unit 4625 may also acquire additional information such as sensor information, and present information based on the additional information. Additionally, the presentation unit 4625 may also perform presentation based on the instruction of the user acquired by the user interface 4626.

[0121] The sensor terminal 4603 generates the information acquired by the sensor, i.e., sensor information. The sensor terminal 4603 is a terminal equipped with a sensor or a camera, such as a moving body like a car, a flying object like an airplane, a mobile terminal, or a camera.

[0122] The sensor information that can be acquired by the sensor terminal 4603 is, for example, (1) the distance between the sensor terminal 4603 and the object, or the reflectivity of the object, obtained by LIDAR, millimeter-wave radar, or infrared sensor, (2) the distance between the camera and the object or the reflectivity of the object obtained from multiple monocular camera images or stereo camera images. Additionally, the sensor information may also include the posture, orientation, rotation (angular velocity), position (GPS information or altitude), speed, or acceleration of the sensor. Additionally, the sensor information may also include temperature, air pressure, humidity, or magnetism.

[0123] The external connection unit 4604 is implemented by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, or broadcasting, etc.

[0124] Next, the point cloud data will be described. Figure 2 It is a diagram showing the structure of the point cloud data. Figure 3 It is a diagram showing an example of the structure of a data file describing the information of the point cloud data.

[0125] The point cloud data contains data of multiple points. The data of each point contains position information (three-dimensional coordinates) and attribute information corresponding to the position information. A group of multiple such points is called a point cloud. For example, the point cloud represents the three-dimensional shape of an object (target).

[0126] Sometimes, position information such as three-dimensional coordinates (Position) is also called geometry. In addition, the data of each point can also contain attribute information (attribute) of multiple attribute categories. The attribute categories are, for example, color or reflectivity, etc.

[0127] One piece of attribute information can be corresponding to one piece of position information, or attribute information with multiple different attribute categories can be corresponding to one piece of position information. In addition, multiple pieces of attribute information of the same attribute category can be corresponding to one piece of position information.

[0128] Figure 3 The example of the structure of the data file shown is an example of the case where the position information and the attribute information are in one-to-one correspondence, and represents the position information and the attribute information of N points constituting the point cloud data.

[0129] The position information is, for example, the information of the three axes of x, y, and z. The attribute information is, for example, the RGB color information. As a representative data file, there is a ply file, etc.

[0130] Next, the types of the point cloud data will be described. Figure 4 It is a diagram showing the types of the point cloud data. As Figure 4 shown, the point cloud data contains static objects and dynamic objects.

[0131] The static object is the three-dimensional point cloud data at any time (a certain moment). The dynamic object is the three-dimensional point cloud data that changes with time. Hereinafter, the three-dimensional point cloud data at a certain moment is called a PCC frame or a frame.

[0132] The object can be a point cloud with a certain degree of restricted area like ordinary image data, or a large-scale point cloud with an unrestricted area like map information.

[0133] In addition, there are point cloud data of various densities, and there may also be sparse point cloud data and dense point cloud data.

[0134] Hereinafter, the details of each processing unit will be described. Sensor information is obtained by various methods such as distance sensors such as LIDAR or rangefinders, combinations of stereo cameras or multiple monocular cameras, etc. The point cloud data generation unit 4618 generates point cloud data based on the sensor information obtained by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as point cloud data and attaches attribute information for the position information.

[0135] The point cloud data generation unit 4618 may also process the point cloud data when generating the position information or attaching the attribute information. For example, the point cloud data generation unit 4618 may reduce the data volume by deleting point clouds with duplicate positions. In addition, the point cloud data generation unit 4618 may transform the position information (position transformation, rotation, or normalization, etc.), and may also render the attribute information.

[0136] In addition, in Figure 1 the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but may also be independently provided outside the three-dimensional data encoding system 4601.

[0137] The encoding unit 4613 encodes the point cloud data based on a pre-specified encoding method, thereby generating encoded data. There are roughly the following two types of encoding methods. The first type is an encoding method that uses position information, and this encoding method will be described as the first encoding method hereafter. The second type is an encoding method that uses a video codec, and this storage method will be described as the second storage method hereafter.

[0138] The decoding unit 4624 decodes the encoded data based on a pre-specified encoding method, thereby decoding the point cloud data.

[0139] The multiplexing unit 4614 multiplexes the encoded data by using an existing multiplexing method, thereby generating multiplexed data. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 4614 also multiplexes other media such as images, sounds, subtitles, applications, files, etc., or reference time information. In addition, the multiplexing unit 4614 may also multiplex the attribute information associated with the sensor information or the point cloud data.

[0140] As the multiplexing method or file format, there are ISOBMFF, MPEG-DASH which is a transmission method based on ISOBMFF, MMT, MPEG-2TS Systems, RMP, etc.

[0141] The inverse multiplexing unit 4623 extracts PCC encoded data, other media, time information, etc. from the multiplexed data.

[0142] The input / output unit 4615 transmits the multiplexed data using a method consistent with a transmission medium such as broadcasting or communication or a storage medium. The input / output unit 4615 can communicate with other devices via the Internet or communicate with a storage unit such as a cloud server.

[0143] As a communication protocol, http, ftp, TCP, UDP, etc. are used. Either a PULL-type communication method or a PUSH-type communication method can be used.

[0144] Either wired transmission or wireless transmission can be used. As wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), or coaxial cable, etc. are used. As wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), or millimeter wave, etc. are used.

[0145] In addition, as a broadcasting method, for example, DVB-T2, DVB-S2, DVB-C2, ATSC 3.0, or ISDB-S3, etc. are used.

[0146] Figure 5 It is a diagram showing the structure of the first encoding unit 4630 which is an example of the encoding unit 4613 that performs encoding by the first encoding method. Figure 6 It is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. The first encoding unit 4630 includes a position information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0147] The first encoding unit 4630 is characterized by performing encoding while being aware of the three-dimensional structure. In addition, the first encoding unit 4630 is characterized in that the attribute information encoding unit 4632 performs encoding using the information obtained from the position information encoding unit 4631. The first encoding method is also called GPCC (Geometry based PCC).

[0148] The point cloud data is PCC point cloud data such as a PLY file or PCC point cloud data generated based on sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData). The position information is input to the position information encoding unit 4631, the attribute information is input to the attribute information encoding unit 4632, and the additional information is input to the additional information encoding unit 4633.

[0149] The position information encoding unit 4631 generates encoded position information (Compressed Geometry) as encoded data by encoding the position information. For example, the position information encoding unit 4631 encodes the position information using an N-ary tree structure such as an octree. Specifically, in the octree, the object space is divided into 8 nodes (sub-spaces), and 8-bit information (occupancy encoding) indicating whether each node contains a point group is generated. In addition, the node containing the point group is further divided into 8 nodes, and 8-bit information indicating whether each of the 8 nodes contains a point group is generated. This process is repeated until it reaches a predetermined level or a threshold of the number of point groups contained in the node.

[0150] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute) as encoded data by encoding using the structure information generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 determines a reference point (reference node) to be referred to in the encoding of the object points (object nodes) of the processing target based on the octree structure generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 refers to a node in the octree of neighboring nodes or adjacent nodes where the parent node of the object node is the same as the parent node of the reference node. In addition, the method of determining the reference relationship is not limited to this.

[0151] In addition, the encoding process of the attribute information may include at least one of quantization processing, prediction processing, and arithmetic coding processing. In this case, reference means using the reference node in the calculation of the predicted value of the attribute information, or using the state of the reference node (for example, occupancy information indicating whether the reference node contains a point group) in the determination of the encoding parameters. For example, the encoding parameters are quantization parameters in quantization processing, or contexts in arithmetic coding.

[0152] The additional information encoding unit 4633 generates encoded additional information (Compressed MetaData) as encoded data by encoding compressible data in the additional information.

[0153] The multiplexing unit 4634 generates an encoded stream (Compressed Stream) as encoded data by multiplexing the encoded position information, encoded attribute information, encoded additional information, and other additional information. The generated encoded stream is output to a processing unit in a system layer (not shown).

[0154] Next, the first decoding unit 4640, which is an example of the decoding unit 4624 that performs decoding of the first encoding method, will be described. Figure 7 It is a diagram showing the structure of the first decoding unit 4640. Figure 8It is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point cloud data by decoding encoded data (encoded stream) encoded by the first encoding method using the first encoding method. The first decoding unit 4640 includes a demultiplexing unit 4641, a position information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0155] An encoded stream (Compressed Stream) as encoded data is input from a processing unit at the system layer (not shown) to the first decoding unit 4640.

[0156] The demultiplexing unit 4641 separates encoded position information (Compressed Geometry), encoded attribute information (Compressed Attribute), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0157] The position information decoding unit 4642 generates position information by decoding the encoded position information. For example, the position information decoding unit 4642 restores the position information of the point cloud represented by three-dimensional coordinates based on the encoded position information represented by an N-ary tree structure such as an octree.

[0158] The attribute information decoding unit 4643 decodes the encoded attribute information based on the structure information generated by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 determines a reference point (reference node) to be referred to in the decoding of an object point (object node) to be processed based on the octree structure obtained by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 refers to a node in which the parent node in the octree of surrounding nodes or adjacent nodes is the same as the parent node of the object node. In addition, the method of determining the reference relationship is not limited to this.

[0159] In addition, the decoding process of the attribute information may include at least one of an inverse quantization process, a prediction process, and an arithmetic decoding process. In this case, reference means using a reference node in the calculation of the predicted value of the attribute information, or using the state of the reference node in the determination of the decoded parameter (for example, indicating whether the occupancy information of the point cloud is included in the reference node). For example, the decoded parameter is a quantization parameter in the inverse quantization process, or a context in the arithmetic decoding.

[0160] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. In addition, the first decoding unit 4640 uses additional information required for the decoding processes of the position information and the attribute information during decoding, and outputs additional information required for the external application program.

[0161] Next, the second encoding unit 4650, which is an example of the encoding unit 4613 that performs the second encoding method, will be described. Figure 9 It is a diagram showing the structure of the second encoding unit 4650. Figure 10 It is a block diagram of the second encoding unit 4650.

[0162] The second encoding unit 4650 generates encoded data (encoded stream) by encoding point cloud data using the second encoding method. This second encoding unit 4650 includes an additional information generation unit 4651, a position image generation unit 4652, an attribute image generation unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0163] The second encoding unit 4650 has the following characteristics: it generates a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encodes the generated position image and attribute image using an existing video encoding method. The second encoding method is also known as VPCC (video based PCC, video-based PCC).

[0164] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated based on sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).

[0165] The additional information generation unit 4651 generates mapping information of a plurality of two-dimensional images by projecting a three-dimensional structure onto a two-dimensional image.

[0166] The position image generation unit 4652 generates a position image (Geometry Image) based on the position information and the mapping information generated by the additional information generation unit 4651. This position image is, for example, a distance image representing distance (Depth) as a pixel value. In addition, this distance image can be an image of observing a plurality of point clouds from one viewpoint (an image in which a plurality of point clouds are projected on one two-dimensional plane), can be a plurality of images of observing a plurality of point clouds from a plurality of viewpoints, or can be one image formed by combining these plurality of images.

[0167] The attribute image generation unit 4653 generates an attribute image based on the attribute information and the mapping information generated by the additional information generation unit 4651. This attribute image is, for example, an image representing attribute information (such as color (RGB)) as a pixel value. In addition, this image can be an image of observing a plurality of point clouds from one viewpoint (an image in which a plurality of point clouds are projected on one two-dimensional plane), can be a plurality of images of observing a plurality of point clouds from a plurality of viewpoints, or can be one image formed by combining these plurality of images.

[0168] The image encoding unit 4654 encodes the position image and the attribute image using an image encoding method, thereby generating a compressed geometry image and a compressed attribute image as encoded data. In addition, as the image encoding method, any known encoding method can be used. For example, the image encoding method is AVC or HEVC, etc.

[0169] The additional information encoding unit 4655 encodes the additional information and the mapping information included in the point cloud data to generate compressed metadata.

[0170] The multiplexing unit 4656 multiplexes the compressed geometry image, the compressed attribute image, the compressed metadata, and other additional information to generate a compressed stream as encoded data. The generated compressed stream is output to a processing unit of a system layer (not shown).

[0171] Next, a second decoding unit 4660, which is an example of the decoding unit 4624 that performs decoding using the second encoding method, will be described. Figure 11 This is a diagram showing the structure of the second decoding unit 4660. Figure 12 This is a block diagram of the second decoding unit 4660. The second decoding unit 4660 decodes the encoded data (compressed stream) encoded using the second encoding method using the second encoding method to generate point cloud data. The second decoding unit 4660 includes a demultiplexing unit 4661, an image decoding unit 4662, an additional information decoding unit 4663, a position information generation unit 4664, and an attribute information generation unit 4665.

[0172] The compressed stream as encoded data is input from a processing unit of a system layer (not shown) to the second decoding unit 4660.

[0173] The demultiplexing unit 4661 separates the compressed geometry image, the compressed attribute image, the compressed metadata, and other additional information from the encoded data.

[0174] The image decoding unit 4662 decodes the compressed geometry image and the compressed attribute image using an image encoding method to generate a position image and an attribute image. In addition, as the image encoding method, any known encoding method can be used. For example, the image encoding method is AVC or HEVC, etc.

[0175] The additional information decoding unit 4663 generates additional information including mapping information, etc. by decoding the encoded additional information.

[0176] The position information generation unit 4664 generates position information using the position image and the mapping information. The attribute information generation unit 4665 generates attribute information using the attribute image and the mapping information.

[0177] The second decoding unit 4660 uses the additional information required for decoding during decoding and outputs the additional information required for the external application.

[0178] Hereinafter, the problems in the PCC encoding method will be described. Figure 13 It is a diagram showing the protocol stack related to the PCC encoded data. Figure 13 It shows an example of multiplexing, transmitting, or storing data of other media such as images (e.g., HEVC) or sounds in the PCC encoded data.

[0179] The multiplexing method and the file format have functions for multiplexing, transmitting, or storing various encoded data. In order to transmit or store the encoded data, the encoded data must be transformed into the format of the multiplexing method. For example, in HEVC, a technique is defined in which the encoded data is stored in a data structure called an NAL unit, and the NAL unit is stored in ISOBMFF.

[0180] On the other hand, currently, as encoding methods for point cloud data, the first encoding method (Codec1) and the second encoding method (Codec2) are being studied, but the structure of the encoded data and the method of storing the encoded data into the system format are not defined, and there are problems such as being unable to directly perform the MUX process (multiplexing), transmission, and storage in the encoding unit.

[0181] In addition, hereinafter, if there is no description of a specific encoding method, it means either the first encoding method or the second encoding method.

[0182] Hereinafter, the definition method of the NAL unit in the present embodiment will be described. For example, in conventional codecs such as HEVC, for one codec, one format of the NAL unit is defined. However, a method for supporting a format in which two codecs, the first encoding method and the second encoding method (hereinafter referred to as the PCC codec), coexist like PCC does not exist yet.

[0183] In the present embodiment, a format common to the PCC codecs is defined as the NAL unit, and further, an identifier of the NAL unit dependent on the PCC codec is defined. Figure 14 It is a diagram showing the protocol stack in this case. Figures 15 to 17 It is a diagram showing an example of the NAL unit format common to the codecs.Figure 15 This is a diagram showing the syntax example of a Common PCC NAL Unit. Figure 16 This is a diagram showing the syntax example of a Common PCC NAL Unit Header. Figure 17 This is a diagram showing the semantic example of pcc_codec_type. Figure 18 This is a diagram showing an example of the definition of a codec-dependent NAL unit type and the semantic example of pcc_nal_unit_type.

[0184] As a NAL unit format, the NAL unit format common to the PCC codec is defined. A NAL unit (pcc_nal_unit) includes a header (pcc_nal_unit_header), a payload (pcc_nal_unit_payload), and trailing bits. The same format is used even when storing the data of a codec that stores either the first encoding method or the second encoding method.

[0185] The codec type (pcc_codec_type) and the NAL unit type (pcc_nal_unit_type) are stored in the NAL unit header (pcc_nal_unit_header). The codec type indicates whether the PCC codec of the encoded data stored in the NAL unit is the first encoding method or the second encoding method.

[0186] The NAL unit type indicates the type of the codec-dependent NAL unit and is defined for each codec. When the codec type is the first encoding method, the NAL unit type indicates the NAL unit type defined for the first encoding method. When the codec type is the second encoding method, the NAL unit type indicates the NAL unit type defined for the second encoding method. That is, for the NAL unit type defined for the first encoding method and the NAL unit type defined for the second encoding method, the same value corresponds to different meanings.

[0187] In addition, in the header, the function of the codec type can also be incorporated into the NAL unit type. For example, part of the information of the NAL unit type can also be used to indicate the codec type.

[0188] Next, the encoding process of this embodiment will be described. Figure 19It is a flowchart of the encoding process of this embodiment. The process in this figure represents the process of the first encoding unit 4630 or the second encoding unit 4650 when the above definitions are used. Additionally, hereinafter, the first encoding unit 4630 and the second encoding unit 4650 will not be distinguished and will be denoted as the encoding unit 4613. Moreover, the process in this figure is mainly performed by Figure 6 the multiplexing unit 4634 shown or Figure 10 the multiplexing unit 4656 shown.

[0189] Furthermore, the process in this figure represents an example of encoding PCC data using either the first encoding method or the second encoding method, and it is known in which PCC codec the encoding is performed. For example, which PCC codec to use can also be specified by the user or an external device, etc.

[0190] First, the encoding unit 4613 encodes the PCC data using either the first encoding method or the second encoding method codec (S4601).

[0191] When the codec used is the second encoding method (the second encoding method in S4602), the encoding unit 4613 sets the pcc_codec_type included in the NAL unit header to a value indicating that the data included in the payload of the NAL unit is data encoded by the second encoding method (S4603). In addition, the encoding unit 4613 sets the identifier of the NAL unit for the second encoding method in the pcc_nal_unit_type of the NAL unit header (S4604). Then, the encoding unit 4613 generates a NAL unit that has the set NAL unit header and includes the encoded data in the payload. Then, the encoding unit 4613 transmits the generated NAL unit (S4605).

[0192] On the other hand, when the codec used is the first encoding method (the first encoding method in S4602), the encoding unit 4613 sets the pcc_codec_type of the NAL unit header to a value indicating that the data included in the payload of the NAL unit is data encoded by the first encoding method (S4606). In addition, the encoding unit 4613 sets the identifier of the NAL unit for the first encoding method in the pcc_nal_unit_type of the NAL unit header (S4607). Then, the encoding unit 4613 generates a NAL unit that has the set NAL unit header and includes the encoded data in the payload. Then, the encoding unit 4613 transmits the generated NAL unit (S4605).

[0193] In addition, in steps S4603 and S4606, when the function of pcc_code_type is included in pcc_nal_unit_type, the encoding unit 4613 may also indicate in pcc_nal_unit_type whether the NAL unit is encoded by the first encoding method or the second encoding method.

[0194] Next, the decoding processes of the first decoding unit 4640 and the second decoding unit 4660 of the present embodiment will be described. Figure 20 It is a flowchart showing the decoding process of the second decoding unit 4660. In addition, the process of this figure is mainly performed by Figure 12 the inverse multiplexing unit 4661 shown.

[0195] In addition, the process of this figure shows an example in which PCC data is encoded by either the second encoding method or the first encoding method. In addition, in this method, the inverse multiplexing unit 4661 included in the second decoding unit 4660 can identify the codec type of the NAL unit by referring to the information included in the NAL unit header. Therefore, the inverse multiplexing unit 4661 can output the required information to the video decoding unit 4662 according to the codec type.

[0196] First, the second decoding unit 4660 receives the NAL unit (S4611). For example, this NAL unit is generated in the process in the above-mentioned encoding unit 4613. That is, the header of this NAL unit includes pcc_codec_type and pcc_nal_unit_type.

[0197] Next, the second decoding unit 4660 determines whether pcc_codec_type included in the NAL unit header represents the first encoding method or the second encoding method (S4612).

[0198] When pcc_codec_type represents the second encoding method (the second encoding method in S4612), the second decoding unit 4660 determines that the data included in the payload of the NAL unit is data encoded by the second encoding method (S4613). Then, the second decoding unit 4660 identifies the data by using pcc_nal_unit_type included in the NAL unit header as the identifier of the NAL unit for the second encoding method (S4614). Then, the second decoding unit 4660 decodes the PCC data using the decoding process of the second encoding method (S4615).

[0199] On the other hand, when pcc_codec_type indicates the first encoding method (the first encoding method in S4612), the second decoding unit 4660 determines that the data included in the payload of the NAL unit is the data encoded by the first encoding method (S4616). In this case, the second decoding unit 4660 does not process this NAL unit (S4617).

[0200] In addition, in step S4612, when the function of pcc_code_type is included in pcc_nal_unit_type, the second decoding unit 4660 can also refer to pcc_nal_unit_type to determine whether the codec used in the data included in the NAL unit is the first encoding method or the second encoding method.

[0201] Figure 21 It is a flowchart showing the decoding process of the first decoding unit 4640. In addition, the process of this figure is mainly performed by Figure 8 the demultiplexing unit 4641 shown.

[0202] In addition, the process of this figure shows an example in which PCC data is encoded by either the first encoding method or the second encoding method. In addition, in this method, the demultiplexing unit 4641 included in the first decoding unit 4640 can identify the codec type of the NAL unit by referring to the information included in the NAL unit header. Therefore, the demultiplexing unit 4641 can output the required information to the position information decoding unit 4642 and the attribute information decoding unit 4643 according to the codec type.

[0203] First, the first decoding unit 4640 receives the NAL unit (S4621). For example, this NAL unit is generated in the process in the above-mentioned encoding unit 4613. That is, the header of this NAL unit includes pcc_codec_type and pcc_nal_unit_type.

[0204] Next, the first decoding unit 4640 determines whether pcc_codec_type included in the NAL unit header indicates the first encoding method or the second encoding method (S4622).

[0205] When pcc_codec_type indicates the second encoding method (the second encoding method in S4622), the first decoding unit 4640 determines that the data included in the payload of the NAL unit is the data encoded using the second encoding method (S4623). In this case, the first decoding unit 4640 does not process this NAL unit (S4624).

[0206] On the other hand, when pcc_codec_type indicates the first encoding method (the first encoding method in S4622), the first decoding unit 4640 determines that the data included in the payload of the NAL unit is the data encoded by the first encoding method (S4625). Then, the first decoding unit 4640 identifies the data by using pcc_nal_unit_type included in the NAL unit header as the identifier of the NAL unit for the first encoding method (S4626). Then, the first decoding unit 4640 decodes the PCC data by using the decoding process of the first encoding method (S4627).

[0207] (Embodiment 2)

[0208] In this embodiment, another method for defining the NAL unit is described. In this embodiment, as the NAL unit, different formats are defined for each PCC codec. Furthermore, the identifier of the NAL unit is defined independently for each PCC codec.

[0209] Figure 22 It is a diagram showing the protocol stack in this case. Figure 23 It is a diagram showing a syntax example of the NAL unit (codec2_nal_unit) for codec 2. Figure 24 It is a diagram showing a syntax example of the NAL unit header (codec2_nal_unit_header) for codec 2. Figure 25 It is a diagram showing a semantic example of codec2_nal_unit_type.

[0210] Figure 26 It is a diagram showing a syntax example of the NAL unit (codec1_nal_unit) for codec 1. Figure 27 It is a diagram showing a syntax example of the NAL unit header (codec1_nal_unit_header) for codec 1. Figure 28 It is a diagram showing a semantic example of codec1_nal_unit_type.

[0211] As an NAL unit format, the NAL unit format is defined independently for each PCC codec. The NAL units (codec1_nal_unit, codec2_nal_unit) include headers (codec1_nal_unit_header, codec2_nal_unit_header), payloads (codec1_nal_unit_payload, codec2_nal_unit_payload), and trailing bits. The NAL unit (codec1_nal_unit) used for the first encoding method and the NAL unit (codec2_nal_unit) used for the second encoding method may have the same structure or different structures. The sizes of the NAL unit used for the first encoding method and the NAL unit used for the second encoding method may also be different.

[0212] The data encoded by the first encoding method is stored in the NAL unit used for the first encoding method. The data encoded by the second encoding method is stored in the NAL unit used for the second encoding method.

[0213] In the NAL unit headers (codec1_nal_unit_header, codec2_nal_unit_header), the NAL unit types (codec1_nal_unit_type, codec2_nal_unit_type) are stored. The NAL unit types are independent for each codec and are defined for each codec. That is, in the NAL unit used for the first encoding method, the NAL unit type defined for the first encoding method is recorded. In the NAL unit used for the second encoding method, the NAL unit type defined for the second encoding method is recorded.

[0214] By adopting this method, the first encoding method and the second encoding method can be treated as different codecs.

[0215] Next, the encoding process of this embodiment will be described. Figure 29 is a flowchart of the encoding process of this embodiment. The process in this figure represents the process of the first encoding unit 4630 or the second encoding unit 4650 when the above definitions are used. In addition, the process in this figure is mainly performed by Figure 6 the multiplexing unit 4634 shown or Figure 10 the multiplexing unit 4656 shown.

[0216] In addition, the processing of this figure represents an example in which PCC data is encoded using either the first encoding method or the second encoding method, and it is known in which PCC codec the encoding is performed. For example, which PCC codec to use can also be specified by the user or an external device, etc.

[0217] First, the encoding unit 4613 encodes the PCC data using either one of the codecs of the first encoding method and the second encoding method (S4631).

[0218] When the codec used is the second encoding method (the second encoding method in S4632), the encoding unit 4613 generates NAL units in the NAL unit format used for the second encoding method (S4633). Next, the encoding unit 4613 sets the identifier of the NAL unit used for the second encoding method in the codec2_nal_unit_type in the NAL unit header (S4634). Then, the encoding unit 4613 generates an NAL unit that has the set NAL unit header and includes encoded data in the payload. Then, the encoding unit 4613 transmits the generated NAL unit (S4635).

[0219] On the other hand, when the codec used is the first encoding method (the first encoding method in S4632), the encoding unit 4613 generates NAL units in the NAL unit format used for the first encoding method (S4636). Next, the encoding unit 4613 sets the identifier of the NAL unit used for the first encoding method in the codec1_nal_unit_type in the NAL unit header (S4637). Then, the encoding unit 4613 generates an NAL unit that has the set NAL unit header and includes encoded data in the payload. Then, the encoding unit 4613 transmits the generated NAL unit (S4635).

[0220] Next, the decoding process of this embodiment will be described. Figure 30 is a flowchart of the decoding process of this embodiment. The processing of this figure represents the processing using the first decoding unit 4640 or the second decoding unit 4660 in the case of the above definition. In addition, hereinafter, the first decoding unit 4640 or the second decoding unit 4660 will not be distinguished and will be denoted as the decoding unit 4624. In addition, the processing of this figure is mainly performed by Figure 8 the demultiplexing unit 4641 shown or Figure 12 the demultiplexing unit 4661 shown.

[0221] In addition, the processing of this figure represents an example in which PCC data is encoded using either the first encoding method or the second encoding method, and it is assumed that it is known in which PCC codec the encoding was performed. For example, information indicating the codec used is included in the transmission signal, multiplexed data, or encoded data, and the decoding unit 4624 determines the codec used by referring to this information. In addition, the decoding unit 4624 may also determine the codec used based on a signal obtained separately from these signals.

[0222] When the codec used is the second encoding method (second encoding method in S4641), the decoding unit 4624 receives NAL units in the format for the second encoding method (S4642). Then, assuming that the NAL units are for the second encoding method, the decoding unit 4624 identifies the data using the NAL unit format for the second encoding method and the codec2_nal_unit_type for the second encoding method (S4643). Then, the decoding unit 4624 decodes the PCC data using the decoding process of the second encoding method (S4644).

[0223] On the other hand, when the codec used is the first encoding method (first encoding method in S4641), the decoding unit 4624 receives NAL units in the format for the first encoding method (S4645). Then, assuming that the NAL units are for the first encoding method, the decoding unit 4624 identifies the data using the NAL unit format for the first encoding method and the codec1_nal_unit_type for the first encoding method (S4646). Then, the decoding unit 4624 decodes the PCC data using the decoding process of the first encoding method (S4747).

[0224] (Embodiment 3)

[0225] In this embodiment, another method for defining NAL units will be described. In this embodiment, a format common to PCC codecs is defined as the NAL unit. In addition, an identifier for the NAL unit common to PCC codecs is defined.

[0226] Figure 31 It is a diagram showing the protocol stack in this case. Figures 32 to 34 It is a diagram showing an example of the NAL unit format common to codecs. Figure 32 It is a diagram showing a syntax example of a Common PCC NAL Unit. Figure 33 It is a diagram showing a syntax example of a Common PCC NAL Unit Header. Figure 34 It is a diagram showing a semantic example of pcc_codec_type.

[0227] As the NAL unit format, the common NAL unit format of the PCC codec is defined. The NAL unit (pcc_nal_unit) includes a header (pcc_nal_unit_header), a payload (pcc_nal_unit_payload), and trailing bits. The same format is also used when storing the data of any one of the first encoding method and the second encoding method.

[0228] In the NAL unit header (pcc_nal_unit_header), the NAL unit type (pcc_nal_unit_type) is stored. The NAL unit type is common to the codecs and defines the common type of the codecs. That is, the NAL units used in the first encoding method and the NAL units used in the second encoding method together record the commonly defined NAL unit type. In Figure 34 the example shown, for example, PCC DataA is the encoded data of codec 1, PCC DataB is the encoded data of codec 2, PCCMetaDataA is the additional information of codec 1, and PCC MetaDataB is the additional information of codec 2.

[0229] By adopting this method, the first encoding method and the second encoding method can be processed as the same codec.

[0230] Next, the encoding process of this embodiment will be described. Figure 35 is a flowchart of the encoding process of this embodiment. The process in this figure represents the process of the first encoding unit 4630 or the second encoding unit 4650 when the above definitions are used. In addition, the process in this figure is mainly performed by Figure 6 the multiplexing unit 4634 shown or Figure 10 the multiplexing unit 4656 shown.

[0231] In addition, the process in this figure represents an example in which the PCC data is encoded by either the second encoding method or the first encoding method, and it is known in which PCC codec the encoding is performed. For example, which PCC codec to use can also be specified by a user or an external device, etc.

[0232] First, the encoding unit 4613 encodes the PCC data using any one of the second encoding method and the first encoding method (S4651). Next, the encoding unit 4613 generates a NAL unit in the PCC common NAL unit format (S4652).

[0233] Next, the encoding unit 4613 sets an identifier of the PCC common NAL unit for pcc_nal_unit_type included in the NAL unit header (S4653). Next, it transmits an NAL unit having the set NAL unit header and including encoded data in the payload (S4654).

[0234] Next, the decoding process of this embodiment will be described. Figure 36 FIG. is a flowchart of the decoding process of this embodiment. The process in this figure represents the process of the first decoding unit 4640 or the second decoding unit 4660 when the above definitions are used. In addition, the process in this figure is mainly performed by Figure 8 the demultiplexing unit 4641 shown in Figure 12 or the demultiplexing unit 4661 shown in

[0235] In addition, the process in this figure represents an example in which PCC data is encoded by either the second encoding method or the first encoding method.

[0236] First, the decoding unit 4624 determines the codec used in the encoding of the data included in the NAL unit (S4661). For example, the decoding unit 4624 determines the codec used by referring to pcc_nal_unit_type included in the NAL unit header.

[0237] When the codec used is the second encoding method (the second encoding method in S4661), the decoding unit 4624 receives an NAL unit in the PCC common format (S4662). Next, assuming that the NAL unit is common, the decoding unit 4624 uses the common NAL unit format and the common pcc_nal_unit_type to identify the data (S4663). Next, the decoding unit 4624 decodes the PCC data using the decoding process of the second encoding method (S4664).

[0238] On the other hand, when the codec used is the first encoding method (the first encoding method in S4661), the decoding unit 4624 receives an NAL unit in the PCC common format (S4665). Next, assuming that the NAL unit is common, the decoding unit 4624 uses the common NAL unit format and the common pcc_nal_unit_type to identify the data (S4666). Next, the decoding unit 4624 decodes the PCC data using the decoding process of the first encoding method (S4667).

[0239] Hereinafter, modification examples of the above-described Embodiments 1 to 3 will be described. As another method for indicating the PCC codec type, the following method can also be used.

[0240] In Embodiment 1, Embodiment 2, and Embodiment 3, the case where two codecs, i.e., the first encoding method and the second encoding method, coexist has been described. However, the above method can also be applied when there are three or more PCC codecs.

[0241] In addition, in Embodiment 1 and Embodiment 3, the identification information of the PCC codec (pcc_codec_type in Embodiment 1 and pcc_nal_unit_type in Embodiment 3b) is recorded in the NAL unit header. However, the identification information of the codec can also be stored in other places.

[0242] Furthermore, the first encoding method and the second encoding method are not limited to the above examples and can be any codecs. For example, the first encoding method and the second encoding method can be multiple codecs obtained by subdividing GPCC, or multiple codecs obtained by subdividing VPCC. For example, it can also be the case where both the first encoding method and the second encoding method are VPCC, but different video encoding methods are used. The video encoding method can be, for example, AVC or HEVC. In addition, either one or both of the first encoding method and the second encoding method can be an encoding method that includes other encoding methods such as video, audio, and text applications.

[0243] For example, the identification information of the codec can also be included in the control information contained in the PCC encoded stream. Here, the control information is, for example, metadata such as a parameter set or SEI (Supplemental Enhancement Information).

[0244] Figure 37 It is a flowchart of the encoding process of the encoding unit 4613 in this case. First, the encoding unit 4613 encodes the PCC data (S4671), and records the identification information of the PCC codec at a specified position (e.g., parameter set) in the encoded data (S4672). Then, the encoding unit 4613 generates a NAL unit containing the encoded data and transmits the generated NAL unit (S4673).

[0245] In addition, the identification information of the PCC codec can be defined as a profile, and the identification information of the PCC codec can be represented in the metadata. In addition, in the case where the same codec is used for the entire sequence, the identification information of the PCC codec can also be included in the sequence parameter set. Furthermore, in the case where each PCC frame is encoded with a different codec, the identification information of the PCC codec can also be included in the parameter set that records the information of each frame. For example, in the case where different codecs are used for each piece of PCC data as in the case where the codec is different in the position information and the attribute information, the identification information of the PCC codec can also be included in the parameter set that records the information of each piece of data. That is, the information of the codec representing the position information is included in the control information (parameter set, etc.) of the position information, and the information of the codec representing the attribute information can also be included in the control information (parameter set, etc.) of the attribute information.

[0246] In addition, the identification information of the codec can be stored in any one of the above, or can be stored in multiple locations. For example, the identification information of the codec can also be stored in both the encoded stream and the NAL unit header. In addition, in the case where the identification information of the codec is stored in multiple locations, the same information or different information can be stored in the multiple locations. Different information is, for example, information indicating GPCC or VPCC, and information indicating any one of multiple codecs obtained by subdividing GPCC or VPCC.

[0247] When the demultiplexing unit 4641 or 4661 included in the decoding unit 4624 includes a parameter set in the NAL unit, by analyzing the description in the parameter set, it is possible to determine whether the data included in the payload of the NAL unit is data encoded by the first encoding method or data encoded by the second encoding method. Thus, the decoding unit 4624 can quickly filter out NAL units that are not required for decoding.

[0248] Figure 38 It is a flowchart of the decoding process of the decoding unit 4624 in this case. First, the decoding unit 4624 receives an NAL unit (S4675), and uses the pcc_nal_unit_type included in the NAL unit header to identify specified data (such as the above parameter set) that records the identification information of the PCC codec (S4676). Next, the decoding unit 4624 identifies the PCC codec indicated in the specified data (such as the parameter set) by analyzing the specified data (S4677). Next, the decoding unit 4624 decodes the encoded data using the identified PCC codec (S4678).

[0249] In addition, in the above, an example of saving an encoded stream in an NAL unit has been shown, but a unit (cell) in a predetermined manner may be used instead of the NAL unit.

[0250] (Embodiment 4)

[0251] In this embodiment, an encoding unit 4670 having the functions of both the above-described first encoding unit 4630 and second encoding unit 4650, and a decoding unit 4680 having the functions of both the first decoding unit 4640 and second decoding unit 4660 will be described.

[0252] Figure 39 is a block diagram of the encoding unit 4670 of this embodiment. The encoding unit 4670 includes the above-described first encoding unit 4630, second encoding unit 4650, and a multiplexing unit 4671. The multiplexing unit 4671 multiplexes the encoded data generated by the first encoding unit 4630 and the encoded data generated by the second encoding unit 4650, and outputs the resulting encoded data.

[0253] Figure 40 is a block diagram of the decoding unit 4680 of this embodiment. The decoding unit 4680 includes the above-described first decoding unit 4640, second decoding unit 4660, and a demultiplexing unit 4681. The demultiplexing unit 4681 extracts the encoded data using the first encoding method and the encoded data using the second encoding method from the input encoded data. The demultiplexing unit 4681 outputs the encoded data using the first encoding method to the first decoding unit 4640, and outputs the encoded data using the second encoding method to the second decoding unit 4660.

[0254] According to the above structure, the encoding unit 4670 can selectively use the first encoding method and the second encoding method to encode point cloud data. In addition, the decoding unit 4680 can decode the encoded data encoded using the first encoding method, the encoded data encoded using the second encoding method, and the encoded data encoded using both the first encoding method and the second encoding method.

[0255] For example, the encoding unit 4670 can switch the encoding method (the first encoding method and the second encoding method) in units of point cloud data or frames. In addition, the encoding unit 4670 can also switch the encoding method in units that can be encoded.

[0256] The encoding unit 4670 generates, for example, encoded data (encoded stream) including the identification information of the PCC codec described in the above Embodiment 1 or Embodiment 3.

[0257] The demultiplexing unit 4681 included in the decoding unit 4680 identifies data, for example, using the identification information of the PCC codec described in Embodiment 1 or Embodiment 3. When the data is the data encoded by the first encoding method, the demultiplexing unit 4681 outputs the data to the first decoding unit 4640, and when the data is the data encoded by the second encoding method, the demultiplexing unit 4681 outputs the data to the second decoding unit 4660.

[0258] In addition, the encoding unit 4670 may send, as control information, information indicating whether both encoding methods or any one of the encoding methods is used, in addition to the identification information of the PCC codec.

[0259] Next, the encoding process of this embodiment will be described. Figure 41 It is a flowchart of the encoding process of this embodiment. By using the identification information of the PCC codec described in Embodiment 1, Embodiment 2, Embodiment 3, and the modification, the encoding process corresponding to multiple codecs can be performed. In addition, Figure 41 An example showing the case of using the method of Embodiment 1 is shown, but the same process can be applied to other methods.

[0260] First, the encoding unit 4670 encodes the PCC data by any one or both of the first encoding method and the second encoding method (S4681).

[0261] When the codec used is the second encoding method (the second encoding method in S4682), the encoding unit 4670 sets the pcc_codec_type included in the NAL unit header to a value indicating that the data included in the payload of the NAL unit is the data encoded by the second encoding method (S4683). Next, the encoding unit 4670 sets the identifier of the NAL unit for the second encoding method in the pcc_nal_unit_type of the NAL unit header (S4684). Then, the encoding unit 4670 generates a NAL unit having the set NAL unit header and including the encoded data in the payload. Then, the encoding unit 4670 transmits the generated NAL unit (S4685).

[0262] On the other hand, when the codec used is the first encoding method (the first encoding method in S4682), the encoding unit 4670 sets the pcc_codec_type included in the NAL unit header to a value indicating that the data included in the payload of the NAL unit is the data encoded by the first encoding method (S4686). Next, the encoding unit 4670 sets the identifier of the NAL unit for the first encoding method to the pcc_nal_unit_type included in the NAL unit header (S4687). Next, the encoding unit 4670 generates a NAL unit that has the set NAL unit header and includes encoded data in the payload. Then, the encoding unit 4670 transmits the generated NAL unit (S4685).

[0263] Next, the decoding process of this embodiment will be described. Figure 42 This is a flowchart of the decoding process of this embodiment. By using the identification information of the PCC codec described in Embodiment 1, Embodiment 2, Embodiment 3, and the modification example, the decoding process corresponding to multiple codecs can be performed. In addition, Figure 42 An example showing the case of using the method of Embodiment 1 is shown, but the same process can also be applied to other methods.

[0264] First, the decoding unit 4680 receives a NAL unit (S4691). For example, this NAL unit is generated in the process in the above-described encoding unit 4670.

[0265] Next, the decoding unit 4680 determines whether the pcc_codec_type included in the NAL unit header indicates the first encoding method or the second encoding method (S4692).

[0266] When the pcc_codec_type indicates the second encoding method (the second encoding method in S4692), the decoding unit 4680 determines that the data included in the payload of the NAL unit is the data encoded by the second encoding method (S4693). Then, the second decoding unit 4660 identifies the data by using the pcc_nal_unit_type included in the NAL unit header as the identifier of the NAL unit for the second encoding method (S4694). Then, the decoding unit 4680 decodes the PCC data by using the decoding process of the second encoding method (S4695).

[0267] On the other hand, when pcc_codec_type indicates the first encoding method (the first encoding method in S4692), the decoding unit 4680 determines that the data included in the payload of the NAL unit is the data encoded by the first encoding method (S4696). Then, the decoding unit 4680 identifies the data using pcc_nal_unit_type included in the NAL unit header as an identifier of the NAL unit for the first encoding method (S4697). Then, the decoding unit 4680 decodes the PCC data using the decoding process of the first encoding method (S4698).

[0268] As described above, a three-dimensional data encoding device according to an aspect of the present disclosure generates an encoded stream by encoding three-dimensional data (e.g., point cloud data) (e.g., Figure 37 of S4671), and stores information indicating the encoding method used for the encoding in the first encoding method and the second encoding method (e.g., identification information of the codec) (e.g., Figure 37 of S4672) in the control information (e.g., parameter set) of the encoded stream.

[0269] Accordingly, when decoding the encoded stream generated by the three-dimensional data encoding device, the three-dimensional data decoding device can determine the encoding method used in the encoding using the information stored in the control information. Therefore, even when using multiple encoding methods, the three-dimensional data decoding device can correctly decode the encoded stream.

[0270] For example, the three-dimensional data includes position information. In the encoding, the three-dimensional data encoding device encodes the position information. In the storing, the three-dimensional data encoding device stores information indicating the encoding method used for the encoding of the position information in the first encoding method and the second encoding method in the control information of the position information.

[0271] For example, the three-dimensional data includes position information and attribute information. In the encoding, the three-dimensional data encoding device encodes the position information and the attribute information. In the storing, the three-dimensional data encoding device stores information indicating the encoding method used for the encoding of the position information in the first encoding method and the second encoding method in the control information of the position information, and stores information indicating the encoding method used for the encoding of the attribute information in the first encoding method and the second encoding method in the control information of the attribute information.

[0272] Accordingly, different encoding methods can be used for the position information and the attribute information, and thus the encoding efficiency can be improved.

[0273] For example, the three-dimensional data encoding method further stores the encoded stream in one or more units (e.g., NAL units) (e.g., Figure 37 of S4673).

[0274] For example, as described in Embodiment 1 Figures 15 to 18 above, the unit contains information having a format common to the first encoding method and the second encoding method. The information is information indicating the category of data contained in the unit and has definitions independent of the first encoding method and the second encoding method (e.g., pcc_nal_unit_type).

[0275] For example, as described in Embodiment 2 Figures 23 to 28 above, the unit contains information having formats independent of the first encoding method and the second encoding method. The information is information indicating the category of data contained in the unit and has definitions independent of the first encoding method and the second encoding method (e.g., codec1_nal_unit_type or codec2_nal_unit_type).

[0276] For example, as described in Embodiment 3 Figures 32 to 34 above, the unit contains information having a format common to the first encoding method and the second encoding method. The information is information indicating the category of data contained in the unit and has definitions common to the first encoding method and the second encoding method (e.g., pcc_nal_unit_type).

[0277] For example, the three-dimensional data encoding apparatus includes a processor and a memory, and the processor performs the above processing using the memory.

[0278] In addition, the three-dimensional data decoding apparatus of the present embodiment determines the encoding method used in the encoding of the encoded stream (e.g., Figure 38 of S4677) based on information (e.g., codec identification information) indicating the encoding method for encoding the three-dimensional data in the first encoding method and the second encoding method, which is included in the control information (e.g., parameter set) of the encoded stream generated by encoding the three-dimensional data, and decodes the encoded stream using the determined encoding method (e.g., Figure 38 of S4678).

[0279] Thus, when decoding the encoded stream, the three-dimensional data decoding device can use the information stored in the control information to determine the encoding method used in the encoding. Therefore, even when multiple encoding methods are used, the three-dimensional data decoding device can correctly decode the encoded stream.

[0280] For example, the three-dimensional data includes position information, and the encoded stream includes encoded data of the position information. In the determination, the three-dimensional data decoding device determines the encoding method used in the encoding of the position information based on the information indicating the encoding method used in the encoding of the position information in the control information included in the encoded stream and included in the first encoding method and the second encoding method. In the decoding, the three-dimensional data decoding device decodes the encoded data of the position information using the encoding method determined to be used in the encoding of the position information.

[0281] For example, the three-dimensional data includes position information and attribute information, and the encoded stream includes encoded data of the position information and encoded data of the attribute information. In the determination, the three-dimensional data decoding device determines the encoding method used in the encoding of the position information based on the information indicating the encoding method used in the encoding of the position information in the control information included in the encoded stream and included in the first encoding method and the second encoding method, and determines the encoding method used in the encoding of the attribute information based on the information indicating the encoding method used in the encoding of the attribute information in the control information included in the encoded stream and included in the first encoding method and the second encoding method. In the decoding, the three-dimensional data decoding device decodes the encoded data of the position information using the encoding method determined to be used in the encoding of the position information, and decodes the encoded data of the attribute information using the encoding method determined to be used in the encoding of the attribute information.

[0282] Thus, different encoding methods can be used for the position information and the attribute information, so that the encoding efficiency can be improved.

[0283] For example, the encoded stream is stored in one or more units (e.g., NAL units), and the three-dimensional data decoding device further obtains the encoded stream from the one or more units.

[0284] For example, as in Embodiment 1 Figures 15 to 18As described in [description], the unit contains information which has a format common to the first encoding method and the second encoding method. The information is information representing the category of data contained in the unit and has definitions independent of each other in the first encoding method and the second encoding method (e.g., pcc_nal_unit_type).

[0285] For example, as in Embodiment 2 Figures 23 to 28 As described in [description], the unit contains information which has a format independent of each other in the first encoding method and the second encoding method. The information is information representing the category of data contained in the unit and has definitions independent of each other in the first encoding method and the second encoding method (e.g., codec1_nal_unit_type or codec2_nal_unit_type).

[0286] For example, as in Embodiment 3 Figures 32 to 34 As described in [description], the unit contains information which has a format common to the first encoding method and the second encoding method. The information is information representing the category of data contained in the unit and has definitions common to each other in the first encoding method and the second encoding method (e.g., pcc_nal_unit_type).

[0287] For example, a three-dimensional data decoding device includes a processor and a memory, and the processor uses the memory to perform the above processing.

[0288] (Embodiment 5)

[0289] In the present embodiment, a method for saving NAL units to an ISOBMFF file described in Embodiment 1 will be described.

[0290] ISOBMFF (ISO based media file format) is a file format standard specified by ISO / IEC 14496-12. ISOBMFF specifies a format capable of multiplexing and saving various media such as video, audio, and text, and is a standard independent of media.

[0291] The basic structure (file) of ISOBMFF will be described. The basic unit in ISOBMFF is a box. A box consists of type, length, and data, and a collection of boxes of various types forms a file.

[0292] Figure 43This is a diagram showing the basic structure (file) of ISOBMFF. The files of ISOBMFF mainly include ftyp that represents the version (brand) of the file in 4CC (4-character code), moov that stores metadata such as control information, and boxes such as mdat that stores data.

[0293] The storage method for each medium in the ISOBMFF file is specified separately. For example, the storage methods for AVC video and HEVC video are specified in ISO / IEC 14496-15. Here, in order to store or transmit PCC encoded data, it is possible to consider expanding the functions of ISOBMFF for use, but there is no regulation on storing PCC encoded data in the ISOBMFF file. Therefore, in this embodiment, the method for storing PCC encoded data in the ISOBMFF file will be described.

[0294] Figure 44 This is a diagram showing the protocol stack when storing the NAL unit common to the PCC codec in the ISOBMFF file. Here, the NAL unit common to the PCC codec is stored in the ISOBMFF file. The NAL unit is common to the PCC codec, but since multiple PCC codecs are stored in the NAL unit, it is desirable to specify the storage method corresponding to each codec (Carriage of Codec1, Carriage of Codec2).

[0295] Next, the method for storing the common PCC NAL unit that supports multiple PCC codecs in the ISOBMFF file will be described. Figure 45 This is a diagram showing an example of storing the ISOBMFF file of the method (Carriage of Codec1) for storing the common PCC NAL unit in Codec 1. Figure 46 This is a diagram showing an example of storing the ISOBMFF file of the method (Carriage of Codec2) for storing the common PCC NAL unit in Codec 2.

[0296] Here, ftyp is important information used to identify the file format. As ftyp, an identifier different for each codec is defined. When storing PCC encoded data encoded by the first encoding method (encoding mode) in the file, ftyp = pcc1 is set. When storing PCC encoded data encoded by the second encoding method in the file, ftyp = pcc2 is set.

[0297] Here, pcc1 represents codec 1 that uses PCC (the first encoding method). pcc2 represents codec 2 that uses PCC (the second encoding method). That is, pcc1 and pcc2 indicate that the data is PCC (code data of three-dimensional data (point cloud data)), and also represent PCC codecs (the first encoding method and the second encoding method).

[0298] Hereinafter, a method for saving NAL units to an ISOBMFF file will be described. The multiplexing unit analyzes the NAL unit header, and when pcc_codec_type = Codec1, pcc1 is recorded in the ftyp of ISOBMFF.

[0299] In addition, the multiplexing unit analyzes the NAL unit header, and when pcc_codec_type = Codec2, pcc2 is recorded in the ftyp of ISOBMFF.

[0300] In addition, when the pcc_nal_unit_type is metadata, the multiplexing unit saves the NAL unit to moov or mdat, for example, by a prescribed method. When the pcc_nal_unit_type is data, the multiplexing unit saves the NAL unit to moov or mdat, for example, by a prescribed method.

[0301] For example, the multiplexing unit may save the NAL unit size to the NAL unit in the same way as HEVC.

[0302] By analyzing the ftyp included in the file in the demultiplexing unit (system layer) using this saving method, it is possible to determine whether the PCC encoded data is encoded by the first encoding method or the second encoding method. Furthermore, as described above, by determining whether the PCC encoded data is encoded by the first encoding method or the second encoding method, it is possible to extract the encoded data encoded by one of the encoding methods from the data in which the encoded data encoded by both encoding methods is mixed. Thus, when transmitting the encoded data, the amount of data to be transmitted can be suppressed. In addition, by this saving method, it is possible to use a common data format without setting different data (file) formats for the first encoding method and the second encoding method.

[0303] In addition, when the identification information of the codec is represented in the system layer metadata such as ftyp in ISOBMFF, the multiplexing unit may save the NAL unit after deleting the pcc_nal_unit_type to the ISOBMFF file.

[0304] Next, the configurations and operations of the multiplexing unit included in the three-dimensional data encoding system (three-dimensional data encoding device) according to the present embodiment and the demultiplexing unit included in the three-dimensional data decoding system (three-dimensional data decoding device) according to the present embodiment will be described.

[0305] Figure 47 FIG. 4710 shows the configuration of the first multiplexing unit 4710. The first multiplexing unit 4710 includes a file conversion unit 4711 that generates multiplexed data (file) by storing the encoded data and control information (NAL unit) generated by the first encoding unit 4630 into an ISOBMFF file. The first multiplexing unit 4710 is included in, for example, Figure 1 the multiplexing unit 4614 shown in FIG.

[0306] Figure 48 FIG. 4720 shows the configuration of the first demultiplexing unit 4720. The first demultiplexing unit 4720 includes a file inverse conversion unit 4721 that obtains the encoded data and control information (NAL unit) from the multiplexed data (file) and outputs the obtained encoded data and control information to the first decoding unit 4640. The first demultiplexing unit 4720 is included in, for example, Figure 1 the demultiplexing unit 4623 shown in FIG.

[0307] Figure 49 FIG. 4730 shows the configuration of the second multiplexing unit 4730. The second multiplexing unit 4730 includes a file conversion unit 4731 that generates multiplexed data (file) by storing the encoded data and control information (NAL unit) generated by the second encoding unit 4650 into an ISOBMFF file. The second multiplexing unit 4730 is included in, for example, Figure 1 the multiplexing unit 4614 shown in FIG.

[0308] Figure 50 FIG. 4740 shows the configuration of the second demultiplexing unit 4740. The second demultiplexing unit 4740 includes a file inverse conversion unit 4741 that obtains the encoded data and control information (NAL unit) from the multiplexed data (file) and outputs the obtained encoded data and control information to the second decoding unit 4660. The second demultiplexing unit 4740 is included in, for example, Figure 1 the demultiplexing unit 4623 shown in FIG.

[0309] Figure 51 FIG. 4770 is a flowchart of the multiplexing process performed by the first multiplexing unit 4710. First, the first multiplexing unit 4710 determines whether the codec used is the first encoding method or the second encoding method by analyzing the pcc_codec_type included in the NAL unit header (S4701).

[0310] When pcc_codec_type indicates the second encoding method (the second encoding method in S4702), the first multiplexing unit 4710 does not process this NAL unit (S4703).

[0311] On the other hand, when pcc_codec_type indicates the second encoding method (the first encoding method in S4702), the first multiplexing unit 4710 records pcc1 in ftyp (S4704). That is, the first multiplexing unit 4710 records the information indicating that data encoded by the first encoding method is saved in the file in ftyp.

[0312] Next, the first multiplexing unit 4710 analyzes pcc_nal_unit_type included in the NAL unit header, and saves the data to a box (such as moov or mdat) by a prescribed method corresponding to the data type indicated by pcc_nal_unit_type (S4705). And the first multiplexing unit 4710 creates an ISOBMFF file including the above ftyp and the above box (S4706).

[0313] Figure 52 It is a flowchart of the multiplexing process performed by the second multiplexing unit 4730. First, the second multiplexing unit 4730 determines whether the codec used is the first encoding method or the second encoding method by analyzing pcc_codec_type included in the NAL unit header (S4711).

[0314] When pcc_unit_type indicates the second encoding method (the second encoding method in S4712), the second multiplexing unit 4730 records pcc2 in ftyp (S4713). That is, the second multiplexing unit 4730 records the information indicating that data encoded by the second encoding method is saved in the file in ftyp.

[0315] Next, the second multiplexing unit 4730 analyzes pcc_nal_unit_type included in the NAL unit header, and saves the data to a box (such as moov or mdat) by a prescribed method corresponding to the data type indicated by pcc_nal_unit_type (S4714). And the second multiplexing unit 4730 creates an ISOBMFF file including the above ftyp and the above box (S4715).

[0316] On the other hand, when pcc_unit_type indicates the first encoding method (the first encoding method in S4712), the second multiplexing unit 4730 does not process this NAL unit (S4716).

[0317] In addition, the above processing shows an example in which PCC data is encoded using either the first encoding method or the second encoding method. The first multiplexing unit 4710 and the second multiplexing unit 4730 direct NAL units desired for file storage by identifying the codec type of the NAL unit. In addition, when identification information of the PCC codec is included outside the NAL unit header, the first multiplexing unit 4710 and the second multiplexing unit 4730 may also use the identification information of the PCC codec included outside the NAL unit header to identify the codec type (the first encoding method or the second encoding method) in steps S4701 and S4711.

[0318] In addition, the first multiplexing unit 4710 and the second multiplexing unit 4730 may also, when saving data to a file in steps S4706 and S4714, save the data to the file after deleting pcc_nal_unit_type from the NAL unit header.

[0319] Figure 53 FIG. 47 is a flowchart showing the processing performed by the first demultiplexing unit 4720 and the first decoding unit 4640. First, the first demultiplexing unit 4720 analyzes ftyp included in the file of ISOBMFF (S4721). When the codec represented by ftyp is the second encoding method (pcc2) (Yes in S4722), the first demultiplexing unit 4720 determines that the data included in the payload of the NAL unit is data encoded using the second encoding method (S4723). In addition, the first demultiplexing unit 4720 passes the determination result to the first decoding unit 4640. The first decoding unit 4640 does not process this NAL unit (S4724).

[0320] On the other hand, when the codec represented by ftyp is the first encoding method (pcc1) (Yes in S4722), the first demultiplexing unit 4720 determines that the data included in the payload of the NAL unit is data encoded using the first encoding method (S4725). In addition, the first demultiplexing unit 4720 passes the determination result to the first decoding unit 4640.

[0321] The first decoding unit 4640 identifies the data by assuming that pcc_nal_unit_type included in the NAL unit header is an identifier of the NAL unit for the first encoding method (S4726). Then, the first decoding unit 4640 decodes the PCC data using the decoding process of the first encoding method (S4727).

[0322] Figure 54It is a flowchart showing the processing performed by the second demultiplexing unit 4740 and the second decoding unit 4660. First, the second demultiplexing unit 4740 analyzes the ftyp included in the ISOBMFF file (S4731). When the codec indicated by the ftyp is the second encoding method (pcc2) (it is the second encoding method in S4732), the second demultiplexing unit 4740 determines that the data included in the payload of the NAL unit is data encoded by the second encoding method (S4733). In addition, the second demultiplexing unit 4740 passes the determination result to the second decoding unit 4660.

[0323] The second decoding unit 4660 identifies the data by setting the pcc_nal_unit_type included in the NAL unit header as the identifier of the NAL unit for the second encoding method (S4734). And the second decoding unit 4660 decodes the PCC data using the decoding process of the second encoding method (S4735).

[0324] On the other hand, when the codec indicated by the ftyp is the first encoding method (pcc1) (it is the first encoding method in S4732), the second demultiplexing unit 4740 determines that the data included in the payload of the NAL unit is data encoded by the first encoding method (S4736). In addition, the second demultiplexing unit 4740 passes the determination result to the second decoding unit 4660. The second decoding unit 4660 does not process this NAL unit (S4737).

[0325] In this way, for example, in the first demultiplexing unit 4720 or the second demultiplexing unit 4740, by identifying the codec type of the NAL unit, the codec type can be identified at an earlier stage. Furthermore, the desired NAL unit can be input to the first decoding unit 4640 or the second decoding unit 4660, while removing the unnecessary NAL units. In this case, in the first decoding unit 4640 or the second decoding unit 4660, it may not be necessary to perform the process of analyzing the identification information of the codec. In addition, it is also possible to perform the process of analyzing the identification information of the codec again by referring to the NAL unit type in the first decoding unit 4640 or the second decoding unit 4660.

[0326] In addition, when the pcc_nal_unit_type is deleted from the NAL unit header in the first multiplexing unit 4710 or the second multiplexing unit 4730, the first demultiplexing unit 4720 or the second demultiplexing unit 4740 may also output to the first decoding unit 4640 or the second decoding unit 4660 after assigning the pcc_nal_unit_type to the NAL unit.

[0327] (Embodiment 6)

[0328] In the present embodiment, the multiplexing unit and the demultiplexing unit corresponding to the encoding unit 4670 and the decoding unit 4680 corresponding to multiple codecs described in Embodiment 4 are described. Figure 55 It is a diagram showing the configuration of the encoding unit 4670 and the third multiplexing unit 4750 according to the present embodiment.

[0329] The encoding unit 4670 encodes the point cloud data using either or both of the first encoding method and the second encoding method. The encoding unit 4670 may switch the encoding method (the first encoding method and the second encoding method) in units of point cloud data or frames. In addition, the encoding unit 4670 may switch the encoding method in units that can be encoded.

[0330] The encoding unit 4670 generates encoded data (encoded stream) including the identification information of the PCC codec described in Embodiments 1 to 4.

[0331] The third multiplexing unit 4750 includes a file conversion unit 4751. The file conversion unit 4751 converts the NAL unit output from the encoding unit 4670 into a file of PCC data. The file conversion unit 4751 analyzes the codec identification information included in the NAL unit header and determines whether the PCC encoded data is data encoded using the first encoding method, data encoded using the second encoding method, or data encoded using both methods. The file conversion unit 4751 records the version name capable of identifying the codec in the ftyp. For example, in the case of indicating encoding using both methods, pcc3 is recorded in the ftyp.

[0332] In addition, when the encoding unit 4670 records the identification information of the PCC codec outside the NAL unit, the file conversion unit 4751 may also use this identification information to determine the PCC codec (encoding method).

[0333] Figure 56 It is a diagram showing the configuration of the third demultiplexing unit 4760 and the decoding unit 4680 according to the present embodiment.

[0334] The third demultiplexing unit 4760 includes a file inverse conversion unit 4761. The file inverse conversion unit 4761 analyzes the ftyp included in the file and determines whether the PCC encoded data is data encoded using the first encoding method, data encoded using the second encoding method, or data encoded using both methods.

[0335] When the PCC encoded data is encoded using one of the encoding methods, the data is input to the corresponding decoding unit among the first decoding unit 4640 and the second decoding unit 4660, and the data is not input to the other decoding unit. When the PCC encoded data is encoded using both encoding methods, the data is input to the decoding unit 4680 corresponding to both methods.

[0336] The decoding unit 4680 decodes the PCC encoded data in a manner using one or both of the first encoding method and the second encoding method.

[0337] Figure 57 It is a flowchart showing the processing performed by the third multiplexing unit 4750 according to the present embodiment.

[0338] First, the third multiplexing unit 4750 determines whether the codec used is the first encoding method, the second encoding method, or both the first encoding method and the second encoding method by analyzing the pcc_codec_type included in the NAL unit header (S4741).

[0339] In the case where the second encoding method is used (Yes in S4742 and the second encoding method in S4743), the third multiplexing unit 4750 records pcc2 in the ftyp (S4744). That is, the third multiplexing unit 4750 records information indicating that data encoded using the second encoding method is stored in the file in the ftyp.

[0340] Next, the third multiplexing unit 4750 analyzes the pcc_nal_unit_type included in the NAL unit header and saves the data to a box (such as moov or mdat) using a prescribed method corresponding to the data type indicated by the pcc_unit_type (S4745). And the third multiplexing unit 4750 creates an ISOBMFF file including the above ftyp and the above box (S4746).

[0341] On the other hand, in the case where the first encoding method is used (Yes in S4742 and the first encoding method in S4743), the third multiplexing unit 4750 records pcc1 in the ftyp (S4747). That is, the third multiplexing unit 4750 records information indicating that data encoded using the first encoding method is stored in the file in the ftyp.

[0342] Next, the third multiplexing unit 4750 analyzes the pcc_nal_unit_type included in the NAL unit header and saves the data to a box (such as moov or mdat) using a prescribed method corresponding to the data type indicated by the pcc_unit_type (S4748). And the third multiplexing unit 4750 creates an ISOBMFF file including the above ftyp and the above box (S4746).

[0343] On the other hand, in the case where both the first encoding method and the second encoding method are used (No in S4742), the third multiplexing unit 4750 records pcc3 in ftyp (S4749). That is, the third multiplexing unit 4750 records information indicating that data encoded using both encoding methods is stored in the file in ftyp.

[0344] Next, the third multiplexing unit 4750 analyzes pcc_nal_unit_type included in the NAL unit header, and saves the data to a box (such as moov or mdat) using a specified method corresponding to the data type indicated by pcc_unit_type (S4750). Further, the third multiplexing unit 4750 creates an ISOBMFF file including the above ftyp and the above box (S4746).

[0345] Figure 58 It is a flowchart showing the processing performed by the third demultiplexing unit 4760 and the decoding unit 4680. First, the third demultiplexing unit 4760 analyzes ftyp included in the ISOBMFF file (S4761). In the case where the codec indicated by ftyp is the second encoding method (pcc2) (Yes in S4762 and it is the second encoding method in S4763), the third demultiplexing unit 4760 determines that the data included in the payload of the NAL unit is data encoded using the second encoding method (S4764). Further, the third demultiplexing unit 4760 transmits the determination result to the decoding unit 4680.

[0346] The decoding unit 4680 identifies the data by assuming that pcc_nal_unit_type included in the NAL unit header is an identifier of a NAL unit for the second encoding method. Then, the decoding unit 4680 decodes the PCC data using the decoding process of the second encoding method (S4766).

[0347] On the other hand, in the case where the codec indicated by ftyp is the first encoding method (pcc1) (Yes in S4762 and it is the first encoding method in S4763), the third demultiplexing unit 4760 determines that the data included in the payload of the NAL unit is data encoded using the first encoding method (S4767). Further, the third demultiplexing unit 4760 transmits the determination result to the decoding unit 4680.

[0348] The decoding unit 4680 identifies the data by assuming that pcc_nal_unit_type included in the NAL unit header is an identifier of a NAL unit for the first encoding method. Then, the decoding unit 4680 decodes the PCC data using the decoding process of the first encoding method (S4769).

[0349] On the other hand, in the case where the coding method (pcc3) used by both parties is indicated by ftyp (No in S4762), the third demultiplexing unit 4760 determines that the data included in the payload of the NAL unit is data encoded by both the first coding method and the second coding method (S4770). In addition, the third demultiplexing unit 4760 transmits the determination result to the decoding unit 4680.

[0350] The decoding unit 4680 identifies the data by setting the pcc_nal_unit_type included in the NAL unit header as the identifier of the NAL unit for the codec described in the pcc_codec_type (S4771). And the decoding unit 4680 decodes the PCC data using the decoding processes of both coding methods (S4772). That is, the decoding unit 4680 decodes the data encoded by the first coding method using the decoding process of the first coding method, and decodes the data encoded by the second coding method using the decoding process of the second coding method.

[0351] Hereinafter, a modification example of the present embodiment will be described. As the types of versions indicated by ftyp, the following types may also be represented by identification information. In addition, a combination of multiple types shown below may also be represented by identification information.

[0352] The identification information indicates whether the object of the original data before PCC coding is a point cloud with a restricted area or a large-scale point cloud without a restricted area such as map information.

[0353] The identification information may also indicate whether the original data before PCC coding is a static object or a dynamic object.

[0354] As described above, the identification information may also indicate whether the PCC encoded data is data encoded by the first coding method or data encoded by the second coding method.

[0355] The identification information may also indicate the algorithm used in PCC coding. Here, the algorithm is, for example, a coding method that can be used in the first coding method or the second coding method.

[0356] The identification information may also indicate the difference in the method of saving the PCC encoded data to an ISOBMFF file. For example, the identification information may indicate whether the saving method used is a saving method for accumulation or a saving method for real-time transmission such as dynamic streaming.

[0357] In addition, in Embodiment 5, the method of storing NAL units described in Embodiment 1 was explained, and in Embodiment 6, the method of storing NAL units described in Embodiment 4 was explained. However, the same storage method can also be applied to the method of storing NAL units described in Embodiments 2 and 3, and thereby the identification information of the PCC codec is stored in the ISOBMFF file.

[0358] In addition, in Embodiments 5 and 6, an example of using ISOBMFF as the file format was explained, but other methods can also be used. For example, the same method as in this embodiment can also be used when storing PCC encoded data in MPEG-2TS Systems, MPEG-DASH, MMT, or RMP.

[0359] In addition, in the above, an example of storing metadata such as identification information in ftyp was shown, but these metadata can also be stored outside of ftyp. For example, these metadata can be stored in moov.

[0360] As described above, the three-dimensional data storage device (or three-dimensional data multiplexing device, or three-dimensional data encoding device) performs Figure 59 the processing shown.

[0361] First, the three-dimensional data storage device (for example, including the first multiplexing unit 4710, the second multiplexing unit 4730, or the third multiplexing unit 4750) obtains one or more units (for example, NAL units) that store the encoded stream obtained by encoding point cloud data (S4781). Next, the three-dimensional data storage device stores the one or more units in a file (for example, an ISOBMFF file) (S4782). In addition, in the above storage (S4782), the three-dimensional data storage device stores information (for example, pcc1, pcc2, or pcc3) indicating that the data stored in the file is data obtained by encoding point cloud data in the control information (for example, ftyp) of the above file.

[0362] Thereby, in the device that processes the file generated by this three-dimensional data storage device, it is possible to refer to the control information of the file and determine earlier whether the data stored in the file is encoded data of point cloud data. Therefore, it is possible to reduce the processing amount of the device or speed up the processing.

[0363] For example, the above information also represents the encoding methods used in the encoding of the above point cloud data in the first encoding method and the second encoding method. In addition, the data saved in the file, which is the data obtained by encoding the point cloud data and the encoding methods used in the encoding of the point cloud data in the first encoding method and the second encoding method, can be represented by a single piece of information or by different pieces of information.

[0364] Thus, in a device that processes a file generated by this three-dimensional data storage device, it is possible to refer to the control information of the file and determine earlier the codec used for the data saved in the file. Therefore, it is possible to reduce the processing amount of the device or speed up the processing.

[0365] For example, the above first encoding method is a method (GPCC) of encoding position information representing the positions of point cloud data using an N-ary tree (where N is an integer of 2 or more) and encoding attribute information using the above position information, and the above second encoding method is a method (VPCC) of generating a two-dimensional image from point cloud data and encoding the above two-dimensional image using an image encoding method.

[0366] For example, the above file complies with ISOBMFF (ISO based media file format).

[0367] For example, the three-dimensional data storage device includes a processor and a memory, and the processor uses the memory to perform the above processing.

[0368] In addition, as described above, the three-dimensional data acquisition device (or three-dimensional data demultiplexing device, or three-dimensional data decoding device) performs Figure 60 the processing shown.

[0369] The three-dimensional data acquisition device (for example, including a first demultiplexing unit 4720, a second demultiplexing unit 4740, or a third demultiplexing unit 4760) acquires a file (for example, an ISOBMFF file) that stores one or more units (for example, NAL units), and the one or more units store an encoded stream obtained by encoding point cloud data (S4791). Then, the three-dimensional data acquisition device acquires one or more units from the file (S4792). In addition, the control information (for example, ftyp) of the file includes information (for example, pcc1, pcc2, or pcc3) indicating that the data stored in the file is data obtained by encoding point cloud data.

[0370] For example, the three-dimensional data acquisition device refers to the above information to determine whether the data stored in the file is data obtained by encoding point cloud data. In addition, when the three-dimensional data acquisition device determines that the data stored in the file is data obtained by encoding point cloud data, it generates point cloud data by decoding the data obtained by encoding point cloud data included in one or more units. Alternatively, when the three-dimensional data acquisition device determines that the data stored in the file is data obtained by encoding point cloud data, it outputs (notifies) information indicating that the data included in one or more units is data obtained by encoding point cloud data to a subsequent processing unit (for example, the first decoding unit 4640, the second decoding unit 4660, or the decoding unit 4680).

[0371] Thereby, the three-dimensional data acquisition device can refer to the control information of the file to determine earlier whether the data stored in the file is encoded data of point cloud data. Therefore, it is possible to reduce the processing amount or speed up the processing of the three-dimensional data acquisition device or the subsequent device.

[0372] For example, the above information also represents the encoding method used in the above encoding among the first encoding method and the second encoding method. In addition, the data stored in the file is data obtained by encoding point cloud data, and the encoding method used in the encoding of point cloud data among the first encoding method and the second encoding method can be represented by a single piece of information or by different pieces of information.

[0373] Thereby, the three-dimensional data acquisition device can refer to the control information of the file to determine earlier the codec used for the data stored in the file. Therefore, it is possible to reduce the processing amount or speed up the processing of the three-dimensional data acquisition device or the subsequent device.

[0374] For example, based on the above information, the three-dimensional data acquisition device obtains the data encoded by a certain encoding method from the encoded point cloud data including the data encoded by the first encoding method and the data encoded by the second encoding method.

[0375] For example, the above first encoding method is a method (GPCC) of encoding the position information representing the position of the point cloud data with an N-ary tree (N is an integer of 2 or more) and encoding the attribute information using the above position information, and the above second encoding method is a method (VPCC) of generating a two-dimensional image from the point cloud data and encoding the above two-dimensional image using an image encoding method.

[0376] For example, the above file complies with ISOBMFF (ISO based media file format).

[0377] For example, the three-dimensional data acquisition device includes a processor and a memory, and the processor uses the memory to perform the above-described processing.

[0378] As described above, the three-dimensional data storage device, the three-dimensional data acquisition device, etc. according to the embodiments of the present disclosure have been described, but the present disclosure is not limited to these embodiments.

[0379] In addition, each processing unit included in the three-dimensional data storage device, the three-dimensional data acquisition device, etc. of the above-described embodiments can typically be implemented as an LSI of an integrated circuit. These can be formed into one chip separately, or a part or all of them can be formed into one chip.

[0380] Moreover, the integration into an integrated circuit is not limited to LSI, and can also be implemented by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconstruct the connection or setting of the circuit part inside the LSI can also be used.

[0381] In addition, in each of the above-described embodiments, each component can be constituted by dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0382] Moreover, the present disclosure can be implemented as a three-dimensional data storage method, a three-dimensional data acquisition method, etc. to be executed by a three-dimensional data storage device, a three-dimensional data acquisition device, etc.

[0383] In addition, the division of the functional blocks in the block diagram is an example. Multiple functional blocks can be implemented as one functional block, one functional block can also be divided into multiple, and a part of the function can also be moved to other functional blocks. Moreover, the functions of multiple functional blocks having similar functions can also be processed in parallel or processed in a time-division manner by a single piece of hardware or software.

[0384] Regarding the execution order of each step in the flowchart, it is an example given for specifically explaining the present disclosure, and it can also be an order other than the above. In addition, a part of the above steps can also be executed simultaneously (in parallel) with other steps.

[0385] The above describes one or more forms of a three-dimensional data storage device and a three-dimensional data acquisition device based on the embodiments, but the present disclosure is not limited to these embodiments. Within the scope of the present disclosure, various forms obtained by performing various modifications that can be thought of by those skilled in the art on the present embodiment and forms obtained by combining constituent elements in different embodiments are all included in the scope of one or more forms.

[0386] Industrial Applicability

[0387] The present disclosure is applicable to a three-dimensional data storage device and a three-dimensional data acquisition device.

[0388] Description of the attached figure

[0389] 4601 Three-dimensional data encoding system

[0390] 4602 3D data decoding system

[0391] 4603 Sensor Terminal

[0392] 4604 External connection

[0393] 4611 Point Group Data Generation System

[0394] 4612 Prompt Department

[0395] 4613 Coding Department

[0396] 4614 Multiplexing Department

[0397] 4615 Input and Output

[0398] 4616 Control Department

[0399] 4617 Sensor Information Acquisition Department

[0400] 4618 Point Group Data Generation Department

[0401] 4621 Sensor Information Acquisition Department

[0402] 4622 Input and Output

[0403] 4623 Inverse Multiplexing Department

[0404] 4624 Decoding Department

[0405] 4625 Prompt Department

[0406] 4626 User Interface

[0407] 4627 Control Department

[0408] 4630 1st Coding Department

[0409] 4631 Location Information Encoding Unit

[0410] 4632 Attribute Information Encoding Unit

[0411] 4633 Additional Information Encoding Unit

[0412] 4634 Multiplexing Unit

[0413] 4640 First Decoding Unit

[0414] 4641 Demultiplexing Unit

[0415] 4642 Location Information Decoding Unit

[0416] 4643 Attribute Information Decoding Unit

[0417] 4644 Additional Information Decoding Unit

[0418] 4650 Second Encoding Unit

[0419] 4651 Additional Information Generation Unit

[0420] 4652 Location Image Generation Unit

[0421] 4653 Attribute Image Generation Unit

[0422] 4654 Video Encoding Unit

[0423] 4655 Additional Information Encoding Unit

[0424] 4656 Multiplexing Unit

[0425] 4660 Second Decoding Unit

[0426] 4661 Demultiplexing Unit

[0427] 4662 Video Decoding Unit

[0428] 4663 Additional Information Decoding Unit

[0429] 4664 Location Information Generation Unit

[0430] 4665 Attribute Information Generation Unit

[0431] 4670 Encoding Unit

[0432] 4671 Multiplexing Unit

[0433] 4680 Decoding Unit

[0434] 4681 Demultiplexing Unit

[0435] 4710 First Multiplexing Unit

[0436] 4711 File Transformation Unit

[0437] 4720 First Demultiplexing Unit

[0438] 4721 File Inverse Transformation Unit

[0439] 4730 Second Multiplexing Unit

[0440] 4731 File Transformation Unit

[0441] 4740 Second Demultiplexing Unit

[0442] 4741 File Inverse Transformation Unit

[0443] 4750 Third Multiplexing Unit

[0444] 4751 File Transformation Unit

[0445] 4760 Third Demultiplexing Unit

[0446] 4761 File Inverse Transformation Unit

Claims

1. A three-dimensional data storage method, comprising: Obtaining one or more units storing encoded information generated by encoding three-dimensional data, where the one or more units have a format corresponding to an encoding method used in the encoding of the three-dimensional data among multiple encoding methods; and Storing the one or more units in a file, where The file conforms to a predetermined file format, Each of the one or more units includes a header and a payload, the payload includes encoded data or metadata, and the header includes information indicating the type of data included in the payload.

2. The three-dimensional data storage method according to claim 1, where The file indicates that the data stored in the file is data generated by encoding the three-dimensional data.

3. The three-dimensional data storage method according to claim 1, where The file indicates the encoding method used in the encoding of the three-dimensional data among the multiple encoding methods.

4. The three-dimensional data storage method according to claim 1, where The multiple encoding methods include a method of encoding position information representing the position of the three-dimensional data by an N-ary tree and encoding attribute information with reference to the position information, where N is an integer greater than or equal to 2.

5. The three-dimensional data storage method according to claim 1, where The multiple encoding methods include a method of generating a two-dimensional image from the three-dimensional data and encoding the two-dimensional image using an image encoding method.

6. The three-dimensional data storage method according to claim 1, where The predetermined file format corresponds to the ISO Base Media File Format, i.e., ISOBMFF.

7. A three-dimensional data acquisition method, comprising: Obtaining one or more units storing encoded information generated by encoding three-dimensional data, where the one or more units have a format corresponding to an encoding method used in the encoding of the three-dimensional data among multiple encoding methods; and Obtaining the one or more units from the file, where The file conforms to a predetermined file format, Each of the one or more units includes a header and a payload, the payload includes encoded data or metadata, and the header includes information indicating the type of data included in the payload.

8. The three-dimensional data acquisition method according to claim 7, where The file indicates that the data stored in the file is data generated by encoding the three-dimensional data.

9. The three-dimensional data acquisition method according to claim 7, where The file indicates the encoding method used in the encoding of the three-dimensional data among the multiple encoding methods.

10. The three-dimensional data acquisition method according to claim 7, where The multiple encoding methods include a method of encoding position information representing the position of the three-dimensional data by an N-ary tree and encoding attribute information with reference to the position information, where N is an integer greater than or equal to 2.

11. The three-dimensional data acquisition method according to claim 7, where The plurality of encoding methods include generating a two-dimensional image from the three-dimensional data and encoding the two-dimensional image using an image encoding method.

12. The method for obtaining three-dimensional data according to claim 7, wherein the predetermined file format corresponds to the ISO Base Media File Format, i.e., ISOBMFF.

13. A three-dimensional data storage device, comprising: a processor; and a memory, wherein the processor uses the memory to: obtain one or more units storing encoded information generated by encoding three-dimensional data, the one or more units having a format corresponding to an encoding method used in the encoding of the three-dimensional data among a plurality of encoding methods; and store the one or more units in a file, the file conforming to a predetermined file format, each of the one or more units includes a header and a payload, the payload includes encoded data or metadata, and the header includes information indicating the type of data included in the payload.

14. A three-dimensional data obtaining device, comprising: a processor; and a memory, wherein the processor uses the memory to: obtain one or more units storing encoded information generated by encoding three-dimensional data, the one or more units having a format corresponding to an encoding method used in the encoding of the three-dimensional data among a plurality of encoding methods; and obtain the one or more units from the file, the file conforming to a predetermined file format, each of the one or more units includes a header and a payload, the payload includes encoded data or metadata, and the header includes information indicating the type of data included in the payload.

Images