Three-dimensional data processing method and three-dimensional data processing device

Abstract

To provide a three-dimensional data multiplexing method, a three-dimensional data reverse multiplexing method, a three-dimensional data multiplexing device and a three-dimensional data reverse multiplexing device are provided which appropriately multiplex dot group data and transmit multiplexed data.SOLUTION: A three-dimensional data processing method includes: generating output signals for a predetermined file composition including multiplexed plural kinds of data and meta data obtained by multiplexing plural kinds of data containing three-dimensional data (S7311); and storing, in the meta data, kind information representing respective kinds of the plural kinds of data, and information associating the plural kinds of data with each other (S7312). The kind information includes positional information on a sensor when at least one kind of data among the plural kinds of data is generated, and angular information or time-of-day information on the sensor.SELECTED DRAWING: Figure 33

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional data multiplexing method, a three-dimensional data demultiplexing method, a three-dimensional data multiplexing apparatus, and a three-dimensional data demultiplexing apparatus.

Background Art

[0002] In the future, the spread of devices or services that utilize three-dimensional data is expected in a wide range of fields such as computer vision, map information, monitoring, infrastructure inspection, or video distribution for autonomous operation of automobiles or robots. Three-dimensional data is acquired by various methods such as distance sensors such as lidar, stereo cameras, or combinations of multiple monocular cameras.

[0003] As one of the methods for expressing three-dimensional data, there is a method called point cloud that represents the shape of a three-dimensional structure by a point group in a three-dimensional space. In a point cloud, the positions and colors of the point group are stored. Although the point cloud is expected to become the mainstream as a method for expressing three-dimensional data, the point group has a very large amount of data. Therefore, in the accumulation or transmission of three-dimensional data, as in the case of two-dimensional moving images (for example, MPEG-4 AVC or HEVC standardized by MPEG), compression of the data amount by encoding is essential.

[0004] In addition, regarding the compression of point clouds, it is partially supported by public libraries (Point Cloud Library) that perform point cloud-related processing.

[0005] In addition, a technique for searching for and displaying facilities located around a vehicle using three-dimensional map data is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] In addition, a method for multiplexing and transmitting point cloud data is required.

[0008] An object of the present disclosure is to provide a three-dimensional data multiplexing method, a three-dimensional data demultiplexing method, a three-dimensional data multiplexing apparatus, or a three-dimensional data demultiplexing apparatus that can appropriately multiplex and transmit point cloud data.

MEANS FOR SOLVING THE PROBLEMS

[0009] A three-dimensional data processing method according to an aspect of the present disclosure generates an output signal having a predetermined file configuration including the multiplexed plurality of types of data and metadata by multiplexing a plurality of types of data including three-dimensional data, stores type information indicating each type of the plurality of types of data and information linking the plurality of types of data in the metadata, and the type information includes position information, angle information, or time information of a sensor when at least one type of the plurality of types of data is generated.

[0010] A three-dimensional data processing method according to an aspect of the present disclosure acquires, from an output signal having a predetermined file configuration including the multiplexed plurality of types of data and metadata in which the plurality of types of data including three-dimensional data are multiplexed, type information indicating each type of the plurality of types of data and information linking the plurality of types of data stored in the metadata, acquires the plurality of types of data from the output signal using the type information, and the type information includes position information, angle information, or time information of a sensor when at least one type of the plurality of types of data is generated.

EFFECTS OF THE INVENTION

[0011] The present disclosure can provide a three-dimensional data multiplexing method, a three-dimensional data demultiplexing method, a three-dimensional data multiplexing apparatus, or a three-dimensional data demultiplexing apparatus that can appropriately multiplex and transmit point cloud data.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Embodiments for Carrying Out the Invention

[0013] A three-dimensional data multiplexing method according to an aspect of the present disclosure generates an output signal having a predetermined file configuration by multiplexing a plurality of types of data including point cloud data, and stores information indicating the type of each of the plurality of data included in the output signal in metadata in the file configuration.

[0014] According to this, the three-dimensional data multiplexing method stores information indicating the type of each of the plurality of data included in the output signal in the metadata in the file configuration. Thereby, in the three-dimensional data demultiplexing device that receives the output signal, the type of each data can be easily determined. In this way, the three-dimensional data multiplexing method can appropriately multiplex and transmit the point cloud data.

[0015] For example, the information may indicate (1) the coding method applied to the data, (2) the configuration of the data, (3) the type of sensor that generated the data, or (4) the data format.

[0016] For example, the metadata may include synchronization information for synchronizing the times of the plurality of data included in the output signal.

[0017] According to this, in the three-dimensional data demultiplexing device that receives the output signal, the plurality of data can be synchronized.

[0018] For example, the synchronization information may indicate the difference in timestamps among the plurality of data.

[0019] According to this, the data amount of the output signal can be reduced.

[0020] The three-dimensional data demultiplexing method according to one aspect of the present disclosure obtains information indicating each type of a plurality of data included in an output signal having a predetermined file configuration in which a plurality of types of data including point cloud data are multiplexed, from metadata in the file configuration, and uses the information to obtain the plurality of data from the output signal.

[0021] According to this, the three-dimensional data demultiplexing method can easily determine the type of each data.

[0022] For example, the information may indicate (1) an encoding method applied to the data, (2) the configuration of the data, (3) the type of sensor that generated the data, or (4) the data format.

[0023] For example, the metadata may include synchronization information for synchronizing the times of the plurality of data included in the output signal.

[0024] According to this, the three-dimensional data demultiplexing method can synchronize a plurality of data.

[0025] For example, the synchronization information may indicate the difference in timestamps among the plurality of data.

[0026] According to this, the data amount of the output signal can be reduced.

[0027] In addition, a three-dimensional data multiplexing device according to one aspect of the present disclosure includes a processor and a memory. The processor uses the memory to multiplex a plurality of types of data including point cloud data to generate an output signal having a predetermined file configuration, and stores information indicating the type of each of the plurality of data included in the output signal in metadata in the file configuration.

[0028] According to this, the three-dimensional data multiplexing device stores information indicating the type of each of the plurality of data included in the output signal in the metadata in the file configuration. Thereby, in the three-dimensional data demultiplexing device that receives the output signal, the type of each data can be easily determined. In this way, the three-dimensional data multiplexing device can appropriately multiplex and transmit the point cloud data.

[0029] In addition, a three-dimensional data demultiplexing device according to one aspect of the present disclosure includes a processor and a memory. The processor uses the memory to obtain, from an output signal having a predetermined file configuration in which a plurality of types of data including point cloud data are multiplexed, information indicating the type of each of the plurality of data included in the output signal, which is stored in the metadata in the file configuration, and uses the information to obtain the plurality of data from the output signal.

[0030] According to this, the three-dimensional data demultiplexing device can easily determine the type of each data.

[0031] Note that these general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0032] Hereinafter, embodiments will be specifically described with reference to the drawings. Note that all the embodiments described below show specific examples of the present disclosure. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0033] (Embodiment) First, the configuration of the three-dimensional data (point cloud data) encoding / decoding system according to the present embodiment will be described. FIG. 1 is a diagram showing a configuration example of the three-dimensional data encoding / decoding system according to the present embodiment. As shown in FIG. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.

[0034] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data which is three-dimensional data. Note that the three-dimensional data encoding system 4601 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by a plurality of devices. Further, the three-dimensional data encoding device may include a part of a plurality of processing units included in the three-dimensional data encoding system 4601.

[0035] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a presentation unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.

[0036] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603 and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data from the sensor information and outputs the point cloud data to the encoding unit 4613.

[0037] The presentation unit 4612 presents the sensor information or the point cloud data to the user. For example, the presentation unit 4612 displays information or an image based on the sensor information or the point cloud data.

[0038] The encoding unit 4613 encodes (compresses) the point cloud data and outputs the obtained encoded data, the control information obtained in the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, the sensor information.

[0039] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data, the control information, and the additional information input from the encoding unit 4613. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0040] The input / output unit 4615 (for example, a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 4616 (or the application execution unit) controls each processing unit. That is, the control unit 4616 performs control such as encoding and multiplexing.

[0041] Note that the sensor information may be input to the encoding unit 4613 or the multiplexing unit 4614. Also, the input / output unit 4615 may output the point cloud data or the encoded data to the outside as it is.

[0042] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604.

[0043] The three-dimensional data decoding system 4602 generates point cloud data, which is three-dimensional data, by decoding encoded data or multiplexed data. Note that the three-dimensional data decoding system 4602 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by a plurality of devices. Further, the three-dimensional data decoding device may include a part of a plurality of processing units included in the three-dimensional data decoding system 4602.

[0044] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621, an input / output unit 4622, a demultiplexing unit 4623, a decoding unit 4624, a presentation unit 4625, a user interface 4626, and a control unit 4627.

[0045] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603.

[0046] The input / output unit 4622 acquires a transmission signal, decodes multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexing unit 4623.

[0047] The demultiplexing unit 4623 acquires encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 4624.

[0048] The decoding unit 4624 reconstructs point cloud data by decoding the encoded data.

[0049] The presentation unit 4625 presents the point cloud data to the user. For example, the presentation unit 4625 displays information or an image based on the point cloud data. The user interface 4626 acquires an instruction based on the user's operation. The control unit 4627 (or the application execution unit) controls each processing unit. That is, the control unit 4627 performs control such as demultiplexing, decoding, and presentation.

[0050] Note that the input / output unit 4622 may directly acquire point cloud data or encoded data from the outside. Further, the presentation unit 4625 may acquire additional information such as sensor information and present information based on the additional information. Further, the presentation unit 4625 may perform presentation based on an instruction of the user acquired by the user interface 4626.

[0051] The sensor terminal 4603 generates sensor information which is information obtained by a sensor. The sensor terminal 4603 is a terminal equipped with a sensor or a camera, and examples thereof include a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

[0052] The sensor information that can be acquired by the sensor terminal 4603 is, for example, (1) the distance (position information), color, or reflectivity of the object from the sensor terminal 4603 obtained from a LIDAR, millimeter-wave radar, or infrared sensor, (2) the distance (position information), color, or reflectivity of the object from the camera obtained from a plurality of monocular camera images or stereo camera images, etc. Further, the sensor information may include the attitude, orientation, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, or acquisition time of the sensor information of the sensor. Further, the sensor information may include temperature, atmospheric pressure, humidity, or magnetism, etc.

[0053] The external connection unit 4604 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, or broadcasting or the like.

[0054] Next, the point cloud data will be described. FIG. 2 is a diagram showing the configuration of the point cloud data. FIG. 3 is a diagram showing a configuration example of a data file in which information of the point cloud data is described.

[0055] The point cloud data includes data of a plurality of points. The data of each point includes position information (three-dimensional coordinates) and attribute information for the position information. A collection of such a plurality of points is called a point cloud. For example, the point cloud shows the three-dimensional shape of an object.

[0056] Position information such as three-dimensional coordinates is sometimes referred to as geometry. Also, the data for each point may include attribute information (attribute) of multiple attribute types. The attribute types are, for example, color or reflectance.

[0057] One piece of attribute information may be associated with one piece of position information, or attribute information having multiple different attribute types may be associated with one piece of position information. Also, multiple pieces of attribute information of the same attribute type may be associated with one piece of position information.

[0058] The configuration example of the data file shown in FIG. 3 is an example where the position information and the attribute information correspond one-to-one, and shows the position information and the attribute information of N points constituting the point cloud data.

[0059] The position information is, for example, information on three axes of x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file or the like.

[0060] Next, the types of point cloud data will be described. FIG. 4 is a diagram showing the types of point cloud data. As shown in FIG. 4, the point cloud data includes a static object and a dynamic object.

[0061] The static object is three-dimensional point cloud data at an arbitrary time (a certain moment). The dynamic object is three-dimensional point cloud data that changes over time. Hereinafter, the three-dimensional point cloud data at a certain moment is called a PCC frame or a frame.

[0062] The object may be a point cloud with a limited area like ordinary video data, or a large-scale point cloud with an unlimited area like map information.

[0063] Also, there is point cloud data of various densities, and there may be sparse point cloud data and dense point cloud data.

[0064] Details of each processing unit will be described below. Sensor information is obtained by various methods such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of a plurality of monocular cameras. The point cloud data generation unit 4618 generates point cloud data based on the sensor information obtained by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as the point cloud data and adds attribute information for the position information to the position information.

[0065] When generating the position information or adding the attribute information, the point cloud data generation unit 4618 may process the point cloud data. For example, the point cloud data generation unit 4618 may reduce the data amount by deleting point clouds with overlapping positions. Also, the point cloud data generation unit 4618 may convert the position information (such as position shift, rotation, or normalization), or may render the attribute information.

[0066] Note that in FIG. 1, the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but may be provided independently outside the three-dimensional data encoding system 4601.

[0067] The encoding unit 4613 generates encoded data by encoding the point cloud data based on a predefined encoding method. There are roughly two types of encoding methods as follows. The first is an encoding method using position information, which will be described as the first encoding method hereinafter. The second is an encoding method using a video codec, which will be described as the second encoding method hereinafter.

[0068] The decoding unit 4624 decodes the point cloud data by decoding the encoded data based on a predefined encoding method.

[0069] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. The multiplexing unit 4614 multiplexes, in addition to the PCC encoded data, other media such as video, audio, subtitles, applications, files, or reference time information. Further, the multiplexing unit 4614 may further multiplex sensor information or attribute information related to point cloud data.

[0070] Examples of the multiplexing method or file format include ISOBMFF, MPEG-DASH which is an ISOBMFF-based transmission method, MMT, MPEG-2 TS Systems, RMP, and the like.

[0071] The demultiplexing unit 4623 extracts PCC encoded data, other media, time information, etc. from the multiplexed data.

[0072] The input / output unit 4615 transmits the multiplexed data using a method suitable for the medium for transmission such as broadcasting or communication, or the medium for storage. The input / output unit 4615 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0073] Examples of the communication protocol include http, ftp, TCP, UDP, or IP. A PULL-type communication method may be used, or a PUSH-type communication method may be used.

[0074] Either wired transmission or wireless transmission may be used. Examples of wired transmission include Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), or coaxial cable. Examples of wireless transmission include 3G / 4G / 5G defined by IEEE of 3GPP (registered trademark), wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), or millimeter wave.

[0075] Examples of the broadcasting method include DVB-T2, DVB-S2, DVB-C2, ATSC 3.0, or ISDB-S3.

[0076] FIG. 5 is a diagram showing the configuration of a first encoding unit 4630 which is an example of an encoding unit 4613 that performs encoding according to a first encoding method. FIG. 6 is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data according to the first encoding method. The first encoding unit 4630 includes a position information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0077] The first encoding unit 4630 is characterized by performing encoding while being aware of the three-dimensional structure. Further, the first encoding unit 4630 is characterized in that the attribute information encoding unit 4632 performs encoding using the information obtained from the position information encoding unit 4631. The first encoding method is also called GPCC (Geometry based PCC).

[0078] The point cloud data is PCC point cloud data such as a PLY file or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information. The position information is input to the position information encoding unit 4631, the attribute information is input to the attribute information encoding unit 4632, and the additional information is input to the additional information encoding unit 4633.

[0079] The position information encoding unit 4631 generates encoded position information (Compressed Geometry), which is encoded data, by encoding the position information. For example, the position information encoding unit 4631 encodes the position information using an N-ary tree structure such as an octree. Specifically, in an octree, the target space is divided into eight nodes (sub-spaces), and 8-bit information (occupancy code) indicating whether or not each node contains a point cloud is generated. Further, a node containing a point cloud is further divided into eight nodes, and 8-bit information indicating whether or not each of the eight nodes contains a point cloud is generated. This process is repeated until the number of point clouds included in a predetermined layer or node becomes equal to or less than a threshold value.

[0080] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute), which is encoded data, by encoding using the configuration information generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 determines a reference point (reference node) to be referred to in the encoding of the target point (target node) to be processed based on the octree structure generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 refers to a node among the peripheral nodes or adjacent nodes whose parent node in the octree is the same as the target node. Note that the method for determining the reference relationship is not limited to this.

[0081] Also, the encoding process of the attribute information may include at least one of quantization processing, prediction processing, and arithmetic encoding processing. In this case, reference means using the reference node to calculate the predicted value of the attribute information, or using the state of the reference node (for example, occupancy information indicating whether the reference node includes a point cloud) to determine the encoding parameter. For example, the encoding parameter is a quantization parameter in quantization processing or a context in arithmetic encoding.

[0082] The additional information encoding unit 4633 generates encoded additional information (Compressed MetaData), which is encoded data, by encoding compressible data among the additional information.

[0083] The multiplexing unit 4634 generates an encoded stream (Compressed Stream), which is encoded data, by multiplexing the encoded position information, encoded attribute information, encoded additional information, and other additional information. The generated encoded stream is output to a processing unit in a system layer (not shown).

[0084] Next, a description will be given of a first decoder 4640, which is an example of a decoder 4624 that decodes using the first encoding method. FIG. 7 is a diagram showing the configuration of the first decoder 4640. FIG. 8 is a block diagram of the first decoder 4640. The first decoder 4640 generates point cloud data by decoding encoded data (encoded stream) encoded using the first encoding method using the first encoding method. This first decoder 4640 includes a demultiplexing unit 4641, a position information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0085] An encoded stream (Compressed Stream), which is encoded data, is input to the first decoder 4640 from a processing unit in a system layer (not shown).

[0086] The demultiplexing unit 4641 separates encoded position information (Compressed Geometry), encoded attribute information (Compressed Attribute), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0087] The position information decoding unit 4642 generates position information by decoding the encoded position information. For example, the position information decoding unit 4642 restores the position information of the point cloud represented by three-dimensional coordinates from the encoded position information represented by an N-ary tree structure such as an octree.

[0088] The attribute information decoding unit 4643 decodes the encoded attribute information based on the configuration information generated by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 determines a reference point (reference node) to be referred to in decoding the target point (target node) to be processed based on the octree structure obtained by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 refers to a node in which the parent node in the octree is the same as the target node among the surrounding nodes or adjacent nodes. Note that the method for determining the reference relationship is not limited to this.

[0089] Also, the decoding process of the attribute information may include at least one of inverse quantization processing, prediction processing, and arithmetic decoding processing. In this case, reference means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (for example, occupancy information indicating whether a point cloud is included in the reference node) to determine decoding parameters. For example, the decoding parameters are quantization parameters in inverse quantization processing or contexts in arithmetic decoding, etc.

[0090] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. Also, the first decoding unit 4640 uses the additional information required for the decoding processes of the position information and the attribute information during decoding and outputs the additional information required for the application to the outside.

[0091] Next, a second encoding unit 4650, which is an example of an encoding unit 4613 that performs encoding using the second encoding method, will be described. FIG. 9 is a diagram showing the configuration of the second encoding unit 4650. FIG. 10 is a block diagram of the second encoding unit 4650.

[0092] The second encoding unit 4650 generates encoded data (encoded stream) by encoding point cloud data using the second encoding method. This second encoding unit 4650 includes an additional information generation unit 4651, a position image generation unit 4652, an attribute image generation unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0093] The second encoding unit 4650 is characterized by generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image and encoding the generated position image and attribute image using an existing video encoding method. The second encoding method is also called VPCC (Video based PCC).

[0094] The point cloud data is PCC point cloud data such as a PLY file or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).

[0095] The additional information generation unit 4651 generates map information of a plurality of two-dimensional images by projecting the three-dimensional structure onto a two-dimensional image.

[0096] The position image generation unit 4652 generates a position image (Geometry Image) based on the position information and the map information generated by the additional information generation unit 4651. This position image is, for example, a distance image in which the distance (Depth) is indicated as a pixel value. Note that this distance image may be an image of a plurality of point clouds viewed from one viewpoint (an image obtained by projecting a plurality of point clouds onto one two-dimensional plane), or may be a plurality of images of a plurality of point clouds viewed from a plurality of viewpoints, or may be a single image obtained by integrating these plurality of images.

[0097] The attribute image generation unit 4653 generates an attribute image based on the attribute information and the map information generated by the additional information generation unit 4651. This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is indicated as a pixel value. Note that this image may be an image of a plurality of point clouds viewed from one viewpoint (an image obtained by projecting a plurality of point clouds onto one two-dimensional plane), or may be a plurality of images of a plurality of point clouds viewed from a plurality of viewpoints, or may be a single image obtained by integrating these plurality of images.

[0098] The video encoding unit 4654 generates an encoded position image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image), which are encoded data, by encoding the position image and the attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC or HEVC or the like.

[0099] The additional information encoding unit 4655 generates encoded additional information (Compressed MetaData) by encoding the additional information included in the point cloud data and map information and the like.

[0100] The multiplexing unit 4656 generates an encoded stream (Compressed Stream), which is encoded data, by multiplexing the encoded position image, the encoded attribute image, the encoded additional information, and other additional information. The generated encoded stream is output to a processing unit in a system layer (not shown).

[0101] Next, a second decoding unit 4660, which is an example of a decoding unit 4624 that decodes using the second encoding method, will be described. FIG. 11 is a diagram showing the configuration of the second decoding unit 4660. FIG. 12 is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point cloud data by decoding encoded data (encoded stream) encoded using the second encoding method using the second encoding method. This second decoding unit 4660 includes a demultiplexing unit 4661, a video decoding unit 4662, an additional information decoding unit 4663, a position information generation unit 4664, and an attribute information generation unit 4665.

[0102] An encoded stream (Compressed Stream), which is encoded data, is input to the second decoding unit 4660 from a processing unit in a system layer (not shown).

[0103] The demultiplexing unit 4661 separates the encoded position image (Compressed Geometry Image), the encoded attribute image (Compressed Attribute Image), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0104] The video decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC or HEVC, etc.

[0105] The additional information decoding unit 4663 generates additional information including map information and the like by decoding the encoded additional information.

[0106] The position information generation unit 4664 generates position information using the position image and the map information. The attribute information generation unit 4665 generates attribute information using the attribute image and the map information.

[0107] The second decoding unit 4660 uses the additional information necessary for decoding during decoding and outputs the additional information necessary for the application to the outside.

[0108] Hereinafter, the PCC encoding method will be described. FIG. 13 is a diagram showing a protocol stack related to PCC encoded data. FIG. 13 shows an example in which data of other media such as video (for example, HEVC) or audio, or sensor information is multiplexed with the PCC encoded data and transmitted or stored.

[0109] The multiplexing method and the file format have functions for multiplexing various encoded data and transmitting or storing it. In order to transmit or store the encoded data, the encoded data is converted into the format of the multiplexing method. For example, in HEVC, a technique is defined in which the encoded data is stored in a data structure called a NAL unit, and the NAL unit is stored in ISOBMFF.

[0110] A similar configuration is assumed in PCC. The sensor information, together with the point cloud data, may be PCC encoded, may be encoded using another encoding method, or may be directly stored in the multiplex layer without being encoded, and these can be combined. The other encoding method is specifically another three-dimensional encoding method or an encoding method for encoding data obtained by converting the point cloud data into two-dimensional or one-dimensional data.

[0111] Hereinafter, an example of a configuration for generating point cloud data from a sensor signal (also referred to as sensor information) will be described. FIGS. 14 to 16 are diagrams each showing a configuration example of a point cloud data generation device that generates point cloud data from a sensor signal.

[0112] The point cloud data generation device shown in FIG. 14 generates point cloud data from sensor signals obtained from one sensing device 7301. The point cloud data generation device shown in FIG. 14 includes a sensing device 7301, a sensor information input unit 7302, and a point cloud data generation unit 7303. The sensor information input unit 7302 acquires the sensor signals obtained by the sensing device 7301. The point cloud data generation unit 7303 generates point cloud data from the sensor signals acquired by the sensor information input unit 7302. The generated point cloud data is output to an encoding unit (not shown) for subsequent point cloud data, for example.

[0113] As shown in FIG. 15, point cloud data may be generated based on sensor signals obtained from two or more sensing devices. The point cloud data generation device shown in FIG. 15 includes sensing devices 7301A and 7301B, sensor information input units 7302A and 7302B, and a point cloud data generation unit 7303A. The sensor information input unit 7302A acquires the first sensor signal obtained by the sensing device 7301A. The sensor information input unit 7302B acquires the second sensor signal obtained by the sensing device 7301B. The point cloud data generation unit 7303A generates point cloud data from the two sensor signals acquired by the sensor information input units 7302A and 7302B. The generated point cloud data is output to an encoding unit (not shown) for subsequent point cloud data, for example.

[0114] The point cloud data generation device shown in FIG. 16 includes a sensing device 7301C, a sensor information input unit 7302C, and a point cloud data generation unit 7303C. The sensing device 7301C generates a sensor signal in which two pieces of information sensed using two or more sensing methods are merged in a predetermined method. The sensing device 7301C includes sensing units 7304A and 7304B and a merging unit 7305.

[0115] The sensing unit 7304A generates a first sensor signal by a first sensing method. The sensing unit 7304B generates a second sensor signal by a second sensing method. The merging unit 7305 merges the first sensor signal and the second sensor signal, and outputs the generated sensor signal to the sensor information input unit 7302C.

[0116] Note that the merging unit 7305 may select one of the first sensor signal and the second sensor signal based on a predetermined condition and output the selected sensor signal. Also, when the merging unit 7305 merges two sensor signals, it may change the weighting factor used for the merging.

[0117] For example, the merging unit 7305 may determine which sensor signal to select based on the acquired sensor signal or based on another sensor signal.

[0118] For example, the first sensing method and the second sensing method may have different sensor parameters, or different sensing frequencies or mechanisms. Also, the center signal may include information indicating the sensing method or parameters during sensing.

[0119] When the merging unit 7305 switches between a plurality of sensing methods, it may include in the sensor signal information indicating which sensing method was used or data on the switching determination criteria. When the merging unit 7305 merges sensor signals, it may include in the sensor signal information for identifying the merged sensing method, data on the merging determination criteria, or a merging coefficient.

[0120] Also, the sensing device 7301C may output a plurality of sensor signals. Further, as the plurality of sensor signals, the sensing device 7301C may output the absolute value of the first sensor signal and the difference value between the first sensor signal and the second sensor signal.

[0121] Further, the sensor signal may include information indicating the relationship between the first sensing method and the second sensing method. For example, the sensor signal may include the absolute value or relative value of the reference position information of the first sensing method and the second sensing method, or may include the acquisition time of the sensor signal, the reference time information, or information indicating the angle of the sensor. By including this information in the sensor signal, it becomes possible to correct or synthesize the relationship between the two sensor signals based on this information in the subsequent processing.

[0122] The sensor information input unit 7302C acquires the sensor signal obtained by the sensing device 7301C. The point cloud data generation unit 7303C generates point cloud data from the sensor signal acquired by the sensor information input unit 7302C. The generated point cloud data is output to an encoding unit (not shown) for the subsequent point cloud data, for example.

[0123] In this way, the point cloud data generation device generates point cloud data based on any one or two or more of the above various sensor signals. Note that the point cloud data generation device may correct the position information or attribute information of the points during the generation process of the point cloud data.

[0124] Note that the point cloud data generation device may have the configuration shown in any of FIGS. 14 to 16, or may have a configuration combining a plurality of these. Also, the point cloud data generation device may use a fixed method, or may adaptively change the method used according to, for example, the purpose of sensing or the use case.

[0125] Next, a configuration example of the point cloud data encoding system according to the present embodiment will be described. FIG. 17 is a diagram showing a configuration example of the point cloud data encoding system according to the present embodiment. The point cloud data encoding system shown in FIG. 17 includes a first device 7310 and a second device 7320.

[0126] The first device 7310 includes a sensing unit 7311 and an output I / F (interface) 7312. The second device 7320 includes a sensing unit 7321, an output I / F 7322, an input I / F 7323, and a processing unit 7324. The processing unit 7324 includes a point cloud data generation unit 7325 and an encoding unit 7326.

[0127] The sensing unit 7311 or 7321 may be included in the same hardware or device as the processing unit 7324 composed of a CPU or the like, or may be included in different hardware or devices.

[0128] The sensing unit 7321 is included in the same device (the second device 7320) as the processing unit 7324. In this case, the output signal (referred to as RAW data) of the sensing unit 7321 is directly input to the point cloud data generation unit 7325.

[0129] The sensing unit 7311 is included in a device different from the processing unit 7324 (the first device 7310). In this case, the RAW data output from the sensing unit 7311 is converted into an input / output format (external output format) at the output I / F 7312, and the formatted signal is input to the second device 7320. The input I / F 7323 included in the second device 7320 converts the formatted signal into RAW data and outputs the obtained RAW data to the point cloud data generation unit 7325. The output I / F 7312 and the input I / F 7323 have the functions of, for example, the multiplexing unit 4614 and the input / output unit 4615 shown in FIG. 1.

[0130] Alternatively, the output signal (RAW data) from the sensing unit 7321 included in the same device as the processing unit 7324 may be converted into an input / output format by the output I / F 7322, the formatted signal may be converted into RAW data at the input I / F 7323, and the obtained RAW data may be input to the point cloud data generation unit 7325.

[0131] Also, when multiple sensor signals are input, for example, when sensor signals input from other devices and sensor signals input from the same device are mixed, these sensor signals may be converted into the same format. Also, at the time of conversion, an identifier capable of identifying the signal may be assigned to each signal. For example, when transmission is performed using UDP (User Datagram Protocol), each signal may be identified by the IP (Internet Protocol) or UDP source address or source port number. As a result, the format input to the point cloud data generation unit 7325 can be unified, facilitating signal control.

[0132] The point cloud data generation unit 7325 generates point cloud data using the input RAW data. The encoding unit 7326 encodes the generated point cloud data.

[0133] Next, a configuration example of the three-dimensional data multiplexing apparatus (three-dimensional data multiplexing system) according to the present embodiment will be described. FIG. 18 is a diagram showing a configuration example of the three-dimensional data multiplexing apparatus according to the present embodiment. The three-dimensional data multiplexing apparatus generates an output signal by encoding and multiplexing various sensor signals, and accumulates or transmits the generated output signal.

[0134] As shown in FIG. 18, the three-dimensional data multiplexing apparatus includes sensing units 7331A, 7331B, and 7331C, sensor information input units 7332A, 7332B, and 7332C, point cloud data generation units 7333A and 7333B, encoding units 7334A and 7334B, a synchronization unit 7335, and a multiplexing unit 7336. Here, an example in which three sensing units are used is shown, but the number of sensing units is not limited to this. Also, any combination of the following processing methods can be used for the processing method of the sensor signal from each sensing unit.

[0135] The sensor information input unit 7332A acquires the first sensor signal generated by sensing in the sensing unit 7331A. The sensor information input unit 7332B acquires the second sensor signal generated by sensing in the sensing unit 7331B. The sensor information input unit 7332C acquires the third sensor signal generated by sensing in the sensing unit 7331C.

[0136] The point cloud data generation unit 7333A generates the first point cloud data from the first sensor signal. The point cloud data generation unit 7333B generates the second point cloud data from the second sensor signal. At this time, due to differences in the sensing methods (for example, direction, range, acquirable attributes, frequency, resolution, etc., methods, or means) used by the sensing unit 7331A and the sensing unit 7331B, the number of points, the range of points, and the attribute information in the generated point cloud data may be different.

[0137] The encoding unit 7334A generates the first encoded data by encoding the first point cloud data. The encoding unit 7334B generates the second encoded data by encoding the second point cloud data. For example, the encoding unit 7334A and the encoding unit 7334B apply different encoding methods. For example, the encoding unit 7334A may use the first encoding method, and the encoding unit 7334B may use a second encoding method different from the first encoding method. Note that the encoding unit 7334A and the encoding unit 7334B may use the same encoding method.

[0138] The encoding units 7334A and 7334B may compress the position information or attribute information of the points in the point cloud data using entropy encoding or the like. Further, the encoding units 7334A and 7334B may store sensor signals, sensor position information or angle information, or time information as metadata.

[0139] The symbolization units 7334A and 7334B use a symbolization method suitable for the point cloud data. For example, the first symbolization method is a symbolization method that can expect a high symbolization rate for map information or still content, and the second symbolization method is a symbolization method that can expect a high symbolization rate for content such as AR or VR. In this case, the symbolization units 7334A and 7334B may use a symbolization method suitable for the content.

[0140] Alternatively, for example, the first symbolization method is a symbolization method that can expect a high symbolization rate for a point cloud based on information sensed by a sensing unit such as beam LiDAR, and the second symbolization method is a symbolization method that can expect a high symbolization rate for a point cloud based on information sensed by a sensing unit such as FLASH LiDAR. In this case, the symbolization units 7334A and 7334B may use a symbolization method suitable for the sensing unit.

[0141] Also, the symbolization units 7334A and 7334B may use a symbolization tool or a parameter related to symbolization suitable for the content or the sensing unit in the same symbolization method instead of changing the symbolization method.

[0142] The generated first symbolized data and second symbolized data are input to the multiplexing unit 7336. For example, the third sensor signal sensed by the sensing unit 7331C is data that does not require symbolization. In this case, generation and symbolization of the point cloud data are not performed, and the third sensor signal is directly input to the multiplexing unit 7336. Note that symbolization may not be performed for the purpose of low-latency transmission.

[0143] The synchronization unit 7335 has a function for synchronizing a plurality of sensing units. For example, the synchronization unit 7335 uses, as information related to synchronization, sensing time information, timestamp information, angle information, and the like. The information related to these synchronizations may be multiplexed into the output signal as a common synchronization signal. Alternatively, the information related to these synchronizations may be included in each sensor signal.

[0144] The multiplexing unit 7336 generates an output signal by multiplexing one or more encoded data, metadata, RAW data of sensor signals, and synchronization signals. Further, the multiplexing unit 7336 stores information for identifying each data and information indicating the correspondence of each data in the output signal.

[0145] FIG. 19 is a diagram showing a specific example of a three-dimensional data multiplexing device. As shown in FIG. 19, a beam LiDAR is used as the sensing unit 7331A, and a FLASH LiDAR is used as the sensing unit 7331B. Depending on the characteristics of the LiDAR, the range and distance of the point cloud, as well as the resolution, etc. are different. FIG. 20 is a diagram showing an example of the sensor ranges of the beam LiDAR and the FLASH LiDAR. For example, the beam LiDAR detects all directions around the vehicle (sensor), and the FLASH LiDAR detects the range in one direction (e.g., the front) of the vehicle.

[0146] The point cloud data generation unit 7333A generates first point cloud data based on the distance information and reflectance information with respect to the beam irradiation angle obtained from the beam LiDAR. The point cloud data generation unit 7333B generates second point cloud data based on the two-dimensional distance information and reflectance obtained from the FLASH LiDAR. Note that the point cloud data generation units 7333A and 7333B may further use two-dimensional color information obtained by a camera, etc., and generate point cloud data having both color information and reflectance.

[0147] In addition, as the sensing unit 7331C, an in-vehicle sensor such as a three-axis gyro sensor, a three-axis acceleration sensor, or a position information sensor such as GPS is used. These sensor information represents the state of the entire vehicle, and can also be said to be common information related to the first sensor signal and the second sensor signal. These common sensor information may be encoded and multiplexed, or may be multiplexed without being encoded. Further, these information may be stored and encoded in the first sensor signal or the second sensor signal as additional information common to the point cloud data. Alternatively, the common sensor information may be stored in one of the first sensor signal and the second sensor signal. In this case, information indicating in which sensor signal the common sensor information is stored may be indicated in, for example, other sensor signals or synchronization signals.

[0148] In addition, as information regarding the time when the sensor is acquired, for example, a time stamp based on reference time information such as NTP (Network Time Protocol) or PTP (Precision Time Protocol) is attached to the first point cloud data based on beam LiDAR and the second point cloud data based on FLASH LiDAR. The time stamps of the respective sensors are synchronized with a common reference time and are encoded by the encoding units 7334A and 7334B.

[0149] In addition, the reference time information indicating the common reference time may be multiplexed as a synchronization signal. The reference time information does not have to be multiplexed. The three-dimensional data demultiplexing device (three-dimensional data decoding device) acquires the respective time stamps from the encoded data of the plurality of sensor signals. Since the time stamps are synchronized with a common reference time, the three-dimensional data demultiplexing device can synchronize between the plurality of sensors by operating the decoded data of the plurality of sensor signals based on the respective time stamps.

[0150] Note that corresponding time information may be set for each of the beam LiDAR and the FLASH LiDAR. Further, a three-axis sensor may be provided for each of the beam LiDAR and the FLASH LiDAR. In that case, a common time such as the Internet time is used as each NTP. Further, each three-axis sensor is pre-calibrated, and a plurality of three-axis sensors that are pre-synchronized are used.

[0151] FIG. 21 is a diagram showing another configuration example of the three-dimensional data multiplexing device. As shown in FIG. 21, the three-dimensional data multiplexing device includes sensing units 7341A, 7341B, and 7341C, input / output units 7342A, 7342B, and 7342C, a point cloud data generation unit 7343, encoding units 7344A and 7344B, a synchronization unit 7345, and a multiplexing unit 7346.

[0152] The input / output unit 7342A acquires the first sensor signal generated by sensing in the sensing unit 7341A. The input / output unit 7342B acquires the second sensor signal generated by sensing in the sensing unit 7341B. The input / output unit 7342C acquires the third sensor signal generated by sensing in the sensing unit 7341C. Note that the input / output units 7342A, 7342B, and 7342C may have a memory for storing the acquired sensor signals.

[0153] The point cloud data generation unit 7343 generates first point cloud data from the first sensor signal. The encoding unit 7344A generates first encoded data by encoding the first point cloud data. The encoding unit 7344B generates second encoded data by encoding the second sensor information.

[0154] The synchronization unit 7345 has a function for synchronizing a plurality of sensing units. The multiplexing unit 7346 generates an output signal by multiplexing one or more encoded data, metadata, RAW data of sensor signals, and synchronization signals.

[0155] Thus, in the configuration shown in FIG. 21, point cloud data is not generated from the sensor signals (RAW data) obtained by the sensing unit 7341B, and the sensor signals are encoded as RAW data. For example, when the second sensor signal is two-dimensional information obtained by a CMOS sensor such as a FLASH LiDAR or a camera, the encoding unit 7344B encodes the second sensor signal using a video codec such as AVC or HEVC. Thereby, highly efficient encoding can be realized. Also, by utilizing existing codecs, it becomes possible to construct a low-cost system.

[0156] Thus, the three-dimensional data multiplexing device combines, according to the sensing unit, means for encoding after converting point cloud data and means for encoding RAW data as it is without converting it into point cloud data, and multiplexes the respective encoded data.

[0157] Next, an example of a method for generating an output signal in a predetermined file format by multiplexing will be described. Hereinafter, an example in the case where the predetermined file format is ISOBMFF (ISO based media file format) will be described. Note that the file format is not limited to ISOBMFF, and other file formats may be used.

[0158] ISOBMFF is a file format standard defined in ISO / IEC 14496-12. ISOBMFF defines a format capable of multiplexing and storing data of various media such as video, audio, and text, and is a media-independent standard.

[0159] The storage method for each media in ISOBMFF is defined separately. For example, the storage methods for AVC video and HEVC video are defined in ISO / IEC 14496-15.

[0160] On the other hand, a method for storing the encoded data of data obtained from a plurality of sensor information (sensor signals) in ISOBMFF is required. FIG. 22 is a diagram showing a protocol for encoding a plurality of sensor information by various encoding methods and storing them in ISOBMFF.

[0161] Data1 to Data5 are sensor data (sensor signals) obtained from various types of sensors, respectively, and are, for example, RAW data or the like. Data1 and Data2 are converted into a 3D point cloud format and encoded using an encoding method Codec1 or Codec2 for the 3D point cloud format. Also, Data3 and Data4 are converted into a format of 2D data such as an image and encoded using an encoding method Codec3 or Codec4 for the 2D data format.

[0162] Each encoded data is converted into a NAL unit by a predetermined method and stored in ISOBMFF. Note that the NAL unit may be in a common format for 3DFormat and 2DFormat, or may be in different formats. Also, the NAL units of different encoding methods may be in a common format or may be in different formats. Note that the format of the sensor data may be a 1D format or other formats in addition to the 3D and 2D formats listed here.

[0163] Data5 is a case where the sensor data obtained from the sensor is directly stored in ISOBMFF without being encoded.

[0164] By providing a format for integrally storing any combination of these data, the management of the data of a system handling a plurality of sensors becomes easy, and various functions can be realized.

[0165] Next, the configuration of ISOBMFF will be described. The three-dimensional data multiplexing device stores a plurality of sensor data in ISOBMFF. FIG. 23 is a diagram showing a configuration example of input data to be multiplexed. FIG. 24 is a diagram showing a configuration example of a NAL unit. FIG. 25 is a diagram showing a configuration example of ISOBMFF. FIG. 26 is a diagram showing configuration examples of moov and mdat.

[0166] The encoded data included in the input data is mainly classified into encoded data (Data) and metadata (Meta). As metadata, there is metadata indicated by a header for each encoded data and metadata stored in an independent NAL unit as a parameter set. Also, metadata may be included in the encoded data. The three-dimensional data multiplexing device stores NAL units and RAW data for these plurality of different codecs in one ISOBMFF.

[0167] ISOBMFF is composed of a box structure. As boxes of ISOBMFF, there are mainly "moov" and "meta" for storing metadata, and "mdat" for storing data.

[0168] The encoded data and RAW data are stored in "mdat" in ISOBMFF in sample units. Also, the metadata in the input data is stored in a predetermined format in "trak" of "moov" in ISOMBFF for each encoded data. The metadata and synchronization information included in the encoded data are also stored in "moov".

[0169] Information for acquiring data from "mdat" (address information (offset information) of data from the beginning of the file and the size of the data, etc.) is stored in the metadata for each encoded data. Also, "ftyp" indicates the file type of subsequent data and the like.

[0170] Note that the format and box name may be other than those listed here, as long as they have the same function.

[0171] Also, in use cases such as real-time communication, units obtained by dividing boxes such as “moov” and “mdat” may be transmitted separately in time. Also, the data of the divided units may be interleaved.

[0172] The three-dimensional data multiplexing device defines a box indicating configuration information (hereinafter simply referred to as configuration information), and stores the identification information of a plurality of data included in the file in the configuration information. Also, the three-dimensional data multiplexing device stores, in the configuration information, identification information that can access the metadata of each data.

[0173] FIG. 27 is a diagram showing a configuration example of configuration information. FIG. 28 is a diagram showing a syntax example of configuration information.

[0174] The configuration information indicates information on contents and components constituting the ISOBMFF file, sensor information when acquiring the original data of the components, format information, and an encoding method.

[0175] As shown in FIG. 27, the configuration information includes overall configuration information and configuration information for each encoded data. The plurality of configuration information may have the same data structure or box, or may have different data structures or boxes.

[0176] In the type in the mp4 box, it is indicated by a 4CC such as “msuc” that it is a configuration information box. The overall configuration information (data()) indicates the configurations of a plurality of data with different encoding methods. data() includes num_of_data, data_type, and data_configuration.

[0177] num_of_data indicates the number of encoded data and RAW data constituting the file. data_type indicates identification information for each data. That is, data_type indicates the types of a plurality of data.

[0178] Specifically, data_type indicates whether the data is point cloud data or sensor signals (e.g., RAW data). Also, data_type may indicate whether the data is encoded or not. Further, data_type may indicate the encoding method (encoding format) used for encoding the encoded data. The encoding method is, for example, GPPC or VPPC, etc. Also, the encoding method may be Codec1 - 4 shown in FIG. 22. Additionally, data_type may indicate information for identifying configuration information.

[0179] Moreover, data_type may indicate the type of the original data of the point cloud data. The type of the original data refers to, when the original data is a sensor signal, the type of the sensor that generated the sensor signal (e.g., whether it is a 2D sensor or a 3D sensor, etc.). Also, data_type may include information indicating the data format of the sensor signal (e.g., whether it is 1D information, 2D information, 3D information, etc.).

[0180] For example, data_type = 0 indicates PCC Codec1, data_type = 1 indicates PCC Codec2, data_type = 2 indicates Video Codec3, and data_type = 4 indicates 3D axis sensor RAW data. data_configuration indicates the configuration information for each data.

[0181] data_configuration() is the configuration information for each encoded data and includes num_of_component, component_type, and component_id.

[0182] num_of_component indicates the number of components in the encoded data. component_type indicates the type of the component. For example, in the case of PCC encoding, component_type indicates whether the component is geometry, attribute, or metadata.

[0183] The component_id indicates a unique identifier for associating a component with other metadata and data.

[0184] Note that the encoding method may be an encoding method used for audio, text, applications, 360-degree images, etc., in addition to video codecs and PCC codecs. Also, the data may be processed data such as mesh or CAD. Also, the encoding method may be the same codec, or different encoding methods of profiles, levels, or tools, and any encoding method can be handled integrally.

[0185] In this way, by multiplexing the data necessary for utilizing the point cloud data decoded in the three-dimensional data demultiplexing device in an application into one file, file management and synchronization management handled by the application can be facilitated.

[0186] FIG. 29 is a diagram showing a configuration example of a data box "mdat". Each encoded data or RAW data is individually stored in a sample which is the minimum unit of the data box.

[0187] Also, synchronization information such as a timestamp for each encoded data included in the file is set based on overall synchronization information such as a common reference time. Also, the synchronization information is information that is synchronized respectively.

[0188] Also, for example, the reference time, time resolution, and time interval in the timestamps of a plurality of encoded data may be aligned, and the synchronization information may be made common among the plurality of encoded data. In that case, the synchronization information may be stored in any one or more of the synchronization information for each encoded data and the common synchronization information. In that case, the metadata includes at least one of information indicating the location where the common time information is stored and information indicating that the synchronization information is common.

[0189] Also, when synchronization is achieved among the encoded data, the three-dimensional data multiplexing device may store the encoded data that are synchronized as one sample. On the other hand, when at least one of the reference time, time resolution, and time interval is not aligned among the plurality of encoded data, the three-dimensional data multiplexing device may separately derive difference information indicating the difference in timestamps between the encoded data and store the derived difference information in the output signal. Further, the three-dimensional data multiplexing device may store a flag indicating whether synchronization is achieved in the output signal.

[0190] The three-dimensional data demultiplexing device synchronizes the encoded data by processing each sample at the time indicated by the timestamp shown in the metadata using the synchronization information of each encoded data and the overall synchronization information.

[0191] Hereinafter, an example of application processing will be described. FIG. 30 is a flowchart showing an example of application processing. When the application operation is started, the three-dimensional data demultiplexing device acquires an ISOBMFF file including point cloud data and a plurality of encoded data (S7301). For example, the three-dimensional data demultiplexing device may acquire the ISOBMFF file by communication or read it from the stored data.

[0192] Next, the three-dimensional data demultiplexing device analyzes the overall configuration information in the ISOBMFF file and specifies the data to be used in the application (S7302). For example, the three-dimensional data demultiplexing device acquires the data to be used in the processing and does not acquire the data not to be used in the processing.

[0193] Next, the three-dimensional data demultiplexing device extracts one or more data to be used in the application and analyzes the configuration information of the data (S7303).

[0194] When the data type is encoded data (encoded data in S7304), the 3D data demultiplexing device converts the ISOBMFF into an encoded stream and extracts the time stamp (S7305). Also, the 3D data demultiplexing device determines whether the synchronization between the data is complete, for example, by referring to a flag indicating whether the synchronization between the data is complete, and may perform a synchronization process if it is not complete.

[0195] Next, the 3D data demultiplexing device decodes the data in a predetermined method according to the time stamp and other instructions, and processes the decoded data (S7306).

[0196] On the other hand, when the data type is encoded data (RAW data in S7304), the 3D data demultiplexing device extracts the data and the time stamp (S7307). Also, the 3D data demultiplexing device determines whether the synchronization between the data is complete, for example, by referring to a flag indicating whether the synchronization between the data is complete, and may perform a synchronization process if it is not complete. Next, the 3D data demultiplexing device processes the data according to the time stamp and other instructions (S7308).

[0197] For example, an example in which sensor signals acquired by beam LiDAR, FLASH LiDAR, and a camera are encoded and multiplexed in different encoding methods will be described. FIG. 31 is a diagram showing an example of the sensor ranges of beam LiDAR, FLASH LiDAR, and a camera. For example, beam LiDAR detects all directions around a vehicle (sensor), and FLASH LiDAR and a camera detect the range in one direction (for example, the front) of the vehicle.

[0198] In the case of an application that integrally handles LiDAR point clouds, the 3D data demultiplexing device extracts and decodes the encoded data of beam LiDAR and FLASH LiDAR with reference to the overall configuration information. Also, the 3D data demultiplexing device does not extract camera images.

[0199] The three-dimensional data demultiplexing device processes the respective encoded data at the time of the same timestamp according to the timestamps of LiDAR and FLASH LiDAR.

[0200] For example, the three-dimensional data demultiplexing device may present the processed data on a presentation device, synthesize the point cloud data of beam LiDAR and FLASH LiDAR, or perform processing such as rendering.

[0201] Also, in the case of an application for calibration between data, the three-dimensional data demultiplexing device may extract sensor position information and use it in the application.

[0202] For example, in an application, the three-dimensional data demultiplexing device may select whether to use beam LiDAR information or FLASH LiDAR and switch the processing according to the selection result.

[0203] In this way, since the acquisition and encoding processing of data can be adaptively changed according to the processing of the application, the processing amount and power consumption can be reduced.

[0204] Hereinafter, use cases in autonomous driving will be described. FIG. 32 is a diagram showing a configuration example of an autonomous driving system. This autonomous driving system includes a cloud server 7350 and an edge 7360 such as an in-vehicle device or a mobile device. The cloud server 7350 includes a demultiplexing unit 7351, decoding units 7352A, 7352B, and 7355, a point cloud data synthesis unit 7353, a large-scale data storage unit 7354, a comparison unit 7356, and an encoding unit 7357. The edge 7360 includes sensors 7361A and 7361B, point cloud data generation units 7362A and 7362B, a synchronization unit 7363, encoding units 7364A and 7364B, a multiplexing unit 7365, an updated data storage unit 7366, a demultiplexing unit 7367, a decoding unit 7368, a filter 7369, a self-position estimation unit 7370, and a driving control unit 7371.

[0205] In this system, the edge 7360 downloads large-scale data, which is large-scale point cloud map data stored in the cloud server 7350. The edge 7360 performs self-position estimation processing of the edge 7360 (vehicle or terminal) by matching the large-scale data with the sensor information obtained at the edge 7360. Also, the edge 7360 uploads the acquired sensor information to the cloud server 7350 to update the large-scale data to the latest map data.

[0206] Also, in various applications that handle point cloud data within the system, point cloud data with different encoding methods is handled.

[0207] The cloud server 7350 encodes and multiplexes the large-scale data. Specifically, the encoding unit 7357 performs encoding using a third encoding method suitable for encoding the large-scale point cloud. Also, the encoding unit 7357 multiplexes the encoded data. The large-scale data storage unit 7354 stores the data encoded and multiplexed by the encoding unit 7357.

[0208] The edge 7360 performs sensing. Specifically, the point cloud data generation unit 7362A generates first point cloud data (position information (geometry) and attribute information) using the sensing information acquired by the sensor 7361A. The point cloud data generation unit 7362B generates second point cloud data (position information and attribute information) using the sensing information acquired by the sensor 7361B. The generated first point cloud data and second point cloud data are used for self-position estimation of autonomous driving or vehicle control, or map updating. In each process, a part of the information of the first point cloud data and the second point cloud data may be used.

[0209] Edge 7360 performs self-position estimation. Specifically, Edge 7360 downloads large-scale data from cloud server 7350. Demultiplexing unit 7367 obtains encoded data by demultiplexing large-scale data in file format. Decoding unit 7368 obtains large-scale data, which is large-scale point cloud map data, by decoding the obtained encoded data.

[0210] Self-position estimation unit 7370 estimates the vehicle's self-position on the map by matching the obtained large-scale data with the first point cloud data and the second point cloud data generated by point cloud data generation units 7362A and 7362B. Further, driving control unit 7371 uses the matching result or the self-position estimation result for driving control.

[0211] Note that self-position estimation unit 7370 and driving control unit 7371 may extract specific information such as position information from the large-scale data and perform processing using the extracted information. Further, filter 7369 performs processing such as correction or decimation on the first point cloud data and the second point cloud data. Self-position estimation unit 7370 and driving control unit 7371 may use the first point cloud data and the second point cloud data after such processing is performed. Further, self-position estimation unit 7370 and driving control unit 7371 may use the sensor signals obtained by sensors 7361A and 7361B.

[0212] Synchronization unit 7363 performs time synchronization and position correction between data of a plurality of sensor signals or a plurality of point cloud data. Further, synchronization unit 7363 may correct the position information of the sensor signal or the point cloud data to match the large-scale data based on the position correction information between the large-scale data and the sensor data generated by the self-position estimation process.

[0213] Note that the synchronization and position correction may be performed not by Edge 7360 but by cloud server 7350. In this case, Edge 7360 may multiplex the synchronization information and the position information and transmit them to cloud server 7350.

[0214] Edge 7360 encodes and multiplexes sensor signals or point cloud data. Specifically, the sensor signals or point cloud data are encoded using a first encoding method or a second encoding method suitable for encoding each signal. For example, the encoding unit 7364A generates first encoded data by encoding first point cloud data using the first encoding method. The encoding unit 7364B generates second encoded data by encoding second point cloud data using the second encoding method.

[0215] The multiplexing unit 7365 generates a multiplexed signal by multiplexing the first encoded data, the second encoded data, and synchronization information. The updated data storage unit 7366 stores the generated multiplexed signal. Also, the updated data storage unit 7366 uploads the multiplexed signal to the cloud server 7350.

[0216] The cloud server 7350 synthesizes point cloud data. Specifically, the demultiplexing unit 7351 obtains the first encoded data and the second encoded data by demultiplexing the multiplexed signal uploaded to the cloud server 7350. The decoding unit 7352A obtains the first point cloud data (or sensor signal) by decoding the first encoded data. The decoding unit 7352B obtains the second point cloud data (or sensor signal) by decoding the second encoded data.

[0217] The point cloud data synthesis unit 7353 synthesizes the first point cloud data and the second point cloud data in a predetermined method. When synchronization information and position correction information are multiplexed in the multiplexed signal, the point cloud data synthesis unit 7353 may perform the synthesis using those information.

[0218] The decoding unit 7355 demultiplexes and decodes the large-scale data stored in the large-scale data storage unit 7354. The comparison unit 7356 compares the point cloud data generated based on the sensor signal obtained at the edge 7360 with the large-scale data that the cloud server 7350 has, and determines the point cloud data that needs to be updated. The comparison unit 7356 updates the point cloud data determined to need updating among the large-scale data with the point cloud data obtained from the edge 7360.

[0219] The encoding unit 7357 encodes and multiplexes the updated large-scale data, and stores the obtained data in the large-scale data storage unit 7354.

[0220] As described above, depending on the application or application to be used, the signals to be handled may be different, and the signals to be multiplexed or the encoding method may be different. Even in such a case, by multiplexing data of various encoding methods using the present embodiment, flexible decoding and application processing become possible. Also, even when the signal encoding method is different, by converting to a more suitable encoding method through demultiplexing, decoding, data conversion, encoding, and multiplexing processes, various applications and systems can be constructed, and flexible service provision becomes possible.

[0221] As described above, the three-dimensional data multiplexing device according to the present embodiment performs the processing shown in FIG. 33. The three-dimensional data multiplexing device generates an output signal of a predetermined file configuration (for example, ISOBMFF) by multiplexing a plurality of types of data including point cloud data (S7311). Next, the three-dimensional data multiplexing device stores information (for example, data_type) indicating the type of each of the plurality of data included in the output signal in the metadata (control information) in the file configuration (S7312).

[0222] According to this, the three-dimensional data multiplexing device stores, in the metadata in the file configuration, information indicating the type of each of the plurality of data included in the output signal. Thereby, in the three-dimensional data demultiplexing device that receives the output signal, the type of each data can be easily determined. Thus, the three-dimensional data multiplexing method can appropriately multiplex and transmit the point cloud data.

[0223] For example, the information indicating the type of each of the plurality of data indicates (1) the encoding method applied to the data, (2) the configuration of the data, (3) the type of sensor that generated the data, or (4) the data format.

[0224] For example, the metadata in the file configuration includes synchronization information for synchronizing the times of the plurality of data included in the output signal. According to this, in the three-dimensional data demultiplexing device that receives the output signal, the plurality of data can be synchronized.

[0225] For example, the synchronization information indicates the difference in timestamps between the plurality of data. According to this, the data amount of the output signal can be reduced.

[0226] For example, the three-dimensional data multiplexing device includes a processor and a memory, and the processor performs the above processing using the memory.

[0227] In addition, the three-dimensional data demultiplexing apparatus according to the present embodiment performs the process shown in FIG. 34. The three-dimensional data demultiplexing apparatus acquires information (e.g., data_type) indicating the type of each of a plurality of data included in an output signal having a predetermined file configuration (e.g., ISOBMFF) in which a plurality of types of data including point cloud data are multiplexed, from the metadata in the file configuration (S7321). The three-dimensional data demultiplexing apparatus acquires a plurality of data from the output signal using the information indicating the type of each of the plurality of data (S7322). For example, the three-dimensional data demultiplexing apparatus selectively acquires necessary data from the output signal using the information indicating the type of each of the plurality of data. According to this, the three-dimensional data demultiplexing apparatus can easily determine the type of each data.

[0228] For example, the information indicating the type of each of the plurality of data indicates (1) the encoding method applied to the data, (2) the configuration of the data, (3) the type of sensor that generated the data, or (4) the data format.

[0229] For example, the metadata in the file configuration includes synchronization information for synchronizing the times of a plurality of data included in the output signal. For example, the three-dimensional data demultiplexing apparatus synchronizes a plurality of data using the synchronization information.

[0230] For example, the synchronization information indicates the difference in timestamps between a plurality of data. For example, the synchronization information includes information indicating the timestamp of any one of the plurality of data, and the three-dimensional data demultiplexing apparatus restores the timestamps of the other data among the plurality of data by adding the difference indicated by the synchronization information to the timestamp of any one of the plurality of data. Thereby, the data amount of the output signal can be reduced.

[0231] For example, the three-dimensional data demultiplexing apparatus includes a processor and a memory, and the processor performs the above-described process using the memory.

[0232] As described above, the three-dimensional data multiplexing device, three-dimensional data demultiplexing device, three-dimensional data encoding device, three-dimensional data decoding device, etc. according to the embodiments of the present disclosure have been described. However, the present disclosure is not limited to these embodiments.

[0233] Also, each processing unit included in the three-dimensional data multiplexing device, three-dimensional data demultiplexing device, three-dimensional data encoding device, three-dimensional data decoding device, etc. according to the above embodiments is typically realized as an LSI which is an integrated circuit. These may be individually formed into one chip, or may be formed into one chip so as to include some or all of them.

[0234] Also, the integration into an integrated circuit is not limited to an LSI, and it may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) which can be programmed after LSI manufacturing, or a reconfigurable processor which can reconfigure the connection and setting of circuit cells inside the LSI may be used.

[0235] Also, in each of the above embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0236] Also, the present disclosure may be realized as a three-dimensional data multiplexing method, three-dimensional data demultiplexing method, three-dimensional data encoding method, three-dimensional data decoding method, etc. executed by a three-dimensional data multiplexing device, three-dimensional data demultiplexing device, three-dimensional data encoding device, three-dimensional data decoding device, etc.

[0237] In addition, the division of the functional blocks in the block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality, or some functions may be transferred to other functional blocks. Also, the functions of a plurality of functional blocks having similar functions may be processed by a single piece of hardware or software in parallel or in a time-sharing manner.

[0238] In addition, the order in which each step in the flowchart is executed is for the purpose of illustration in order to specifically describe the present disclosure, and may be an order other than the above. Also, some of the above steps may be executed simultaneously (in parallel) with other steps.

[0239] As described above, the three-dimensional data multiplexing device, three-dimensional data demultiplexing device, three-dimensional data encoding device, three-dimensional data decoding device, etc. according to one or more aspects have been described based on the embodiments. However, the present disclosure is not limited to these embodiments. As long as the gist of the present disclosure is not deviated from, various modifications conceived by those skilled in the art applied to these embodiments, or forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects.

Industrial Applicability

[0240] The present disclosure can be applied to a three-dimensional data multiplexing device, a three-dimensional data demultiplexing device, a three-dimensional data encoding device, and a three-dimensional data decoding device.

Explanation of Signs

[0241] 4601 Three-dimensional data encoding system 4602 Three-dimensional data decoding system 4603 Sensor terminal 4604 External connection part 4611 Point cloud data generation system 4612 Presentation part 4613 Encoding part 4614 Multiplexing part 4615 Input / output part 4616 Control part 4617 Sensor information acquisition unit 4618 Point cloud data generation unit 4621 Sensor information acquisition unit 4622 Input / output unit 4623 Demultiplexing unit 4624 Decoding unit 4625 Presentation unit 4626 User interface 4627 Control unit 4630 First encoding unit 4631 Position information encoding unit 4632 Attribute information encoding unit 4633 Additional information encoding unit 4634 Multiplexing unit 4640 First decoding unit 4641 Demultiplexing unit 4642 Position information decoding unit 4643 Attribute information decoding unit 4644 Additional information decoding unit 4650 Second encoding unit 4651 Additional information generation unit 4652 Position image generation unit 4653 Attribute image generation unit 4654 Video encoding unit 4655 Additional information encoding unit 4656 Multiplexing unit 4660 Second decoding unit 4661 Demultiplexing unit 4662 Video decoding unit 4663 Additional information decoding unit 4664 Position information generation unit 4665 Attribute information generation unit 7301, 7301A, 7301B, 7301C Sensing devices 7302, 7302A, 7302B, 7302C Sensor information input unit 7303, 7303A, 7303C Point cloud data generation unit 7304A, 7304B Sensing unit 7305 Merge unit 7310 First device 7311 and 7321 Sensing Units 7312 and 7322 Output I / F 7320 Second Device 7323 Input I / F 7324 Processing Unit 7325 Point Cloud Data Generation Unit 7326 Encoding Unit 7331A, 7331B, and 7331C Sensing Units 7332A, 7332B, and 7332C Sensor Information Input Units 7333A, 7333B, and Point Cloud Data Generation Units 7334A, 7334B, and Encoding Units 7335 Synchronization Unit 7336 Multiplexing Unit 7341A, 7341B, and 7341C Sensing Units 7342A, 7342B, and Input / Output Units 7343 Point Cloud Data Generation Unit 7344A, 7344B, and Encoding Units 7345 Synchronization Unit 7346 Multiplexing Unit 7350 Cloud Server 7351 Demultiplexing Unit 7352A, 7352B, and Decoding Units 7353 Point Cloud Data Synthesis Unit 7354 Large-Scale Data Storage Unit 7355 Decoding Unit 7356 Comparison Unit 7357 Encoding Unit 7360 Edge 7361A and 7361B Sensors 7362A, 7362B, and Point Cloud Data Generation Units 7363 Synchronization Unit 7364A, 7364B, and Encoding Units 7365 Multiplexing Unit 7366 Update Data Storage Unit 7367 Demultiplexing Unit 7368 Decoding Unit 7369 Filter 7370 Self-Position Estimation Unit 7371 Driving Control Unit

Claims

1. generating an output signal having a predetermined file structure including the multiplexed data and metadata by multiplexing a plurality of types of data including three-dimensional data; storing type information indicating a type of each of the plurality of types of data and information linking the plurality of types of data in the metadata; The type information includes position information, angle information, or time information of a sensor when at least one type of data among the plurality of types of data is generated. Three-dimensional data processing method.

2. The three-dimensional data and any one of the plurality of types of data are presented based on the metadata.

2. The three-dimensional data processing method according to claim 1.

3. One of the plurality of types of data is two-dimensional image data, and the two-dimensional image data is data acquired based on the three-dimensional data.

2. The three-dimensional data processing method according to claim 1.

4. The type information may further indicate (1) the encoding method applied to the data, (2) the structure of the data, (3) the type of sensor that generated the data, or (4) the data format.

2. The three-dimensional data processing method according to claim 1.

5. The metadata includes synchronization information for synchronizing the times of the plurality of types of data included in the output signal.

2. The three-dimensional data processing method according to claim 1.

6. The synchronization information indicates a difference in time stamps between the plurality of types of data.

6. The three-dimensional data processing method according to claim 5.

7. The type information further includes time information when the plurality of types of data were generated.

2. The three-dimensional data processing method according to claim 1.

8. obtaining type information indicating the types of each of the plurality of types of data and information linking the plurality of types of data, which are stored in the metadata, from an output signal having a predetermined file configuration including metadata and the plurality of types of data multiplexed with the plurality of types of data including three-dimensional data; Using the type information, the plurality of types of data are obtained from the output signal; The type information includes position information, angle information, or time information of a sensor when at least one type of data among the plurality of types of data is generated. Three-dimensional data processing method.

9. The three-dimensional data and any one of the plurality of types of data are presented based on the metadata.

9. The three-dimensional data processing method according to claim 8.

10. One of the plurality of types of data is two-dimensional image data, and the two-dimensional image data is data acquired based on the three-dimensional data.

9. The three-dimensional data processing method according to claim 8.

11. The type information may further indicate (1) the encoding method applied to the data, (2) the structure of the data, (3) the type of sensor that generated the data, or (4) the data format.

9. The three-dimensional data processing method according to claim 8.

12. The metadata includes synchronization information for synchronizing the times of the plurality of types of data included in the output signal.

9. The three-dimensional data processing method according to claim 8.

13. The synchronization information indicates a difference in time stamps between the plurality of types of data.

13. The three-dimensional data processing method according to claim 12.

14. The type information further includes time information when the plurality of types of data were generated.

9. The three-dimensional data processing method according to claim 8.

15. A processor; A memory. The processor uses the memory to: generating an output signal having a predetermined file structure including the multiplexed data and metadata by multiplexing a plurality of types of data including three-dimensional data; storing type information indicating a type of each of the plurality of types of data and information linking the plurality of types of data in the metadata; The type information includes position information, angle information, or time information of a sensor when at least one type of data among the plurality of types of data is generated. Three-dimensional data processing device.

16. A processor; A memory. The processor uses the memory to: obtaining type information indicating the types of each of the plurality of types of data and information linking the plurality of types of data, which are stored in the metadata, from an output signal having a predetermined file configuration including metadata and the plurality of types of data multiplexed with the plurality of types of data including three-dimensional data; Using the type information, the plurality of types of data are obtained from the output signal; The type information includes position information, angle information, or time information of a sensor when at least one type of data among the plurality of types of data is generated. Three-dimensional data processing device.

Images