Information processing device, information processing method, and program

Abstract

The present invention provides a three-dimensional data multiplexing method, a three-dimensional data demultiplexing method, a three-dimensional data multiplexing apparatus, and a three-dimensional data demultiplexing apparatus for appropriately multiplexing and transmitting point cloud data. [Solution] The three-dimensional data multiplexing device comprises a processor and a memory. The three-dimensional data multiplexing process performed by the processor involves using the memory to acquire a first sensor signal, generating point cloud data based on the first sensor signal, generating first data corresponding to the point cloud data, acquiring a second sensor signal, generating second data derived from the second sensor signal without generating point cloud data for the second sensor signal, and generating an output signal with a predetermined file structure by multiplexing the first data, the second data, and metadata (S7311). The metadata stores information indicating the type of each of the first data and the second data (S7312).

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional data multiplexing method, a three-dimensional data demultiplexing method, a three-dimensional data multiplexing apparatus, and a three-dimensional data demultiplexing apparatus.

Background Art

[0002] In a wide range of fields such as computer vision, map information, monitoring, infrastructure inspection, or video distribution for autonomous operation of automobiles or robots, the spread of devices or services utilizing three-dimensional data is expected in the future. Three-dimensional data is acquired by various methods such as a distance sensor such as a range finder, a stereo camera, or a combination of a plurality of monocular cameras.

[0003] As one method of representing three-dimensional data, there is a representation method called point cloud that represents the shape of a three-dimensional structure by a point group in a three-dimensional space. In a point cloud, the position and color of the point group are stored. Although the point cloud is expected to become mainstream as a method of representing three-dimensional data, the amount of data of the point group is very large. Therefore, in the accumulation or transmission of three-dimensional data, as in the case of two-dimensional moving images (for example, MPEG-4 AVC or HEVC standardized by MPEG), compression of the data amount by encoding is essential.

[0004] Also, regarding the compression of the point cloud, it is partially supported by a publicly available library (Point Cloud Library) that performs point cloud-related processing.

[0005] Also, a technique of searching for and displaying facilities located around a vehicle using three-dimensional map data is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] Furthermore, there is a need for methods to transmit point cloud data in multiplexed form.

[0008] The purpose of this disclosure is to provide a three-dimensional data multiplexing method, a three-dimensional data demultiplexing method, a three-dimensional data multiplexing device, or a three-dimensional data demultiplexing device that can appropriately multiplex and transmit point cloud data. [Means for solving the problem]

[0009] An information processing device according to one aspect of the present disclosure comprises a processor and a memory, wherein the processor uses the memory to acquire a first sensor signal, generate point cloud data based on the first sensor signal, generate first data corresponding to the point cloud data, acquire a second sensor signal, generate second data derived from the second sensor signal without generating point cloud data for the second sensor signal, generate an output signal with a predetermined file structure by multiplexing the first data, the second data, and metadata, and stores information indicating the type of each of the first data and the second data in the metadata.

[0010] An information processing method according to one aspect of the present disclosure is an information processing method using an information processing device comprising a processor and a memory, wherein the processor uses the memory to acquire a first sensor signal, generate point cloud data based on the first sensor signal, generate first data corresponding to the point cloud data, acquire a second sensor signal, generate second data derived from the second sensor signal without generating point cloud data for the second sensor signal, generate an output signal with a predetermined file structure by multiplexing the first data, the second data, and metadata, and store information indicating the type of each of the first data and the second data in the metadata. [Effects of the Invention]

[0011] This disclosure provides a three-dimensional data multiplexing method, a three-dimensional data demultiplexing method, a three-dimensional data multiplexing device, or a three-dimensional data demultiplexing device that can appropriately multiplex and transmit point cloud data. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 shows the configuration of a three-dimensional data encoding and decoding system according to an embodiment. [Figure 2] Figure 2 shows an example of the configuration of point cloud data according to the embodiment. [Figure 3] Figure 3 shows an example of the structure of a data file containing point cloud data information according to the embodiment. [Figure 4] Figure 4 is a diagram showing the types of point cloud data according to the embodiment. [Figure 5] Figure 5 shows the configuration of the first encoding unit according to the embodiment. [Figure 6] Figure 6 is a block diagram of the first encoding unit according to the embodiment. [Figure 7] Figure 7 shows the configuration of the first decoding unit according to the embodiment. [Figure 8] Figure 8 is a block diagram of the first decoding unit according to the embodiment. [Figure 9] Figure 9 shows the configuration of the second encoding unit according to the embodiment. [Figure 10] Figure 10 is a block diagram of the second encoding unit according to the embodiment. [Figure 11] Figure 11 shows the configuration of the second decoding unit according to the embodiment. [Figure 12] Figure 12 is a block diagram of the second decoding unit according to the embodiment. [Figure 13] Figure 13 is a diagram showing the protocol stack related to PCC encoded data according to the embodiment. [Figure 14]FIG. 14 is a diagram showing a configuration example of a point group data generation device according to an embodiment. [Figure 15] FIG. 15 is a diagram showing a configuration example of a point group data generation device according to an embodiment. [Figure 16] FIG. 16 is a diagram showing a configuration example of a point group data generation device according to an embodiment. [Figure 17] FIG. 17 is a diagram showing a configuration example of a point group data encoding system according to an embodiment. [Figure 18] FIG. 18 is a diagram showing a configuration example of a three-dimensional data multiplexing device according to an embodiment. [Figure 19] FIG. 19 is a diagram showing a specific example of a three-dimensional data multiplexing device according to an embodiment. [Figure 20] FIG. 20 is a diagram showing the sensor ranges of various sensors according to an embodiment. [Figure 21] FIG. 21 is a diagram showing another configuration example of a three-dimensional data multiplexing device according to an embodiment. [Figure 22] FIG. 22 is a diagram showing a protocol for storing a plurality of pieces of information in a file format according to an embodiment. [Figure 23] FIG. 23 is a diagram showing a configuration example of input data according to an embodiment. [Figure 24] FIG. 24 is a diagram showing a configuration example of a NAL unit according to an embodiment. [Figure 25] FIG. 25 is a diagram showing a configuration example of an ISOBMFF according to an embodiment. [Figure 26] FIG. 26 is a diagram showing a configuration example of moov and mdat according to an embodiment. [Figure 27] FIG. 27 is a diagram showing a configuration example of configuration information according to an embodiment. [Figure 28] FIG. 28 is a diagram showing a syntax example of configuration information according to an embodiment. [Figure 29] FIG. 29 is a diagram showing a configuration example of mdat according to an embodiment. [Figure 30]Figure 30 is a flowchart showing an example of application processing according to the embodiment. [Figure 31] Figure 31 is a diagram showing the sensor range of various sensors according to the embodiment. [Figure 32] Figure 32 shows an example of the configuration of an automated driving system according to an embodiment. [Figure 33] Figure 33 is a flowchart of the three-dimensional data multiplexing process according to the embodiment. [Figure 34] Figure 34 is a flowchart of the three-dimensional data demultiplexing process according to the embodiment. [Modes for carrying out the invention]

[0013] A three-dimensional data multiplexing method according to one aspect of the present disclosure generates an output signal with a predetermined file structure by multiplexing multiple types of data, including point cloud data, and stores information indicating the type of each of the multiple data included in the output signal in the metadata of the file structure.

[0014] According to this, the three-dimensional data multiplexing method stores information indicating the type of each of the multiple data included in the output signal in the metadata of the file structure. This allows the three-dimensional data demultiplexing device receiving the output signal to easily determine the type of each data. In this way, the three-dimensional data multiplexing method can appropriately multiplex and transmit point cloud data.

[0015] For example, the information may include (1) the encoding scheme applied to the data, (2) the structure of the data, (3) the type of sensor that generated the data, or (4) the data format.

[0016] For example, the metadata may include synchronization information for synchronizing the timing of the multiple data included in the output signal.

[0017] According to this, multiple data can be synchronized in a three-dimensional data demultiplexer that receives the output signal.

[0018] For example, the synchronization information may indicate the difference in timestamps between the multiple data points.

[0019] According to this, the amount of data in the output signal can be reduced.

[0020] A three-dimensional data demultiplexing method according to one aspect of the present disclosure involves obtaining information indicating the type of each of the multiple data included in the output signal, which is stored in the metadata of the file configuration, from an output signal of a predetermined file configuration in which multiple types of data including point cloud data are multiplexed, and using the information, obtaining the multiple data from the output signal.

[0021] According to this, the three-dimensional data demultiplexing method can easily determine the type of each data point.

[0022] For example, the information may include (1) the encoding scheme applied to the data, (2) the structure of the data, (3) the type of sensor that generated the data, or (4) the data format.

[0023] For example, the metadata may include synchronization information for synchronizing the timing of the multiple data included in the output signal.

[0024] According to this, the three-dimensional data demultiplexing method can synchronize multiple data sets.

[0025] For example, the synchronization information may indicate the difference in timestamps between the multiple data points.

[0026] According to this, the amount of data in the output signal can be reduced.

[0027] Furthermore, a three-dimensional data multiplexing device according to one aspect of the present disclosure comprises a processor and a memory, wherein the processor generates an output signal with a predetermined file structure by multiplexing multiple types of data, including point cloud data, using the memory, and stores information indicating the type of each of the multiple data included in the output signal in the metadata of the file structure.

[0028] According to this, the three-dimensional data multiplexer stores information indicating the type of each of the multiple data included in the output signal in the metadata of the file structure. This allows the three-dimensional data demultiplexer receiving the output signal to easily determine the type of each data. In this way, the three-dimensional data multiplexer can appropriately multiplex and transmit point cloud data.

[0029] Furthermore, a three-dimensional data demultiplexing device according to one aspect of the present disclosure comprises a processor and a memory, wherein the processor uses the memory to obtain information indicating the type of each of the multiple data included in the output signal, which is stored in metadata in the file configuration, from an output signal of a predetermined file configuration in which multiple types of data including point cloud data are multiplexed, and uses the information to obtain the multiple data from the output signal.

[0030] According to this, the three-dimensional data demultiplexing device can easily determine the type of each data.

[0031] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0032] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, components in the following embodiments that are not described in an independent claim will be described as optional components.

[0033] (Embodiment) First, the configuration of the three-dimensional data (point cloud data) encoding and decoding system according to this embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the three-dimensional data encoding and decoding system according to this embodiment. As shown in Figure 1, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.

[0034] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data, which is three-dimensional data. The three-dimensional data encoding system 4601 may be a three-dimensional data encoding device implemented by a single device, or it may be a system implemented by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 4601.

[0035] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a presentation unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.

[0036] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603 and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data from the sensor information and outputs the point cloud data to the encoding unit 4613.

[0037] The display unit 4612 presents sensor information or point cloud data to the user. For example, the display unit 4612 displays information or images based on sensor information or point cloud data.

[0038] The encoding unit 4613 encodes (compresses) the point cloud data and outputs the resulting encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, sensor information.

[0039] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data input from the encoding unit 4613, control information, and additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0040] The input / output unit 4615 (for example, the communication unit or interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as internal memory. The control unit 4616 (or application execution unit) controls each processing unit. In other words, the control unit 4616 performs control such as encoding and multiplexing.

[0041] The sensor information may also be input to the encoding unit 4613 or the multiplexing unit 4614. Furthermore, the input / output unit 4615 may output the point cloud data or encoded data directly to the outside.

[0042] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604.

[0043] The three-dimensional data decoding system 4602 generates point cloud data, which is three-dimensional data, by decoding encoded data or multiplexed data. The three-dimensional data decoding system 4602 may be a three-dimensional data decoding device implemented by a single device, or it may be a system implemented by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 4602.

[0044] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621, an input / output unit 4622, a demultiplexing unit 4623, a decoding unit 4624, a presentation unit 4625, a user interface 4626, and a control unit 4627.

[0045] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603.

[0046] The input / output unit 4622 acquires the transmission signal, decodes the multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexing unit 4623.

[0047] The demultiplexing unit 4623 acquires encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 4624.

[0048] The decoding unit 4624 reconstructs the point cloud data by decoding the encoded data.

[0049] The presentation unit 4625 presents point cloud data to the user. For example, the presentation unit 4625 displays information or images based on the point cloud data. The user interface 4626 acquires instructions based on user operations. The control unit 4627 (or application execution unit) controls each processing unit. In other words, the control unit 4627 performs control such as demultiplexing, decoding, and presentation.

[0050] The input / output unit 4622 may acquire point cloud data or encoded data directly from an external source. The presentation unit 4625 may acquire additional information such as sensor information and present information based on that additional information. The presentation unit 4625 may also make presentations based on user instructions acquired through the user interface 4626.

[0051] The sensor terminal 4603 generates sensor information, which is information obtained from the sensor. The sensor terminal 4603 is a terminal equipped with a sensor or camera, and may be, for example, a mobile object such as an automobile, an aerial object such as an airplane, a mobile terminal, or a camera.

[0052] The sensor information that can be acquired by the sensor terminal 4603 includes, for example, (1) the distance (position information), color, or reflectivity of the object between the sensor terminal 4603 and the object obtained from a LiDAR, millimeter-wave radar, or infrared sensor, and (2) the distance (position information), color, or reflectivity of the object between the camera and the object obtained from multiple monocular camera images or stereo camera images. The sensor information may also include the sensor's attitude, orientation, gyroscope (angular velocity), position (GPS information or altitude), speed, acceleration, or the time the sensor information was acquired. The sensor information may also include temperature, atmospheric pressure, humidity, or magnetism.

[0053] The external connection unit 4604 is implemented by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the internet, or broadcasting, etc.

[0054] Next, we will explain point cloud data. Figure 2 shows the structure of point cloud data. Figure 3 shows an example of the structure of a data file containing information about point cloud data.

[0055] Point cloud data contains data for multiple points. Each point's data includes location information (three-dimensional coordinates) and attribute information related to that location. A collection of these points is called a point cloud. For example, a point cloud represents the three-dimensional shape of an object.

[0056] Position information, such as three-dimensional coordinates, is sometimes referred to as geometry. Furthermore, the data for each point may include attribute information of multiple attribute types. Attribute types include, for example, color or reflectance.

[0057] One location information may be associated with one attribute information, or multiple attribute information of different attribute types may be associated with one location information. Furthermore, multiple attribute information of the same attribute type may be associated with one location information.

[0058] The example data file structure shown in Figure 3 represents a case where location information and attribute information correspond one-to-one, and it shows the location information and attribute information of the N points that make up the point cloud data.

[0059] Location information includes, for example, information for the three axes: x, y, and z. Attribute information includes, for example, RGB color information. A typical data file is a ply file.

[0060] Next, we will explain the types of point cloud data. Figure 4 is a diagram illustrating the types of point cloud data. As shown in Figure 4, point cloud data includes static objects and dynamic objects.

[0061] A static object is three-dimensional point cloud data for any given time (a specific moment). A dynamic object is three-dimensional point cloud data that changes over time. Hereafter, three-dimensional point cloud data for a given time will be referred to as a PCC frame, or simply a frame.

[0062] The object can be a point cloud with a somewhat limited area, like regular video data, or it can be a large-scale point cloud with no area limitations, like map information.

[0063] Furthermore, point cloud data of various densities may exist, including both sparse and dense point cloud data.

[0064] The details of each processing unit are described below. Sensor information is acquired by various methods, such as distance sensors like LIDAR or rangefinders, stereo cameras, or combinations of multiple monocular cameras. The point cloud data generation unit 4618 generates point cloud data based on the sensor information obtained by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as point cloud data and adds attribute information to the position information.

[0065] The point cloud data generation unit 4618 may process the point cloud data when generating position information or adding attribute information. For example, the point cloud data generation unit 4618 may reduce the amount of data by deleting point clouds with overlapping positions. The point cloud data generation unit 4618 may also transform the position information (such as position shifting, rotation, or normalization) or render the attribute information.

[0066] In Figure 1, the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but it may also be provided independently outside of the three-dimensional data encoding system 4601.

[0067] The encoding unit 4613 generates encoded data by encoding the point cloud data based on a predetermined encoding method. There are two main types of encoding methods. The first is an encoding method using positional information, which will be referred to as the first encoding method hereafter. The second is an encoding method using a video codec, which will be referred to as the second encoding method hereafter.

[0068] The decoding unit 4624 decodes the point cloud data by decoding the encoded data based on a predetermined encoding method.

[0069] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to PCC encoded data, the multiplexing unit 4614 multiplexes other media such as video, audio, subtitles, applications, files, or reference time information. Furthermore, the multiplexing unit 4614 may also multiplex attribute information related to sensor information or point cloud data.

[0070] Multiplexing methods or file formats include ISOBMFF, ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RMP.

[0071] The demultiplexing unit 4623 extracts PCC encoded data, other media, and time information from the multiplexed data.

[0072] The input / output unit 4615 transmits the multiplexed data using a method appropriate to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 4615 may communicate with other devices via the internet or with storage units such as cloud servers.

[0073] Communication protocols such as HTTP, FTP, TCP, UDP, or IP can be used. Either a pull-type or push-type communication method may be employed.

[0074] Either wired or wireless transmission may be used. Wired transmission methods include Ethernet®, USB, RS-232C, HDMI®, or coaxial cable. Wireless transmission methods include 3G / 4G / 5G as defined by IEEE under 3GPP®, wireless LAN, Wi-Fi®, Bluetooth®, or millimeter wave.

[0075] Furthermore, broadcasting formats such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0076] Figure 5 shows the configuration of a first encoding unit 4630, which is an example of an encoding unit 4613 that performs encoding using the first encoding method. Figure 6 is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit 4630 includes a location information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0077] The first encoding unit 4630 is characterized by performing encoding while being aware of the three-dimensional structure. Furthermore, the first encoding unit 4630 is characterized by the attribute information encoding unit 4632 performing encoding using information obtained from the location information encoding unit 4631. The first encoding method is also called GPCC (Geometry-based PCC).

[0078] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (metadata). The position information is input to the position information encoding unit 4631, the attribute information is input to the attribute information encoding unit 4632, and the additional information is input to the additional information encoding unit 4633.

[0079] The location information encoding unit 4631 generates encoded location information (Compressed Geometry), which is encoded data, by encoding location information. For example, the location information encoding unit 4631 encodes location information using an N-tree structure such as an octree. Specifically, in an octree, the target space is divided into 8 nodes (subspaces), and 8 bits of information (occupancy code) are generated to indicate whether or not a point cloud is contained in each node. Furthermore, nodes containing point clouds are further divided into 8 nodes, and 8 bits of information are generated to indicate whether or not a point cloud is contained in each of these 8 nodes. This process is repeated until the number of point clouds contained in a predetermined hierarchy or node falls below a threshold.

[0080] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute), which is encoded data, by encoding it using the configuration information generated by the location information encoding unit 4631. For example, the attribute information encoding unit 4632 determines the reference point (reference node) to be referenced in encoding the target point (target node) to be processed, based on the octave tree structure generated by the location information encoding unit 4631. For example, the attribute information encoding unit 4632 references a surrounding node or adjacent node whose parent node in the octave tree is the same as the target node. Note that the method for determining the reference relationship is not limited to this.

[0081] Furthermore, the attribute information encoding process may include at least one of the following: quantization, prediction, and arithmetic encoding. In this case, a reference means using a reference node to calculate the predicted value of the attribute information, or using the state of a reference node (for example, occupancy information indicating whether or not the reference node contains a point cloud) to determine the encoding parameters. For example, encoding parameters may be quantization parameters in the quantization process, or context in arithmetic encoding.

[0082] The additional information encoding unit 4633 generates encoded data, or compressed additional information (Compressed MetaData), by encoding the compressible data from the additional information.

[0083] The multiplexing unit 4634 generates a compressed stream, which is encoded data, by multiplexing encoded position information, encoded attribute information, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of the system layer (not shown).

[0084] Next, we will describe a first decoding unit 4640, which is an example of a decoding unit 4624 that performs decoding of the first encoding method. Figure 7 is a diagram showing the configuration of the first decoding unit 4640. Figure 8 is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point cloud data by decoding the encoded data (encoded stream) encoded by the first encoding method using the first encoding method. This first decoding unit 4640 includes a demultiplexing unit 4641, a location information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0085] A compressed stream, which is encoded data, is input to the first decoding unit 4640 from a processing unit of the system layer (not shown).

[0086] The demultiplexing unit 4641 separates encoded location information (Compressed Geometry), encoded attribute information (Compressed Attribute), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0087] The location information decoding unit 4642 generates location information by decoding the encoded location information. For example, the location information decoding unit 4642 reconstructs the location information of a point cloud represented by three-dimensional coordinates from encoded location information represented by an N-tree structure such as an octree.

[0088] The attribute information decoding unit 4643 decodes the encoded attribute information based on the configuration information generated by the location information decoding unit 4642. For example, the attribute information decoding unit 4643 determines the reference point (reference node) to be referenced in the decoding of the target point (target node) to be processed, based on the octave tree structure obtained by the location information decoding unit 4642. For example, the attribute information decoding unit 4643 references a surrounding node or adjacent node whose parent node in the octave tree is the same as the target node. Note that the method for determining the reference relationship is not limited to this.

[0089] Furthermore, the attribute information decoding process may include at least one of the following: inverse quantization, prediction, and arithmetic decoding. In this case, "reference" means using a reference node to calculate the predicted value of the attribute information, or using the state of the reference node (for example, occupancy information indicating whether or not the reference node contains a point cloud) to determine the decoding parameters. For example, decoding parameters may be quantization parameters in the inverse quantization process, or context in arithmetic decoding.

[0090] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. The first decoding unit 4640 uses the additional information necessary for decoding location information and attribute information during decoding and outputs the additional information necessary for the application to the outside.

[0091] Next, we will describe a second encoding unit 4650, which is an example of an encoding unit 4613 that performs encoding using the second encoding method. Figure 9 is a diagram showing the configuration of the second encoding unit 4650. Figure 10 is a block diagram of the second encoding unit 4650.

[0092] The second encoding unit 4650 generates encoded data (encoded stream) by encoding the point cloud data using a second encoding method. This second encoding unit 4650 includes an additional information generation unit 4651, a position image generation unit 4652, an attribute image generation unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0093] The second encoding unit 4650 generates a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and then encodes the generated position image and attribute image using an existing video encoding scheme. The second encoding method is also called VPCC (Video based PCC).

[0094] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (metadata).

[0095] The additional information generation unit 4651 generates map information for multiple two-dimensional images by projecting a three-dimensional structure onto a two-dimensional image.

[0096] The position image generation unit 4652 generates a position image (geometry image) based on position information and map information generated by the additional information generation unit 4651. This position image is, for example, a depth image in which the distance is indicated as a pixel value. This depth image may be an image of multiple point clouds viewed from one viewpoint (an image of multiple point clouds projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image formed by integrating these multiple images.

[0097] The attribute image generation unit 4653 generates an attribute image based on attribute information and map information generated by the additional information generation unit 4651. This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is shown as pixel values. This image may be an image of multiple point clouds viewed from one viewpoint (an image of multiple point clouds projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image formed by integrating these multiple images.

[0098] The video encoding unit 4654 generates encoded data, namely a compressed geometric image and a compressed attribute image, by encoding the position image and attribute image using a video encoding scheme. Any known encoding scheme may be used as the video encoding scheme. For example, the video encoding scheme may be AVC or HEVC.

[0099] The additional information encoding unit 4655 generates encoded additional information (Compressed MetaData) by encoding additional information and map information included in the point cloud data.

[0100] The multiplexing unit 4656 generates a compressed stream, which is encoded data, by multiplexing the encoded position image, encoded attribute image, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of the system layer (not shown).

[0101] Next, we will describe a second decoding unit 4660, which is an example of a decoding unit 4624 that performs decoding of the second encoding method. Figure 11 is a diagram showing the configuration of the second decoding unit 4660. Figure 12 is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point cloud data by decoding the encoded data (encoded stream) encoded by the second encoding method using the second encoding method. This second decoding unit 4660 includes a demultiplexing unit 4661, a video decoding unit 4662, an additional information decoding unit 4663, a location information generation unit 4664, and an attribute information generation unit 4665.

[0102] A compressed stream, which is encoded data, is input to the second decoding unit 4660 from a processing unit of the system layer (not shown).

[0103] The demultiplexing unit 4661 separates the encoded location image (Compressed Geometry Image), encoded attribute image (Compressed Attribute Image), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0104] The video decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding scheme. Any known encoding scheme may be used as the video encoding scheme. For example, the video encoding scheme may be AVC or HEVC.

[0105] The additional information decoding unit 4663 generates additional information, including map information, by decoding the encoded additional information.

[0106] The location information generation unit 4664 generates location information using the location image and map information. The attribute information generation unit 4665 generates attribute information using the attribute image and map information.

[0107] The second decoding unit 4660 uses the additional information necessary for decoding during the decoding process and outputs the additional information necessary for the application to the outside.

[0108] The PCC encoding method will be explained below. Figure 13 is a diagram showing the protocol stack involved in PCC encoded data. Figure 13 shows an example in which data from other media such as video (e.g., HEVC) or audio, or sensor information, is multiplexed with PCC encoded data and transmitted or stored.

[0109] Multiplexing schemes and file formats have the function of multiplexing, transmitting, or storing various encoded data. To transmit or store encoded data, it is converted into the format of the multiplexing scheme. For example, HEVC specifies a technique for storing encoded data in a data structure called a NAL unit, and then storing the NAL unit in ISOBMFF.

[0110] A similar configuration is expected in PCC. Sensor information may be PCC encoded along with point cloud data, encoded using a different encoding method, stored directly in multiple layers without encoding, or a combination of these methods. Specifically, a different encoding method may be a different three-dimensional encoding method, or an encoding method that encodes data converted from point cloud data to two-dimensional or one-dimensional data.

[0111] The following describes an example of a configuration for generating point cloud data from sensor signals (also called sensor information). Figures 14 to 16 show example configurations of point cloud data generation devices that generate point cloud data from sensor signals.

[0112] The point cloud data generation device shown in Figure 14 generates point cloud data from sensor signals obtained from a single sensing device 7301. The point cloud data generation device shown in Figure 14 comprises a sensing device 7301, a sensor information input unit 7302, and a point cloud data generation unit 7303. The sensor information input unit 7302 acquires sensor signals obtained from the sensing device 7301. The point cloud data generation unit 7303 generates point cloud data from the sensor signals acquired by the sensor information input unit 7302. The generated point cloud data is output to, for example, a subsequent point cloud data encoding unit (not shown).

[0113] As shown in Figure 15, point cloud data may be generated based on sensor signals obtained from two or more sensing devices. The point cloud data generation device shown in Figure 15 comprises sensing devices 7301A and 7301B, sensor information input units 7302A and 7302B, and a point cloud data generation unit 7303A. Sensor information input unit 7302A acquires a first sensor signal obtained from sensing device 7301A. Sensor information input unit 7302B acquires a second sensor signal obtained from sensing device 7301B. Point cloud data generation unit 7303A generates point cloud data from the two sensor signals acquired by sensor information input units 7302A and 7302B. The generated point cloud data is output to, for example, a subsequent point cloud data encoding unit (not shown).

[0114] The point cloud data generation device shown in Figure 16 comprises a sensing device 7301C, a sensor information input unit 7302C, and a point cloud data generation unit 7303C. The sensing device 7301C generates a sensor signal by merging two pieces of information sensed using two or more sensing methods in a predetermined manner. The sensing device 7301C comprises sensing units 7304A and 7304B and a merging unit 7305.

[0115] Sensing unit 7304A generates a first sensor signal using a first sensing method. Sensing unit 7304B generates a second sensor signal using a second sensing method. Merging unit 7305 merges the first sensor signal and the second sensor signal and outputs the generated sensor signal to sensor information input unit 7302C.

[0116] The merging unit 7305 may select one of the first sensor signal and the second sensor signal based on predetermined conditions and output the selected sensor signal. Furthermore, when merging two sensor signals, the merging unit 7305 may change the weight coefficient used for merging.

[0117] For example, the merging unit 7305 may determine which sensor signal to select based on the acquired sensor signal, or it may determine based on a different sensor signal.

[0118] For example, the first sensing method and the second sensing method may differ in sensor parameters, sensing frequency, or sensing mechanism. Furthermore, the sensor signal may include information indicating the sensing method or parameters used during sensing.

[0119] When the merging unit 7305 switches between multiple sensing methods, it may include information indicating which sensing method was used, or data indicating the criteria for switching, in the sensor signal. When the merging unit 7305 merges sensor signals, it may include information for identifying the merged sensing method, data indicating the criteria for merging, or a merging coefficient in the sensor signal.

[0120] Furthermore, the sensing device 7301C may output multiple sensor signals. In addition, the sensing device 7301C may output, as multiple sensor signals, the absolute value of the first sensor signal and the difference between the first sensor signal and the second sensor signal.

[0121] Furthermore, the sensor signal may include information indicating the relationship between the first sensing method and the second sensing method. For example, the sensor signal may include absolute or relative values ​​of reference position information for the first sensing method and the second sensing method, or it may include information indicating the acquisition time of the sensor signal, reference time information, or the angle of the sensor. By including this information in the sensor signal, it becomes possible to correct or combine the relationship between the two sensor signals based on this information in subsequent processing.

[0122] The sensor information input unit 7302C acquires the sensor signal obtained by the sensing device 7301C. The point cloud data generation unit 7303C generates point cloud data from the sensor signal acquired by the sensor information input unit 7302C. The generated point cloud data is output to, for example, a subsequent point cloud data encoding unit (not shown).

[0123] Thus, the point cloud data generation device generates point cloud data based on one or more of the various sensor signals described above. The point cloud data generation device may also correct the positional information or attribute information of the points during the point cloud data generation process.

[0124] The point cloud data generation device may have any of the configurations shown in Figures 14 to 16, or a combination of several of these configurations. Furthermore, the point cloud data generation device may use a fixed method, or it may adaptively change the method used depending on the sensing purpose or use case, for example.

[0125] Next, an example of the configuration of the point cloud data encoding system according to this embodiment will be described. Figure 17 is a diagram showing an example of the configuration of the point cloud data encoding system according to this embodiment. The point cloud data encoding system shown in Figure 17 includes a first device 7310 and a second device 7320.

[0126] The first device 7310 includes a sensing unit 7311 and an output interface 7312. The second device 7320 includes a sensing unit 7321, an output interface 7322, an input interface 7323, and a processing unit 7324. The processing unit 7324 includes a point cloud data generation unit 7325 and an encoding unit 7326.

[0127] The sensing unit 7311 or 7321 may be included in the same hardware or device as the processing unit 7324, which is composed of a CPU or the like, or it may be included in different hardware or devices.

[0128] The sensing unit 7321 is included in the same device (second device 7320) as the processing unit 7324. In this case, the output signal of the sensing unit 7321 (referred to as RAW data) is directly input to the point cloud data generation unit 7325.

[0129] The sensing unit 7311 is included in a different device (first device 7310) from the processing unit 7324. In this case, the RAW data output from the sensing unit 7311 is converted to an input / output format (external output format) by the output I / F 7312, and the formatted signal is input to the second device 7320. The input I / F 7323 included in the second device 7320 converts the formatted signal back into RAW data and outputs the obtained RAW data to the point cloud data generation unit 7325. The output I / F 7312 and input I / F 7323 have the functions of, for example, the multiplexing unit 4614 and input / output unit 4615 shown in Figure 1.

[0130] Alternatively, the output signal (RAW data) from the sensing unit 7321, which is included in the same device as the processing unit 7324, may be converted to an input / output format by the output I / F 7322, the formatted signal may be converted back to RAW data by the input I / F 7323, and the resulting RAW data may be input to the point cloud data generation unit 7325.

[0131] Furthermore, when multiple sensor signals are input, for example, when sensor signals from other devices and sensor signals from the same device are mixed, these sensor signals may be converted to the same format. Also, during the conversion, each signal may be assigned an identifier that can identify the signal. For example, when transmission is performed using UDP (User Datagram Protocol), each signal may be identified by the IP (Internet Protocol) or UDP source address or source port number. This makes it possible to unify the format input to the point cloud data generation unit 7325, thus simplifying signal control.

[0132] The point cloud data generation unit 7325 generates point cloud data using the input RAW data. The encoding unit 7326 encodes the generated point cloud data.

[0133] Next, an example of the configuration of a three-dimensional data multiplexing device (three-dimensional data multiplexing system) according to this embodiment will be described. Figure 18 is a diagram showing an example of the configuration of a three-dimensional data multiplexing device according to this embodiment. The three-dimensional data multiplexing device generates output signals by encoding and multiplexing various sensor signals, and stores or transmits the generated output signals.

[0134] As shown in Figure 18, the three-dimensional data multiplexing device comprises sensing units 7331A, 7331B, and 7331C, sensor information input units 7332A, 7332B, and 7332C, point cloud data generation units 7333A and 7333B, encoding units 7334A and 7334B, synchronization unit 7335, and multiplexing unit 7336. While this example shows the use of three sensing units, the number of sensing units is not limited to this. Furthermore, any combination of the following processing methods can be used for processing sensor signals from each sensing unit.

[0135] The sensor information input unit 7332A acquires the first sensor signal generated by sensing in the sensing unit 7331A. The sensor information input unit 7332B acquires the second sensor signal generated by sensing in the sensing unit 7331B. The sensor information input unit 7332C acquires the third sensor signal generated by sensing in the sensing unit 7331C.

[0136] The point cloud data generation unit 7333A generates first point cloud data from the first sensor signal. The point cloud data generation unit 7333B generates second point cloud data from the second sensor signal. At this time, the number of points, the range of points, and attribute information in the generated point cloud data may differ depending on the difference in sensing methods used by the sensing unit 7331A and the sensing unit 7331B (for example, direction, range, obtainable attributes, frequency, resolution, etc., method or means).

[0137] The encoding unit 7334A generates first encoded data by encoding the first point cloud data. The encoding unit 7334B generates second encoded data by encoding the second point cloud data. For example, the encoding unit 7334A and the encoding unit 7334B may apply different encoding methods to each other. For example, the encoding unit 7334A may use a first encoding method, and the encoding unit 7334B may use a second encoding method different from the first encoding method. Alternatively, the encoding unit 7334A and the encoding unit 7334B may use the same encoding method.

[0138] The encoding units 7334A and 7334B may compress the point location information or attribute information in the point cloud data using entropy coding or the like. Furthermore, the encoding units 7334A and 7334B may store sensor signals, sensor location information or angle information, or time information as metadata.

[0139] The encoding units 7334A and 7334B use encoding schemes suitable for point cloud data. For example, the first encoding scheme is one that can be expected to achieve a high encoding rate for map information or static content, and the second encoding scheme is one that can be expected to achieve a high encoding rate for content such as AR or VR. In this case, the encoding units 7334A and 7334B may use encoding schemes suitable for the content.

[0140] Alternatively, for example, the first encoding scheme is one that can be expected to achieve a high encoding rate for point clouds based on information sensed by a sensing unit such as a beam LiDAR, and the second encoding scheme is one that can be expected to achieve a high encoding rate for point clouds based on information sensed by a sensing unit such as a flash LiDAR. In this case, the encoding units 7334A and 7334B may use encoding schemes suitable for the sensing unit.

[0141] Furthermore, the encoding units 7334A and 7334B may use encoding tools or encoding-related parameters suitable for the content or sensing unit within the same encoding scheme, rather than changing the encoding scheme.

[0142] The generated first encoded data and second encoded data are input to the multiplexing unit 7336. For example, the third sensor signal sensed by the sensing unit 7331C is data that does not need to be encoded. In this case, point cloud data is not generated or encoded, and the third sensor signal is input directly to the multiplexing unit 7336. Note that encoding may not be performed for the purpose of low-latency transmission.

[0143] The synchronization unit 7335 has a function for synchronizing multiple sensing units. For example, the synchronization unit 7335 uses sensing time information, timestamp information, and angle information as information related to synchronization. This information related to synchronization may be multiplexed into the output signal as a synchronization signal, which is common information. Alternatively, this information related to synchronization may be included in each sensor signal.

[0144] The multiplexing unit 7336 generates an output signal by multiplexing one or more encoded data, metadata, raw data of sensor signals, and a synchronization signal. The multiplexing unit 7336 also stores information for identifying each data and information indicating the correspondence between each data in the output signal.

[0145] Figure 19 shows a specific example of a three-dimensional data multiplexing device. As shown in Figure 19, a beam LiDAR is used as the sensing unit 7331A, and a flash LiDAR is used as the sensing unit 7331B. Depending on the characteristics of the LiDAR, the range and distance of the point cloud, as well as the resolution, will differ. Figure 20 shows examples of the sensor ranges of beam LiDAR and flash LiDAR. For example, beam LiDAR detects in all directions around the vehicle (sensor), while flash LiDAR detects in one direction of the vehicle (e.g., forward).

[0146] The point cloud data generation unit 7333A generates first point cloud data based on distance information and reflectance information with respect to the beam irradiation angle acquired from the beam LiDAR. The point cloud data generation unit 7333B generates second point cloud data based on two-dimensional distance information and reflectance obtained from the FLASH LiDAR. In addition, the point cloud data generation units 7333A and 7333B may also use two-dimensional color information acquired from a camera to generate point cloud data that has both color information and reflectance.

[0147] Furthermore, the sensing unit 7331C may be an in-vehicle sensor such as a 3-axis gyro sensor, a 3-axis accelerometer, or a position information sensor such as GPS. This sensor information represents the overall state of the vehicle and can also be described as common information related to the first and second sensor signals. This common sensor information may be encoded and multiplexed, or it may be multiplexed without encoding. This information may also be stored and encoded in the first or second sensor signal as common additional information to the point cloud data. Alternatively, the common sensor information may be stored in one of the sensor signals, either the first or the second. In this case, information indicating which sensor signal the common sensor information is stored in may be indicated, for example, in another sensor signal or synchronization signal.

[0148] Furthermore, as information regarding the time of sensor acquisition, timestamps based on reference time information such as NTP (Network Time Protocol) or PTP (Precision Time Protocol) are added to the first point cloud data based on beam LiDAR and the second point cloud data based on FLASH LiDAR. The timestamps of each sensor are synchronized to a common reference time and are encoded by encoding units 7334A and 7334B.

[0149] Furthermore, reference time information indicating a common reference time may be multiplexed as a synchronization signal. Reference time information does not necessarily have to be multiplexed. A three-dimensional data demultiplexer (three-dimensional data decoder) obtains the timestamps from the encoded data of multiple sensor signals. Since the timestamps are synchronized to a common reference time, the three-dimensional data demultiplexer can synchronize multiple sensors by operating the decoded data of multiple sensor signals based on their respective timestamps.

[0150] Furthermore, time information may be set separately for each beam LiDAR and FLASH LiDAR. Also, each beam LiDAR and FLASH LiDAR may be equipped with a 3-axis sensor. In that case, a common time such as Internet Time will be used as the NTP for each. Additionally, multiple 3-axis sensors that have been pre-calibrated and pre-synchronized may be used.

[0151] Figure 21 shows another example of the configuration of a three-dimensional data multiplexing device. As shown in Figure 21, the three-dimensional data multiplexing device comprises sensing units 7341A, 7341B, and 7341C, input / output units 7342A, 7342B, and 7342C, a point cloud data generation unit 7343, encoding units 7344A and 7344B, a synchronization unit 7345, and a multiplexing unit 7346.

[0152] The input / output unit 7342A acquires the first sensor signal generated by sensing in the sensing unit 7341A. The input / output unit 7342B acquires the second sensor signal generated by sensing in the sensing unit 7341B. The input / output unit 7342C acquires the third sensor signal generated by sensing in the sensing unit 7341C. The input / output units 7342A, 7342B, and 7342C may have a memory for storing the acquired sensor signals.

[0153] The point cloud data generation unit 7343 generates first point cloud data from the first sensor signal. The encoding unit 7344A generates first encoded data by encoding the first point cloud data. The encoding unit 7344B generates second encoded data by encoding second sensor information.

[0154] The synchronization unit 7345 has the function of synchronizing multiple sensing units. The multiplexing unit 7346 generates an output signal by multiplexing one or more encoded data, metadata, raw data of the sensor signal, and the synchronization signal.

[0155] As shown in Figure 21, point cloud data is not generated from the sensor signal (RAW data) obtained by the sensing unit 7341B; the sensor signal is encoded as RAW data. For example, if the second sensor signal is two-dimensional information obtained from a FLASH LiDAR or a CMOS sensor such as a camera, the encoding unit 7344B encodes the second sensor signal using a video codec such as AVC or HEVC. This enables highly efficient encoding. Furthermore, by utilizing existing codecs, it becomes possible to build a low-cost system.

[0156] Thus, the three-dimensional data multiplexing device uses both a means for encoding point cloud data after conversion and a means for encoding raw data without converting it to point cloud data, depending on the sensing unit, and multiplexes the respective encoded data.

[0157] Next, we will explain an example of a method for generating an output signal in a predetermined file format through multiplexing. The following example describes the case where the predetermined file format is ISOBMFF (ISO based media file format). Note that the file format is not limited to ISOBMFF; other file formats may also be used.

[0158] ISOBMFF is a file format standard defined in ISO / IEC 14496-12. ISOBMFF specifies a format that can store data from various media, such as video, audio, and text, and is a media-independent standard.

[0159] The method for storing each type of media in ISOBMFF is specified separately. For example, the storage method for AVC video and HEVC video is specified in ISO / IEC 14496-15.

[0160] On the other hand, a method is needed to store encoded data obtained from multiple sensor information (sensor signals) into ISOBMFF. Figure 22 shows a protocol for encoding multiple sensor information using various encoding methods and storing them in ISOBMFF.

[0161] Data1 to Data5 are sensor data (sensor signals) acquired from various types of sensors, such as RAW data. Data1 and Data2 are converted to a 3D point cloud format and encoded using either Codec1 or Codec2, which are encoding methods for 3D point cloud formats. Data3 and Data4 are converted to a 2D data format, such as an image, and encoded using either Codec3 or Codec4, which are encoding methods for 2D data formats.

[0162] Each encoded data is converted to a NAL unit using a predetermined method and stored in ISOBMFF. The NAL unit may be in a common format for 3D and 2D formats, or it may be in a different format. Similarly, NAL units for different encoding methods may be in a common format, or they may be in a different format. Furthermore, the sensor data format may be a 1D format or other format, in addition to the 3D and 2D formats listed here.

[0163] Data5 is the case where sensor data acquired from the sensor is stored directly in ISOBMFF without encoding.

[0164] By providing a format that integrates and stores data from any combination of these data points, data management for systems handling multiple sensors becomes easier, enabling the implementation of various functions.

[0165] Next, the configuration of ISOBMFF will be explained. The three-dimensional data multiplexing device stores multiple sensor data into ISOBMFF. Figure 23 shows an example of the configuration of the input data to be multiplexed. Figure 24 shows an example of the configuration of the NAL unit. Figure 25 shows an example of the configuration of ISOBMFF. Figure 26 shows an example of the configuration of moov and mdat.

[0166] The encoded data included in the input data is broadly divided into encoded data (Data) and metadata (Meta). Metadata includes metadata indicated in the header for each encoded data and metadata stored as a parameter set in an independent NAL unit. In some cases, metadata is also included in the encoded data. A three-dimensional data multiplexer stores these multiple NAL units and RAW data for each different codec into a single ISOBMFF.

[0167] ISOBMFF is structured as a box system. The boxes in ISOBMFF primarily consist of "moov" and "meta" for storing metadata, and "mdat" for storing data.

[0168] Encoded data and RAW data are stored in the "mdat" folder in ISOMBFF on a sample-by-sample basis. Metadata in the input data is stored in the "trak" folder within the "moov" folder in ISOMBFF in a predetermined format for each encoded data set. Metadata and synchronization information contained within the encoded data are also stored within the "moov" folder.

[0169] Information for retrieving data from "mdat" (such as the data address information (offset information) from the beginning of the file and the data size) is stored in the metadata for each encoded data file. In addition, "ftyp" indicates the file type of the subsequent data.

[0170] Note that the format and box name may differ from those listed here, as long as they have the same functionality.

[0171] Furthermore, in use cases such as real-time communication, the boxes, such as "moov" and "mdat," may be divided into units and transmitted at different times. The data of these divided units may also be interleaved.

[0172] The three-dimensional data multiplexing device defines boxes that represent configuration information (hereinafter simply referred to as "configuration information") and stores identification information for multiple data contained in a file within the configuration information. Furthermore, the three-dimensional data multiplexing device stores identification information that allows access to the metadata of each data within the configuration information.

[0173] Figure 27 shows an example of the configuration information. Figure 28 shows an example of the configuration information syntax.

[0174] The configuration information includes information about the content and components that make up the ISOBMFF file, sensor information when the original data of the components was acquired, format information, and encoding method.

[0175] As shown in Figure 27, the configuration information includes overall configuration information and configuration information for each encoded data. Multiple configuration information items may have the same data structure or box, or they may have different data structures or boxes.

[0176] In the mp4 box, the `type` field indicates that it is a configuration information box, for example, using 4CC such as "msuc". The overall configuration information (data()) shows the configuration of multiple data with different encoding schemes. data() includes num_of_data, data_type, and data_configuration.

[0177] `num_of_data` indicates the number of encoded and raw data files that make up the file. `data_type` indicates identification information for each data item; in other words, `data_type` indicates multiple data types.

[0178] Specifically, `data_type` indicates whether the data is point cloud data or sensor signals (e.g., RAW data). `data_type` may also indicate whether the data is encoded or not. Furthermore, `data_type` may indicate the encoding method (encoding scheme) used to encode the encoded data. An encoding method could be, for example, GPPC or VPPC. The encoding method could also be one of the Codecs 1-4 shown in Figure 22. Additionally, `data_type` may indicate information for identifying configuration information.

[0179] Furthermore, `data_type` may indicate the type of source data for the point cloud data. The type of source data refers to the type of sensor that generated the sensor signal (e.g., whether it is a 2D or 3D sensor) if the source data is a sensor signal. Additionally, `data_type` may include information indicating the data format of the sensor signal (e.g., whether it is 1D, 2D, or 3D information).

[0180] For example, data_type=0 indicates PCC Codec1, data_type=1 indicates PCC Codec2, data_type=2 indicates Video Codec3, and data_type=4 indicates 3D axis sensor RAW data. data_configuration shows the configuration information for each data set.

[0181] `data_configuration()` contains configuration information for each encoded data, including `num_of_component`, `component_type`, and `component_id`.

[0182] `num_of_component` indicates the number of components in the encoded data. `component_type` indicates the type of component. For example, in the case of PCC encoding, `component_type` indicates whether the component is geometry, an attribute, or metadata.

[0183] The component_id is a unique identifier used to link a component to other metadata and data.

[0184] Furthermore, the encoding method may be any method used for audio, text, applications, or 360-degree images, in addition to video codecs and PCC codecs. The data may also be processed data such as meshes or CAD data. The encoding method may be the same codec, or it may be an encoding method with different profiles, levels, or tools; any encoding method can be handled integrally.

[0185] In this way, by multiplexing the data necessary for using the decoded point cloud data in an application into a single file, file management and synchronization management handled by the application can be simplified.

[0186] Figure 29 shows an example of the configuration of the data box "mdat". Each encoded data or RAW data is stored individually in a sample, which is the smallest unit of the data box.

[0187] Furthermore, synchronization information such as timestamps for each encoded data contained in a file is set based on overall synchronization information such as a common reference time. Also, the synchronization information is assumed to be synchronized.

[0188] Alternatively, for example, the reference time, time resolution, and time interval in the timestamps of multiple encoded data may be aligned, and synchronization information may be shared across multiple encoded data. In this case, the synchronization information only needs to be stored in one or more of either the synchronization information for each encoded data or the common synchronization information. In this case, the metadata includes at least one of the following: information indicating where the common time information is stored, and information indicating that the synchronization information is common.

[0189] Furthermore, if the encoded data is synchronized, the three-dimensional data multiplexer may store multiple synchronized encoded data as a single sample. On the other hand, if at least one of the reference time, time resolution, and time interval is not the same for multiple encoded data, the three-dimensional data multiplexer may separately derive difference information indicating the difference in timestamps between the encoded data and store the derived difference information in the output signal. The three-dimensional data multiplexer may also store a flag in the output signal indicating whether or not the data is synchronized.

[0190] The three-dimensional data demultiplexer synchronizes the encoded data by processing each sample at the time indicated by the timestamp shown in the metadata, using the synchronization information of each encoded data and the overall synchronization information.

[0191] The following describes an example of application processing. Figure 30 is a flowchart of an example of application processing. When the application operation is started, the 3D data demultiplexer acquires an ISOBMFF file containing point cloud data and multiple encoded data (S7301). For example, the 3D data demultiplexer may acquire the ISOBMFF file via communication or read it from stored data.

[0192] Next, the 3D data demultiplexer analyzes the overall configuration information in the ISOBMFF file and identifies the data to be used in the application (S7302). For example, the 3D data demultiplexer acquires the data to be used for processing and does not acquire the data that is not used for processing.

[0193] Next, the three-dimensional data demultiplexing device extracts one or more data points to be used in the application and analyzes the configuration information of those data points (S7303).

[0194] If the data type is encoded data (encoded data in S7304), the three-dimensional data demultiplexer converts the ISOBMFF into an encoded stream and extracts the timestamp (S7305). The three-dimensional data demultiplexer may also determine whether the data is synchronized, for example, by referring to a flag indicating whether the data is synchronized, and perform synchronization processing if it is not synchronized.

[0195] Next, the three-dimensional data demultiplexer decodes the data in a predetermined manner according to the timestamp and other instructions, and processes the decoded data (S7306).

[0196] On the other hand, if the data type is encoded data (RAW data in S7304), the 3D data demultiplexer extracts the data and timestamp (S7307). The 3D data demultiplexer may also determine whether the data is synchronized, for example, by referring to a flag indicating whether the data is synchronized, and may perform synchronization processing if it is not synchronized. Next, the 3D data demultiplexer processes the data according to the timestamp and other instructions (S7308).

[0197] For example, we will describe an example where sensor signals acquired by beam LiDAR, flash LiDAR, and a camera are encoded and multiplexed using different encoding schemes. Figure 31 shows an example of the sensor range of beam LiDAR, flash LiDAR, and a camera. For example, beam LiDAR detects in all directions around the vehicle (sensor), while flash LiDAR and a camera detect a range in one direction of the vehicle (e.g., forward).

[0198] In applications that handle LiDAR point clouds in an integrated manner, the 3D data demultiplexer refers to the overall configuration information to extract and decode the encoded data from beam LiDAR and FLASH LiDAR. The 3D data demultiplexer does not extract camera images.

[0199] The three-dimensional data demultiplexer processes the encoded data of both the LiDAR and FLASH LiDAR simultaneously, according to their respective timestamps.

[0200] For example, a three-dimensional data demultiplexing device may display the processed data using a display device, combine point cloud data from beam LiDAR and flash LiDAR, or perform rendering and other processing.

[0201] Furthermore, in applications that perform calibration between data sets, the 3D data demultiplexer may extract sensor position information for use in the application.

[0202] For example, a three-dimensional data demultiplexing device may select whether to use beam LiDAR information or FLASH LiDAR in the application, and switch processing according to the selection.

[0203] In this way, data acquisition and encoding can be adaptively changed according to the application's processing, thus reducing processing load and power consumption.

[0204] The following describes use cases in autonomous driving. Figure 32 shows an example of the configuration of an autonomous driving system. This autonomous driving system includes a cloud server 7350 and edge devices 7360 such as in-vehicle devices or mobile devices. The cloud server 7350 comprises a demultiplexing unit 7351, decoding units 7352A, 7352B, and 7355, a point cloud data synthesis unit 7353, a large-scale data storage unit 7354, a comparison unit 7356, and an encoding unit 7357. The edge 7360 comprises sensors 7361A and 7361B, point cloud data generation units 7362A and 7362B, synchronization unit 7363, encoding units 7364A and 7364B, multiplexing unit 7365, update data storage unit 7366, demultiplexing unit 7367, decoding unit 7368, filter 7369, self-position estimation unit 7370, and operation control unit 7371.

[0205] In this system, the Edge 7360 downloads large-scale point cloud map data stored on the cloud server 7350. The Edge 7360 performs self-localization processing by matching the large-scale data with sensor information obtained from the Edge 7360 (vehicle or terminal). The Edge 7360 also uploads the acquired sensor information to the cloud server 7350 and updates the large-scale data with the latest map data.

[0206] Furthermore, various applications within the system that handle point cloud data deal with point cloud data encoded using different methods.

[0207] The cloud server 7350 encodes and multiplexes large-scale data. Specifically, the encoding unit 7357 encodes using a third encoding method suitable for encoding large-scale point clouds. The encoding unit 7357 also multiplexes the encoded data. The large-scale data storage unit 7354 stores the data encoded and multiplexed by the encoding unit 7357.

[0208] Edge 7360 performs sensing. Specifically, point cloud data generation unit 7362A generates first point cloud data (location information (geometry) and attribute information) using sensing information acquired by sensor 7361A. Point cloud data generation unit 7362B generates second point cloud data (location information and attribute information) using sensing information acquired by sensor 7361B. The generated first and second point cloud data are used for self-position estimation or vehicle control in autonomous driving, or for map updating. In each of these processes, some information from the first and second point cloud data may be used.

[0209] Edge 7360 performs self-localization. Specifically, Edge 7360 downloads large-scale data from cloud server 7350. Demultiplexing unit 7367 obtains encoded data by demultiplexing the large-scale data in a file format. Decoding unit 7368 obtains large-scale data, which is large-scale point cloud map data, by decoding the obtained encoded data.

[0210] The self-position estimation unit 7370 estimates the vehicle's position on the map by matching the acquired large-scale data with the first and second point cloud data generated by the point cloud data generation units 7362A and 7362B. The driving control unit 7371 then uses the matching result or the self-position estimation result for driving control.

[0211] The self-position estimation unit 7370 and the operation control unit 7371 may extract specific information, such as position information, from the large-scale data and process it using the extracted information. The filter 7369 also performs processing such as correction or decimation on the first point cloud data and the second point cloud data. The self-position estimation unit 7370 and the operation control unit 7371 may use the first point cloud data and the second point cloud data after such processing. The self-position estimation unit 7370 and the operation control unit 7371 may also use the sensor signals obtained from sensors 7361A and 7361B.

[0212] The synchronization unit 7363 performs time synchronization and position correction between data of multiple sensor signals or multiple point cloud data. The synchronization unit 7363 may also correct the position information of the sensor signals or point cloud data to match the large-scale data based on position correction information between the large-scale data and sensor data generated by the self-position estimation process.

[0213] Synchronization and position correction may be performed on the cloud server 7350 instead of the edge 7360. In this case, the edge 7360 may multiplex the synchronization information and position information and send it to the cloud server 7350.

[0214] Edge 7360 encodes and multiplexes sensor signals or point cloud data. Specifically, sensor signals or point cloud data are encoded using a first encoding method or a second encoding method suitable for encoding each signal. For example, encoding unit 7364A generates first encoded data by encoding first point cloud data using the first encoding method. Encoding unit 7364B generates second encoded data by encoding second point cloud data using the second encoding method.

[0215] The multiplexing unit 7365 generates a multiplexed signal by multiplexing the first encoded data, the second encoded data, and synchronization information. The update data storage unit 7366 stores the generated multiplexed signal. The update data storage unit 7366 also uploads the multiplexed signal to the cloud server 7350.

[0216] The cloud server 7350 synthesizes the point cloud data. Specifically, the demultiplexing unit 7351 demultiplexes the multiplexed signal uploaded to the cloud server 7350 to obtain the first encoded data and the second encoded data. The decoding unit 7352A decodes the first encoded data to obtain the first point cloud data (or sensor signal). The decoding unit 7352B decodes the second encoded data to obtain the second point cloud data (or sensor signal).

[0217] The point cloud data synthesis unit 7353 synthesizes the first point cloud data and the second point cloud data in a predetermined manner. If synchronization information and position correction information are multiplexed in the multiplexed signal, the point cloud data synthesis unit 7353 may use that information to perform the synthesis.

[0218] The decoding unit 7355 demultiplexes and decodes the large-scale data stored in the large-scale data storage unit 7354. The comparison unit 7356 compares the point cloud data generated based on the sensor signals obtained at the edge 7360 with the large-scale data held by the cloud server 7350 and determines the point cloud data that needs updating. The comparison unit 7356 updates the point cloud data from the large-scale data that it determines needs updating with the point cloud data obtained from the edge 7360.

[0219] The encoding unit 7357 encodes and multiplexes the updated large-scale data, and stores the resulting data in the large-scale data storage unit 7354.

[0220] As described above, the signals handled and the signals to be multiplexed or the encoding methods used may differ depending on the intended use or application. Even in such cases, flexible decoding and application processing become possible by multiplexing data with various encoding methods using this embodiment. Furthermore, even when the encoding methods of the signals differ, various applications and systems can be built and flexible services can be provided by converting to a suitable encoding method through demultiplexing, decoding, data conversion, encoding, and multiplexing processes.

[0221] As described above, the three-dimensional data multiplexing device according to this embodiment performs the processing shown in Figure 33. The three-dimensional data multiplexing device generates an output signal with a predetermined file structure (e.g., ISOBMFF) by multiplexing multiple types of data, including point cloud data (S7311). Next, the three-dimensional data multiplexing device stores information indicating the type of each of the multiple data included in the output signal (e.g., data_type) in the metadata (control information) of the file structure (S7312).

[0222] According to this, the three-dimensional data multiplexer stores information indicating the type of each of the multiple data included in the output signal in the metadata of the file structure. This allows the three-dimensional data demultiplexer that receives the output signal to easily determine the type of each data. In this way, the three-dimensional data multiplexing method can appropriately multiplex and transmit point cloud data.

[0223] For example, information indicating the type of each of multiple data sets may include (1) the encoding scheme applied to the data, (2) the data structure, (3) the type of sensor that generated the data, or (4) the data format.

[0224] For example, metadata in a file structure includes synchronization information to synchronize the timing of multiple data points contained in the output signal. This allows a three-dimensional data demultiplexer receiving the output signal to synchronize the multiple data points.

[0225] For example, synchronization information indicates the difference in timestamps between multiple data points. This allows for a reduction in the amount of data in the output signal.

[0226] For example, a three-dimensional data multiplexing device comprises a processor and memory, and the processor uses the memory to perform the above processing.

[0227] Furthermore, the three-dimensional data demultiplexing device according to this embodiment performs the processing shown in Figure 34. The three-dimensional data demultiplexing device obtains information (e.g., data_type) indicating the type of each of the multiple data included in the output signal, which is stored in the metadata of the file configuration, from the output signal of a predetermined file configuration (e.g., ISOBMFF) in which multiple types of data, including point cloud data, are multiplexed (S7321). The three-dimensional data demultiplexing device obtains multiple data from the output signal using the information indicating the type of each of the multiple data (S7322). For example, the three-dimensional data demultiplexing device selectively obtains the necessary data from the output signal using the information indicating the type of each of the multiple data. As a result, the three-dimensional data demultiplexing device can easily determine the type of each data.

[0228] For example, information indicating the type of each of multiple data sets may include (1) the encoding scheme applied to the data, (2) the data structure, (3) the type of sensor that generated the data, or (4) the data format.

[0229] For example, metadata in a file structure includes synchronization information to synchronize the timing of multiple data points contained in the output signal. For example, a three-dimensional data demultiplexer uses synchronization information to synchronize multiple data points.

[0230] For example, synchronization information indicates the difference in timestamps between multiple data points. For instance, synchronization information includes information indicating the timestamp of one of the multiple data points. A three-dimensional data demultiplexer restores the timestamps of the other data points by adding the difference indicated by the synchronization information to the timestamp of one of the multiple data points. This reduces the amount of data in the output signal.

[0231] For example, a three-dimensional data demultiplexing device comprises a processor and memory, and the processor uses the memory to perform the above processing.

[0232] The three-dimensional data multiplexing device, three-dimensional data demultiplexing device, three-dimensional data encoding device, three-dimensional data decoding device, etc., according to embodiments of the present disclosure have been described above, but the present disclosure is not limited to these embodiments.

[0233] Furthermore, each processing unit included in the three-dimensional data multiplexer, three-dimensional data demultiplexer, three-dimensional data encoding device, and three-dimensional data decoding device according to the above embodiment is typically implemented as an integrated circuit (LSI). These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip.

[0234] Furthermore, integrated circuit implementation is not limited to LSIs; it may also be achieved using dedicated circuits or general-purpose processors. Field-Programmable Gate Arrays (FPGAs), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used.

[0235] Furthermore, in each of the above embodiments, each component may be implemented by being composed of dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0236] Furthermore, this disclosure may be implemented as a three-dimensional data multiplexing method, a three-dimensional data demultiplexing method, a three-dimensional data coding method, or a three-dimensional data decoding method, etc., performed by a three-dimensional data multiplexing device, a three-dimensional data demultiplexing device, a three-dimensional data coding device, and a three-dimensional data decoding device, etc.

[0237] Furthermore, the division of functional blocks in the block diagram is just one example; multiple functional blocks can be implemented as a single functional block, a single functional block can be divided into multiple parts, or some functions can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can be processed in parallel or time-sharing by a single piece of hardware or software.

[0238] Furthermore, the order in which each step in the flowchart is performed is illustrative for the purpose of specifically illustrating this disclosure, and may be in a different order. Also, some of the above steps may be performed simultaneously (in parallel) with other steps.

[0239] The three-dimensional data multiplexing device, three-dimensional data demultiplexing device, three-dimensional data encoding device, and three-dimensional data decoding device, etc., according to one or more embodiments have been described above based on embodiments, but this disclosure is not limited to these embodiments. Without departing from the spirit of this disclosure, various modifications that a person skilled in the art can conceive of may be applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments. [Industrial applicability]

[0240] This disclosure is applicable to three-dimensional data multiplexing devices, three-dimensional data demultiplexing devices, three-dimensional data encoding devices, and three-dimensional data decoding devices. [Explanation of Symbols]

[0241] 4601 Three-Dimensional Data Encoding System 4602 Three-dimensional data decoding system 4603 Sensor terminal 4604 External connection section 4611 Point Cloud Data Generation System 4612 Presentation section 4613 Encoding section 4614 Multiplexer 4615 Input / output section 4616 Control Unit 4617 Sensor Information Acquisition Unit 4618 Point Cloud Data Generation Unit 4621 Sensor Information Acquisition Unit 4622 Input / output section 4623 Demultiplexer 4624 Decoding section 4625 Presentation section 4626 User Interface 4627 Control Unit 4630 First encoding unit 4631 Location information encoder 4632 Attribute information encoder 4633 Additional Information Encoding Unit 4634 Multiplexer 4640 First Decoding Unit 4641 Demultiplexer 4642 Location Information Decoding Unit 4643 Attribute Information Decoding Unit 4644 Additional Information Decoding Unit 4650 Second encoding unit 4651 Additional Information Generation Unit 4652 Position image generation unit 4653 Attribute Image Generation Unit 4654 Video Encoding Unit 4655 Additional Information Encoding Unit 4656 Multiplexer 4660 Second decoding unit 4661 Demultiplexer 4662 Video Decoding Unit 4663 Additional Information Decoding Unit 4664 Location information generator 4665 Attribute information generation section 7301, 7301A, 7301B, 7301C Sensing Devices 7302, 7302A, 7302B, 7302C Sensor Information Input Section 7303, 7303A, 7303C Point Cloud Data Generation Unit 7304A, 7304B Sensing Unit 7305 Merge section 7310 First device 7311 and 7321 Sensing Units 7312 and 7322 Output I / F 7320 Second Device 7323 Input I / F 7324 Processing Unit 7325 Point Cloud Data Generation Unit 7326 Encoding Unit 7331A, 7331B, and 7331C Sensing Units 7332A, 7332B, and 7332C Sensor Information Input Units 7333A and 7333B Point Cloud Data Generation Units 7334A and 7334B Encoding Units 7335 Synchronization Unit 7336 Multiplexing Unit 7341A, 7341B, and 7341C Sensing Units 7342A, 7342B, and 7342C Input / Output Units 7343 Point Cloud Data Generation Unit 7344A and 7344B Encoding Units 7345 Synchronization Unit 7346 Multiplexing Unit 7350 Cloud Server 7351 Demultiplexing Unit 7352A and 7352B Decoding Units 7353 Point Cloud Data Synthesis Unit 7354 Large-Scale Data Storage Unit S7355 Decoding Unit 7356 Comparison Unit 7357 Encoding Unit 7360 Edge 7361A and 7361B Sensors 7362A and 7362B Point Cloud Data Generation Units 7363 Synchronization Unit 7364A and 7364B Encoding Units 7365 Multiplexing Unit 7366 Update Data Storage Unit 7367 Demultiplexing Unit 7368 Decoding Unit 7369 Filter 7370 Self-Position Estimation Unit 7371 Driving Control Unit

Claims

1. Processor and Equipped with memory, The processor uses the memory to: A first sensor signal is acquired, point cloud data is generated based on the first sensor signal, and first data corresponding to the point cloud data is generated. A second sensor signal is acquired, and without generating point cloud data for the second sensor signal, second data derived from the second sensor signal is generated. By multiplexing the first data, the second data, and the metadata, an output signal with a predetermined file structure is generated. The metadata stores information indicating the type of each of the first data and the second data. Information processing device.

2. The first data is encoded data generated by encoding the point cloud data. The information processing apparatus according to claim 1.

3. The second data is the RAW data of the second sensor signal. The information processing apparatus according to claim 1.

4. The second data is two-dimensional data generated by encoding the second sensor signal. The information processing apparatus according to claim 1.

5. The second sensor signal is a signal containing two-dimensional information obtained from a FLASH LiDAR or camera. The information processing apparatus according to claim 4.

6. If the second data is the two-dimensional data, The processor encodes the second sensor signal using AVC or HEVC. The information processing apparatus according to claim 4 or 5.

7. The aforementioned predetermined file structure is ISOBMFF. The information processing apparatus according to any one of claims 1 to 6.

8. The aforementioned predetermined file structure includes an mdat file for storing data and a moov or meta file for storing metadata. The first data and the second data are stored in the mdat. The information processing apparatus according to claim 7.

9. The metadata includes offset information and size information for obtaining the first data or the second data from the mdat. The information processing apparatus according to claim 8.

10. The information indicating each of the aforementioned types stored in the metadata is: This indicates whether the first data or the second data is point cloud data, a sensor signal, encoded or not, or whether the data format is 1D information, 2D information, or 3D information. The information processing apparatus according to any one of claims 1 to 9.

11. The configuration information corresponding to the first data is: This includes the number of components, information indicating whether each component is geometry, an attribute, or metadata, and identification information for linking each component to other metadata or data. The information processing apparatus according to any one of claims 1 to 10.

12. An information processing method using an information processing device comprising a processor and memory, Using the memory, the processor A first sensor signal is acquired, point cloud data is generated based on the first sensor signal, and first data corresponding to the point cloud data is generated. A second sensor signal is acquired, and without generating point cloud data for the second sensor signal, second data derived from the second sensor signal is generated. By multiplexing the first data, the second data, and the metadata, an output signal with a predetermined file structure is generated. The metadata stores information indicating the type of each of the first data and the second data. Information processing methods.

13. On the computer, A first sensor signal is acquired, point cloud data is generated based on the first sensor signal, and first data corresponding to the point cloud data is generated. A second sensor signal is acquired, and without generating point cloud data for the second sensor signal, second data derived from the second sensor signal is generated. By multiplexing the first data, the second data, and the metadata, an output signal with a predetermined file structure is generated. The metadata stores information indicating the type of each of the first data and the second data. A program to execute a process.

Images