Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device
The method and device enhance three-dimensional data encoding and decoding by including sensor identification information, enabling efficient extraction and management of point cloud data based on application-specific requirements.
Patent Information
- Application Number
- JP2024161583
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-05-10
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2040-05-08
AI Technical Summary
Existing three-dimensional data decoding devices struggle to appropriately extract point cloud data based on application requirements, necessitating improved methods for encoding and decoding three-dimensional data to manage large data volumes efficiently.
A method and device for encoding and decoding three-dimensional data that includes sensor identification information in the encoded data, allowing for the extraction of point cloud data from bitstreams based on desired sensors, and a system for generating and multiplexing point cloud data with additional information for efficient storage and transmission.
Enables the appropriate extraction of point cloud data from bitstreams, facilitating efficient storage and transmission while accommodating diverse application needs by incorporating sensor-specific information in the encoding process.
Smart Images

Figure 2024169583000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. [Background technology]
[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, such as computer vision for autonomous operation of automobiles or robots, map information, monitoring, infrastructure inspection, video distribution, etc. 3D data is acquired in various ways, such as distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.
[0003] One method of expressing three-dimensional data is a method called a point cloud, which represents the shape of a three-dimensional structure using a group of points in a three-dimensional space. In a point cloud, the positions and colors of the points are stored. Point clouds are expected to become the mainstream method of expressing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just like two-dimensional video images (examples include MPEG-4 AVC or HEVC standardized by MPEG).
[0004] In addition, compression of point clouds is partially supported by public libraries (Point Cloud Library) that perform point cloud-related processing.
[0005] Furthermore, a technique is known that uses three-dimensional map data to search for and display facilities located around a vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0006] [Patent Document 1] International Publication No. 2014 / 020663 Summary of the Invention [Problem to be solved by the invention]
[0007] For such point cloud data, it is desirable for a three-dimensional data decoding device that receives the point cloud data to be able to extract the point cloud data appropriately depending on the application.
[0008] The present disclosure aims to provide a three-dimensional data encoding method or three-dimensional data encoding device capable of generating a bit stream from which point cloud data can be appropriately extracted, or a three-dimensional data decoding method or three-dimensional data decoding device that decodes the bit stream. [Means for solving the problem]
[0009] A three-dimensional data encoding method according to one aspect of the present disclosure is an encoding method executed by an encoding device, which generates encoded data by encoding point cloud data composed of a plurality of three-dimensional points, and generates a bit stream including the encoded data and control information, wherein the encoded data includes a plurality of point data each corresponding to one of the plurality of three-dimensional points, and when the plurality of point data includes one or more point data to which sensor identification information is assigned, the control information indicates that the encoded data includes the sensor identification information.
[0010] A three-dimensional data decoding method according to one embodiment of the present disclosure obtains encoded data and control information from a bit stream, and decodes the encoded data to obtain point cloud data composed of a plurality of three-dimensional points, wherein the encoded data includes a plurality of point data each corresponding to one of the plurality of three-dimensional points, and when the plurality of point data includes one or more point data to which sensor identification information is assigned, the control information indicates that the encoded data includes the sensor identification information. Effect of the Invention
[0011] The present disclosure can provide a three-dimensional data encoding method or three-dimensional data encoding device that can generate a bit stream from which point cloud data can be appropriately extracted, or a three-dimensional data decoding method or three-dimensional data decoding device that decodes the bit stream. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram showing a configuration of a three-dimensional data encoding / decoding system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of point cloud data according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a configuration of a data file in which point cloud data information according to the first embodiment is described. [Figure 4] FIG. 4 is a diagram showing types of point cloud data according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing a configuration of a first encoding unit according to the first embodiment. [Figure 6] FIG. 6 is a block diagram of a first encoding unit according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing a configuration of a first decoding unit according to the first embodiment. [Figure 8] FIG. 8 is a block diagram of a first decoding unit according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing a configuration of a second encoding unit according to the first embodiment. [Figure 10] FIG. 10 is a block diagram of a second encoding unit according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating a configuration of a second decoding unit according to the first embodiment. [Figure 12] FIG. 12 is a block diagram of a second decoding unit according to the first embodiment. [Figure 13] FIG. 13 is a diagram showing a protocol stack related to PCC encoded data according to the first embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a configuration of a point cloud data generating device according to the first embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of the configuration of a point cloud data generating device according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of the configuration of a point cloud data generating device according to the first embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of a configuration of a point cloud data encoding system according to the first embodiment. [Figure 18] FIG. 18 is a diagram illustrating an example of a configuration of a three-dimensional data multiplexing device according to the first embodiment. As shown in FIG. [Figure 19] FIG. 19 is a diagram showing a specific example of the three-dimensional data multiplexing device according to the first embodiment. [Figure 20] FIG. 20 is a diagram showing the sensor ranges of various sensors according to the first embodiment. [Figure 21] FIG. 21 is a diagram illustrating another example of the configuration of the three-dimensional data multiplexing device according to the first embodiment. [Figure 22] FIG. 22 is a diagram showing a protocol for storing a plurality of pieces of information in a file format according to the first embodiment. [Figure 23] FIG. 23 is a diagram illustrating an example of a configuration of input data according to the first embodiment. [Figure 24] FIG. 24 is a diagram showing an example of a structure of a NAL unit according to the first embodiment. [Diagram 25] FIG. 25 is a diagram illustrating an example of a configuration of an ISOBMFF according to the first embodiment. [Figure 26] FIG. 26 is a diagram showing an example of the configuration of moov and mdat according to the first embodiment. [Figure 27] FIG. 27 is a diagram illustrating an example of the configuration information according to the first embodiment. [Figure 28] FIG. 28 is a diagram illustrating an example of syntax of configuration information according to the first embodiment. [Figure 29] FIG. 29 is a diagram showing an example of the configuration of an mdat according to the first embodiment. [Diagram 30] FIG. 30 is a flowchart showing an example of application processing according to the first embodiment. [Diagram 31] FIG. 31 is a diagram showing sensor ranges of various sensors according to the first embodiment. [Diagram 32] FIG. 32 is a diagram illustrating a configuration example of the autonomous driving system according to the first embodiment. As shown in FIG. [Diagram 33] FIG. 33 is a flowchart of the three-dimensional data multiplexing process according to the first embodiment. [Diagram 34] FIG. 34 is a flowchart of three-dimensional data demultiplexing processing according to the first embodiment. [Diagram 35] FIG. 35 is a diagram illustrating an example of a configuration of a three-dimensional data encoding device according to the second embodiment. [Diagram 36] FIG. 36 is a block diagram of a point cloud data generating unit according to the second embodiment. [Figure 37] FIG. 37 is a diagram showing an example of information of three-dimensional points according to the second embodiment. [Figure 38] FIG. 38 is a diagram showing an example of information of three-dimensional points according to the second embodiment. [Figure 39] FIG. 39 is a diagram showing an example of information of three-dimensional points according to the second embodiment. [Diagram 40] FIG. 40 is a diagram for explaining the synthesis process according to the second embodiment. In FIG. [Diagram 41] FIG. 41 is a diagram showing an example of combined point group information according to the second embodiment. [Diagram 42] FIG. 42 is a diagram for explaining the synthesis process according to the second embodiment. In FIG. [Diagram 43] FIG. 43 is a diagram for explaining the synthesis process according to the second embodiment. In FIG. [Diagram 44] FIG. 44 is a block diagram of a three-dimensional data encoding / decoding system according to the second embodiment. [Diagram 45] FIG. 45 is a diagram showing an example of a configuration of a bit stream according to the second embodiment. [Figure 46] FIG. 46 is a flowchart of the decoding process according to the second embodiment. [Figure 47] FIG. 47 is a flowchart of the extraction process according to the second embodiment. [Figure 48] FIG. 48 is a diagram illustrating an example of the extraction process according to the second embodiment. [Figure 49] FIG. 49 is a block diagram of a three-dimensional data encoding / decoding system according to the second embodiment. [Figure 50] FIG. 50 is a flowchart of the encoding process according to the second embodiment. [Figure 51] FIG. 51 is a diagram for explaining the division process according to the second embodiment. In FIG. [Figure 52] FIG. 52 is a diagram showing an example of a configuration of a bit stream according to the second embodiment. [Figure 53] FIG. 53 is a flowchart of the point cloud selection process according to the second embodiment. [Figure 54] FIG. 54 is a diagram showing an example of a screen for the point cloud selection process according to the second embodiment. [Figure 55] FIG. 55 is a diagram showing an example of a screen for the point cloud selection process according to the second embodiment. [Figure 56] FIG. 56 is a diagram showing an example of a screen for the point cloud selection process according to the second embodiment. [Figure 57] FIG. 57 is a flowchart of three-dimensional data encoding processing according to the second embodiment. [Figure 58] FIG. 58 is a flowchart of three-dimensional data decoding processing according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] A three-dimensional data encoding method according to one embodiment of the present disclosure generates encoded data by encoding point cloud data, and generates a bit stream including the encoded data, wherein the bit stream includes first information indicating a sensor corresponding to the point cloud data.
[0014] According to this, the three-dimensional data decoding device can extract point cloud data obtained by a desired sensor using the first information. Therefore, the three-dimensional data encoding method can generate a bit stream from which the point cloud data can be appropriately extracted.
[0015] For example, the first information may include second information that is generated for each of a plurality of three-dimensional points included in the point cloud data and indicates a sensor that corresponds to the corresponding three-dimensional point.
[0016] For example, the point cloud data may include position information of each of the multiple three-dimensional points and one or more pieces of attribute information, and the second information may be included in the encoded data as one piece of the one or more pieces of attribute information.
[0017] For example, the bit stream may include third information common to the plurality of three-dimensional points, the second information being an identifier of a sensor, and the third information may include fourth information indicating an identification of the sensor and a correspondence with the sensor.
[0018] For example, the bit stream may include third information common to the plurality of 3D points, the third information indicating a number of 3D points per sensor.
[0019] For example, the point cloud data may include position information and one or more pieces of attribute information for each of the plurality of three-dimensional points, and the first information may indicate a sensor corresponding to each of the one or more pieces of attribute information.
[0020] For example, the three-dimensional data encoding method may further generate the point cloud data by synthesizing first point cloud data based on a first sensor and second point cloud data based on a second sensor, and if the first point cloud data and the second point cloud data contain attribute information of a first point having identical position information, attribute information based on the attribute information of the first point contained in the first point cloud data and the attribute information of the first point contained in the second point cloud data may be generated as attribute information after synthesis, and the first information may indicate that the information of the first point contained in the point cloud data is based on the first sensor and the second sensor.
[0021] For example, in encoding the point cloud data, the point cloud data may be divided into first point cloud data based on a first sensor and second point cloud data based on the second sensor, and the encoded data may be generated by encoding the first point cloud data and the second point cloud data, wherein the bit stream includes first control information for the first point cloud data and second control information for the second point cloud data, and the first information may include: (1) information included in the first control information and indicating a sensor corresponding to the first point cloud data, and (2) information included in the second control information and indicating a sensor corresponding to the second point cloud data.
[0022] For example, in encoding the point cloud data, the point cloud data is divided into first point cloud data based on a first sensor and second point cloud data based on the second sensor, and the first point cloud data and the second point cloud data are encoded to generate the encoded data, the bit stream includes control information common to the first point cloud data and the second point cloud data, and the first information may be included in the control information.
[0023] A three-dimensional data decoding method according to one embodiment of the present disclosure obtains first information and the point cloud data by decoding a bit stream that includes encoded data generated by encoding point cloud data and that includes first information indicating a sensor corresponding to the point cloud data.
[0024] According to this, the three-dimensional data decoding method can obtain point cloud data by decoding a bit stream from which point cloud data can be appropriately extracted.
[0025] For example, the first information may include second information that is generated for each of a plurality of three-dimensional points included in the point cloud data and indicates a sensor that corresponds to the corresponding three-dimensional point.
[0026] For example, the point cloud data may include position information of each of the multiple three-dimensional points and one or more pieces of attribute information, and the second information may be included in the encoded data as one piece of the one or more pieces of attribute information.
[0027] For example, the bit stream may include third information common to the plurality of three-dimensional points, the second information being an identifier of a sensor, and the third information may include fourth information indicating an identification of the sensor and a correspondence with the sensor.
[0028] For example, the bit stream may include third information common to the plurality of 3D points, the third information indicating a number of 3D points per sensor.
[0029] For example, the point cloud data may include position information and one or more pieces of attribute information for each of the plurality of three-dimensional points, and the first information may indicate a sensor corresponding to each of the one or more pieces of attribute information.
[0030] For example, the point cloud data may be generated by synthesizing first point cloud data based on a first sensor and second point cloud data based on a second sensor, and in the synthesis, if the first point cloud data and the second point cloud data contain attribute information of a first point having identical position information, attribute information based on the attribute information of the first point contained in the first point cloud data and the attribute information of the first point contained in the second point cloud data is generated as attribute information after synthesis, and the first information may indicate that the information of the first point contained in the point cloud data is based on the first sensor and the second sensor.
[0031] For example, the encoded data may be generated by dividing the point cloud data into first point cloud data based on a first sensor and second point cloud data based on the second sensor, and encoding the first point cloud data and the second point cloud data, the bit stream including first control information for the first point cloud data and second control information for the second point cloud data, and the first information may include: (1) information included in the first control information and indicating a sensor corresponding to the first point cloud data, and (2) information included in the second control information and indicating a sensor corresponding to the second point cloud data.
[0032] For example, the encoded data may be generated by dividing the point cloud data into first point cloud data based on a first sensor and second point cloud data based on the second sensor, and encoding the first point cloud data and the second point cloud data, the bit stream may include control information common to the first point cloud data and the second point cloud data, and the first information may be included in the control information.
[0033] Moreover, a three-dimensional data encoding device according to one embodiment of the present disclosure includes a processor and a memory, wherein the processor uses the memory to generate encoded data by encoding point cloud data, and generates a bit stream including the encoded data, and the bit stream includes first information indicating a sensor corresponding to the point cloud data.
[0034] This allows the three-dimensional data decoding device to extract point cloud data obtained by a desired sensor using the first information, and therefore the three-dimensional data encoding device can generate a bit stream from which the point cloud data can be appropriately extracted.
[0035] In addition, a three-dimensional data decoding device according to one embodiment of the present disclosure includes a processor and a memory, and the processor uses the memory to decode a bit stream that includes encoded data generated by encoding point cloud data and includes first information indicating a sensor corresponding to the point cloud data, thereby obtaining the first information and the point cloud data.
[0036] According to this, the three-dimensional data decoding device can obtain point cloud data by decoding a bit stream from which point cloud data can be appropriately extracted.
[0037] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.
[0038] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement and connection forms of the components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.
[0039] (Embodiment 1) First, the configuration of a three-dimensional data (point cloud data) encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.
[0040] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data, which is three-dimensional data. The three-dimensional data encoding system 4601 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. The three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 4601.
[0041] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a presentation unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.
[0042] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603, and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data from the sensor information, and outputs the point cloud data to the encoding unit 4613.
[0043] The presentation unit 4612 presents the sensor information or the point cloud data to the user. For example, the presentation unit 4612 displays information or an image based on the sensor information or the point cloud data.
[0044] The encoding unit 4613 encodes (compresses) the point cloud data, and outputs the obtained encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, sensor information.
[0045] The multiplexing unit 4614 generates multiplexed data by multiplexing the coded data, control information, and additional information input from the coding unit 4613. The format of the multiplexed data is, for example, a file format for storage, or a packet format for transmission.
[0046] The input / output unit 4615 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 4616 (or an application execution unit) controls each processing unit. In other words, the control unit 4616 controls encoding, multiplexing, etc.
[0047] The sensor information may be input to the encoding unit 4613 or the multiplexing unit 4614. Furthermore, the input / output unit 4615 may output the point cloud data or the encoded data directly to the outside.
[0048] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604.
[0049] The three-dimensional data decoding system 4602 generates point cloud data, which is three-dimensional data, by decoding the encoded data or multiplexed data. The three-dimensional data decoding system 4602 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. The three-dimensional data decoding device may also include some of the multiple processing units included in the three-dimensional data decoding system 4602.
[0050] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621 , an input / output unit 4622 , a demultiplexing unit 4623 , a decoding unit 4624 , a presentation unit 4625 , a user interface 4626 , and a control unit 4627 .
[0051] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603 .
[0052] The input / output unit 4622 acquires a transmission signal, decodes multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 4623.
[0053] The demultiplexer 4623 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoder 4624.
[0054] The decoding unit 4624 reconstructs the point cloud data by decoding the encoded data.
[0055] The presentation unit 4625 presents the point cloud data to the user. For example, the presentation unit 4625 displays information or an image based on the point cloud data. The user interface 4626 acquires an instruction based on a user's operation. The control unit 4627 (or the application execution unit) controls each processing unit. That is, the control unit 4627 controls demultiplexing, decoding, presentation, etc.
[0056] The input / output unit 4622 may obtain the point cloud data or the encoded data directly from the outside. The presentation unit 4625 may obtain additional information such as sensor information and present information based on the additional information. The presentation unit 4625 may present information based on a user instruction obtained by the user interface 4626.
[0057] The sensor terminal 4603 generates sensor information obtained by a sensor. The sensor terminal 4603 is a terminal equipped with a sensor or a camera, and examples of the sensor terminal include a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, and a camera.
[0058] The sensor information that can be acquired by the sensor terminal 4603 is, for example, (1) the distance (location information) between the sensor terminal 4603 and an object, the color, or the reflectance of the object obtained from a LIDAR, millimeter wave radar, or an infrared sensor, and (2) the distance (location information) between a camera and an object obtained from a plurality of monocular camera images or stereo camera images, the color, or the reflectance of the object. The sensor information may also include the attitude, direction, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, or the acquisition time of the sensor information. The sensor information may also include temperature, air pressure, humidity, magnetism, etc.
[0059] The external connection unit 4604 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.
[0060] Next, the point cloud data will be described. Fig. 2 is a diagram showing the configuration of the point cloud data. Fig. 3 is a diagram showing an example of the configuration of a data file in which information on the point cloud data is written.
[0061] Point cloud data includes data on multiple points. The data on each point includes position information (three-dimensional coordinates) and attribute information for that position information. A collection of multiple points is called a point cloud. For example, a point cloud may represent the three-dimensional shape of an object.
[0062] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.
[0063] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.
[0064] The configuration example of the data file shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.
[0065] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.
[0066] Next, the types of point cloud data will be described. Fig. 4 is a diagram showing the types of point cloud data. As shown in Fig. 4, point cloud data includes static objects and dynamic objects.
[0067] A static object is 3D point cloud data at any time (a certain time). A dynamic object is 3D point cloud data that changes over time. Hereinafter, 3D point cloud data at a certain time will be called a PCC frame, or a frame.
[0068] The object may be a point cloud whose area is restricted to a certain extent, such as ordinary video data, or a large-scale point cloud whose area is not restricted, such as map information.
[0069] Also, there may be point cloud data of various densities, and there may be sparse point cloud data and dense point cloud data.
[0070] The details of each processing unit will be described below. The sensor information is acquired by various methods such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The point cloud data generation unit 4618 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as point cloud data, and adds attribute information for the position information to the position information.
[0071] The point cloud data generating unit 4618 may process the point cloud data when generating the position information or adding the attribute information. For example, the point cloud data generating unit 4618 may reduce the amount of data by deleting point clouds with overlapping positions. In addition, the point cloud data generating unit 4618 may convert the position information (position shift, rotation, normalization, etc.) or render the attribute information.
[0072] In FIG. 1, the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but it may be provided independently outside the three-dimensional data encoding system 4601.
[0073] The encoding unit 4613 generates encoded data by encoding the point cloud data based on a predefined encoding method. There are two main types of encoding methods: the first is an encoding method using position information, and this encoding method will be referred to as the first encoding method hereinafter; the second is an encoding method using a video codec, and this encoding method will be referred to as the second encoding method hereinafter.
[0074] The decoding unit 4624 decodes the encoded data based on a predefined encoding method to decode the point group data.
[0075] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 4614 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 4614 may further multiplex attribute information related to sensor information or point cloud data.
[0076] Multiplexing methods or file formats include ISOBMFF, MPEG-DASH, which is an ISOBMFF-based transmission method, MMT, MPEG-2 TS Systems, and RMP.
[0077] The demultiplexer 4623 extracts the PCC encoded data, other media, time information, and the like from the multiplexed data.
[0078] The input / output unit 4615 transmits the multiplexed data using a method suited to a transmission medium or a storage medium, such as broadcasting or communication. The input / output unit 4615 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.
[0079] The communication protocol used may be http, ftp, TCP, UDP, IP, etc. A PULL type communication method or a PUSH type communication method may be used.
[0080] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, 3G / 4G / 5G, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter wave, etc., as defined by IEEE of 3GPP (registered trademark) are used.
[0081] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.
[0082] Fig. 5 is a diagram showing a configuration of a first encoding unit 4630 which is an example of the encoding unit 4613 that performs encoding using the first encoding method. Fig. 6 is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. The first encoding unit 4630 includes a position information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.
[0083] The first encoding unit 4630 has a feature of performing encoding in consideration of a three-dimensional structure. Also, the first encoding unit 4630 has a feature that the attribute information encoding unit 4632 performs encoding using information obtained from the position information encoding unit 4631. The first encoding method is also called GPCC (Geometry based PCC).
[0084] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData). The position information is input to a position information encoding unit 4631, the attribute information is input to an attribute information encoding unit 4632, and the additional information is input to an additional information encoding unit 4633.
[0085] The position information encoding unit 4631 generates encoded position information (Compressed Geometry) that is encoded data by encoding the position information. For example, the position information encoding unit 4631 encodes the position information using an N-ary tree structure such as an octet tree. Specifically, in the octet tree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. In addition, the node that includes the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined hierarchy or node is equal to or less than a threshold value.
[0086] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute) that is encoded data by encoding using the configuration information generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 determines a reference point (reference node) to be referenced in encoding a target point (target node) to be processed based on the octree structure generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 refers to a node, among peripheral nodes or adjacent nodes, whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.
[0087] Furthermore, the encoding process of the attribute information may include at least one of a quantization process, a prediction process, and an arithmetic coding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using a state of the reference node (e.g., occupancy information indicating whether or not a point group is included in the reference node) to determine a parameter of the encoding. For example, the parameter of the encoding is a quantization parameter in the quantization process, or a context in the arithmetic coding.
[0088] The additional information encoding unit 4633 generates encoded additional information (Compressed MetaData) that is encoded data by encoding compressible data from the additional information.
[0089] The multiplexing unit 4634 multiplexes the encoding position information, the encoding attribute information, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).
[0090] Next, a first decoding unit 4640 which is an example of the decoding unit 4624 that performs decoding of the first encoding method will be described. FIG. 7 is a diagram showing a configuration of the first decoding unit 4640. FIG. 8 is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point group data by decoding, by the first encoding method, encoded data (encoded stream) encoded by the first encoding method. The first decoding unit 4640 includes a demultiplexing unit 4641, a position information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.
[0091] A coded stream (compressed stream), which is coded data, is input to the first decoding unit 4640 from a processing unit in a system layer (not shown).
[0092] The demultiplexer 4641 separates the encoded position information (Compressed Geometry), the encoded attribute information (Compressed Attribute), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.
[0093] The position information decoding unit 4642 generates position information by decoding the encoded position information. For example, the position information decoding unit 4642 restores the position information of a point group represented by three-dimensional coordinates from the encoded position information represented by an N-ary tree structure such as an octree.
[0094] The attribute information decoding unit 4643 decodes the encoded attribute information based on the configuration information generated by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 determines a reference point (reference node) to be referenced in decoding a target point (target node) to be processed based on the octree structure obtained by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 refers to a node, among peripheral nodes or adjacent nodes, whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.
[0095] In addition, the decoding process of the attribute information may include at least one of an inverse quantization process, a prediction process, and an arithmetic decoding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using a state of the reference node (e.g., occupancy information indicating whether or not a point group is included in the reference node) to determine a parameter of the decoding. For example, the parameter of the decoding is a quantization parameter in the inverse quantization process, or a context in the arithmetic decoding.
[0096] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. The first decoding unit 4640 uses the additional information required for decoding the position information and attribute information during decoding, and outputs the additional information required for an application to the outside.
[0097] Next, a second encoding unit 4650, which is an example of the encoding unit 4613 that performs encoding by the second encoding method, will be described. Fig. 9 is a diagram showing a configuration of the second encoding unit 4650. Fig. 10 is a block diagram of the second encoding unit 4650.
[0098] The second encoding unit 4650 generates encoded data (encoded stream) by encoding the point cloud data by a second encoding method. The second encoding unit 4650 includes an additional information generating unit 4651, a position image generating unit 4652, an attribute image generating unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.
[0099] The second encoding unit 4650 has a feature of generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position image and attribute image using an existing video encoding method. The second encoding method is also called VPCC (Video based PCC).
[0100] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).
[0101] The additional information generating unit 4651 generates map information of a plurality of two-dimensional images by projecting a three-dimensional structure onto the two-dimensional images.
[0102] The position image generating unit 4652 generates a position image (Geometry Image) based on the position information and the map information generated by the additional information generating unit 4651. This position image is, for example, a distance image in which distance (Depth) is indicated as a pixel value. Note that this distance image may be an image in which a plurality of point clouds are viewed from one viewpoint (an image in which a plurality of point clouds are projected onto one two-dimensional plane), a plurality of images in which a plurality of point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.
[0103] The attribute image generating unit 4653 generates an attribute image based on the attribute information and the map information generated by the additional information generating unit 4651. This attribute image is, for example, an image in which the attribute information (for example, color (RGB)) is indicated as pixel values. Note that this image may be an image in which multiple point clouds are viewed from one viewpoint (an image in which multiple point clouds are projected onto one two-dimensional plane), or multiple images in which multiple point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.
[0104] The video encoding unit 4654 generates an encoded position image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image), which are encoded data, by encoding the position image and the attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC, HEVC, or the like.
[0105] The additional information encoding unit 4655 generates encoded additional information (Compressed MetaData) by encoding the additional information and map information included in the point cloud data.
[0106] The multiplexing unit 4656 multiplexes the encoding position image, the encoding attribute image, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).
[0107] Next, a second decoding unit 4660 which is an example of the decoding unit 4624 that performs decoding of the second encoding method will be described. FIG. 11 is a diagram showing a configuration of the second decoding unit 4660. FIG. 12 is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point group data by decoding, by the second encoding method, encoded data (encoded stream) encoded by the second encoding method. The second decoding unit 4660 includes a demultiplexing unit 4661, a video decoding unit 4662, an additional information decoding unit 4663, a position information generating unit 4664, and an attribute information generating unit 4665.
[0108] A compressed stream, which is encoded data, is input to the second decoding unit 4660 from a processing unit in a system layer (not shown).
[0109] The demultiplexer 4661 separates the encoded position image (Compressed Geometry Image), the encoded attribute image (Compressed Attribute Image), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.
[0110] The video decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC or HEVC.
[0111] The additional information decoding unit 4663 decodes the encoded additional information to generate additional information including map information and the like.
[0112] The position information generating unit 4664 generates position information using the position image and map information. The attribute information generating unit 4665 generates attribute information using the attribute image and map information.
[0113] The second decoding unit 4660 uses the additional information necessary for decoding during decoding, and outputs the additional information necessary for an application to the outside.
[0114] The PCC encoding method will be described below. Fig. 13 is a diagram showing a protocol stack related to PCC encoded data. Fig. 13 shows an example in which other media data such as video (e.g., HEVC) or audio, or sensor information is multiplexed with PCC encoded data and transmitted or stored.
[0115] The multiplexing method and file format have the function of multiplexing various encoded data and transmitting or storing them. In order to transmit or store the encoded data, the encoded data is converted into the format of the multiplexing method. For example, HEVC specifies a technology for storing encoded data in a data structure called a NAL unit and storing the NAL unit in ISOBMFF.
[0116] A similar configuration is assumed for PCC. The sensor information may be PCC-coded together with the point cloud data, or may be coded using a different coding method, or may be stored directly in the multi-layer without being coded, or these may be combined. The different coding method may be, in particular, a different three-dimensional coding method, or a coding method that codes the data obtained by converting the point cloud data into two-dimensional or one-dimensional data.
[0117] An example of a configuration for generating point cloud data from a sensor signal (also called sensor information) will be described below. Fig. 14 to Fig. 16 are diagrams illustrating an example of a configuration of a point cloud data generating device that generates point cloud data from a sensor signal.
[0118] The point cloud data generating device shown in Fig. 14 generates point cloud data from a sensor signal obtained from one sensing device 7301. The point cloud data generating device shown in Fig. 14 includes the sensing device 7301, a sensor information input unit 7302, and a point cloud data generating unit 7303. The sensor information input unit 7302 acquires the sensor signal obtained by the sensing device 7301. The point cloud data generating unit 7303 generates point cloud data from the sensor signal acquired by the sensor information input unit 7302. The generated point cloud data is output to, for example, a subsequent point cloud data encoding unit (not shown).
[0119] As shown in FIG. 15, point cloud data may be generated based on sensor signals obtained from two or more sensing devices. The point cloud data generating apparatus shown in FIG. 15 includes sensing devices 7301A and 7301B, sensor information input units 7302A and 7302B, and a point cloud data generating unit 7303A. The sensor information input unit 7302A acquires a first sensor signal acquired by the sensing device 7301A. The sensor information input unit 7302B acquires a second sensor signal acquired by the sensing device 7301B. The point cloud data generating unit 7303A generates point cloud data from the two sensor signals acquired by the sensor information input units 7302A and 7302B. The generated point cloud data is output to, for example, a subsequent point cloud data encoding unit (not shown).
[0120] The point cloud data generating apparatus shown in Fig. 16 includes a sensing device 7301C, a sensor information input unit 7302C, and a point cloud data generating unit 7303C. The sensing device 7301C generates a sensor signal by merging two pieces of information sensed using two or more sensing methods in a predetermined manner. The sensing device 7301C includes sensing units 7304A and 7304B, and a merging unit 7305.
[0121] The sensing unit 7304A generates a first sensor signal by a first sensing method. The sensing unit 7304B generates a second sensor signal by a second sensing method. The merging unit 7305 merges the first sensor signal and the second sensor signal, and outputs the generated sensor signal to the sensor information input unit 7302C.
[0122] The merging unit 7305 may select one of the first sensor signal and the second sensor signal based on a predetermined condition and output the selected sensor signal. When merging two sensor signals, the merging unit 7305 may change a weighting coefficient used for merging.
[0123] For example, the merge unit 7305 may determine which sensor signal to select based on the acquired sensor signal, or based on another sensor signal.
[0124] For example, the first sensing method and the second sensing method may have different sensor parameters, different sensing frequencies or different mechanisms, etc. The center signal may include information indicating the sensing method or the parameters used during sensing.
[0125] When switching between a plurality of sensing methods, the merging unit 7305 may include, in the sensor signal, information indicating which sensing method was used or data on a criterion for switching. When merging sensor signals, the merging unit 7305 may include, in the sensor signal, information for identifying the merged sensing method, data on a criterion for merging, or a merging coefficient.
[0126] Furthermore, the sensing device 7301C may output a plurality of sensor signals. Furthermore, the sensing device 7301C may output, as the plurality of sensor signals, an absolute value of a first sensor signal and a difference value between the first sensor signal and a second sensor signal.
[0127] The sensor signal may also include information indicating the relationship between the first sensing method and the second sensing method. For example, the sensor signal may include an absolute value or a relative value of reference position information between the first sensing method and the second sensing method, or may include information indicating the acquisition time of the sensor signal, reference time information, or the angle of the sensor. By including such information in the sensor signal, it becomes possible to correct or combine the relationship between the two sensor signals based on such information in subsequent processing.
[0128] The sensor information input unit 7302C acquires a sensor signal obtained by the sensing device 7301C. The point cloud data generation unit 7303C generates point cloud data from the sensor signal acquired by the sensor information input unit 7302C. The generated point cloud data is output to, for example, a subsequent point cloud data encoding unit (not shown).
[0129] In this way, the point cloud data generating device generates point cloud data based on any one or more of the various sensor signals described above. Note that the point cloud data generating device may correct position information or attribute information of points in the process of generating the point cloud data.
[0130] The point cloud data generating device may have any of the configurations shown in Figures 14 to 16, or may have a combination of two or more of these configurations. The point cloud data generating device may use a fixed method, or may adaptively change the method to be used depending on, for example, the purpose of sensing or the use case.
[0131] Next, a configuration example of a point cloud data encoding system according to this embodiment will be described. Fig. 17 is a diagram showing a configuration example of a point cloud data encoding system according to this embodiment. The point cloud data encoding system shown in Fig. 17 includes a first device 7310 and a second device 7320.
[0132] The first device 7310 includes a sensing unit 7311 and an output I / F (interface) 7312. The second device 7320 includes a sensing unit 7321, an output I / F 7322, an input I / F 7323, and a processing unit 7324. The processing unit 7324 includes a point cloud data generation unit 7325 and an encoding unit 7326.
[0133] The sensing unit 7311 or 7321 may be included in the same hardware or device as the processing unit 7324 constituted by a CPU or the like, or may be included in different hardware or device.
[0134] The sensing unit 7321 is included in the same device (second device 7320) as the processing unit 7324. In this case, an output signal (called RAW data) of the sensing unit 7321 is input to the point cloud data generation unit 7325 as it is.
[0135] The sensing unit 7311 is included in a device (first device 7310) different from the processing unit 7324. In this case, the RAW data output from the sensing unit 7311 is converted into an input / output format (external output format) in the output I / F 7312, and the formatted signal is input to the second device 7320. The input I / F 7323 included in the second device 7320 converts the formatted signal into RAW data, and outputs the obtained RAW data to the point cloud data generation unit 7325. The output I / F 7312 and the input I / F 7323 have, for example, the functions of the multiplexing unit 4614 and the input / output unit 4615 shown in FIG. 1.
[0136] In addition, an output signal (RAW data) from a sensing unit 7321, which is included in the same device as the processing unit 7324, may be converted into an input / output format by an output I / F 7322, the formatted signal may be converted into RAW data by an input I / F 7323, and the obtained RAW data may be input to a point cloud data generation unit 7325.
[0137] Furthermore, when multiple sensor signals are input, for example, when sensor signals input from other devices and sensor signals input from the same device are mixed, these sensor signals may be converted into the same format. Furthermore, during conversion, each signal may be assigned an identifier that can identify the signal. For example, when transmission is performed using UDP (User Datagram Protocol), each signal may be identified by IP (Internet Protocol) or UDP source address or source port number. This makes it possible to unify the format input to the point cloud data generation unit 7325, making it easier to control the signals.
[0138] The point cloud data generating unit 7325 generates point cloud data using the input RAW data. The encoding unit 7326 encodes the generated point cloud data.
[0139] Next, a configuration example of a three-dimensional data multiplexing device (three-dimensional data multiplexing system) according to this embodiment will be described. Fig. 18 is a diagram showing a configuration example of a three-dimensional data multiplexing device according to this embodiment. The three-dimensional data multiplexing device generates an output signal by encoding and multiplexing various sensor signals, and stores or transmits the generated output signal.
[0140] As shown in Fig. 18, the three-dimensional data multiplexing device includes sensing units 7331A, 7331B, and 7331C, sensor information input units 7332A, 7332B, and 7332C, point cloud data generating units 7333A and 7333B, encoding units 7334A and 7334B, a synchronization unit 7335, and a multiplexing unit 7336. Here, an example in which three sensing units are used is shown, but the number of sensing units is not limited to this. In addition, any combination of the following processing methods can be used as a processing method for sensor signals from each sensing unit.
[0141] The sensor information input unit 7332A acquires a first sensor signal generated by sensing in the sensing unit 7331 A. The sensor information input unit 7332B acquires a second sensor signal generated by sensing in the sensing unit 7331 B. The sensor information input unit 7332C acquires a third sensor signal generated by sensing in the sensing unit 7331 C.
[0142] The point cloud data generating unit 7333A generates first point cloud data from the first sensor signal. The point cloud data generating unit 7333B generates second point cloud data from the second sensor signal. At this time, the number of points, the range of points, and attribute information in the generated point cloud data may differ due to differences in the sensing methods used by the sensing units 7331A and 7331B (e.g., direction, range, obtainable attributes, frequency, resolution, method, or means).
[0143] The encoding unit 7334A generates first encoded data by encoding the first point cloud data. The encoding unit 7334B generates second encoded data by encoding the second point cloud data. For example, the encoding unit 7334A and the encoding unit 7334B apply different encoding methods. For example, the encoding unit 7334A may use a first encoding method, and the encoding unit 7334B may use a second encoding method different from the first encoding method. Note that the encoding unit 7334A and the encoding unit 7334B may use the same encoding method.
[0144] The encoding units 7334A and 7334B may compress position information or attribute information of points in the point cloud data using entropy coding, etc. Furthermore, the encoding units 7334A and 7334B may store sensor signals, position information or angle information of the sensors, time information, or the like as metadata.
[0145] The encoding units 7334A and 7334B use an encoding method suitable for the point cloud data. For example, the first encoding method is an encoding method that can be expected to have a high encoding rate for map information or still content, and the second encoding method is an encoding method that can be expected to have a high encoding rate for content such as AR or VR. In this case, the encoding units 7334A and 7334B may use an encoding method suitable for the content.
[0146] Alternatively, for example, the first encoding method is an encoding method that can be expected to achieve a high encoding rate for a point cloud based on information sensed by a sensing unit such as a beam LiDAR, and the second encoding method is an encoding method that can be expected to achieve a high encoding rate for a point cloud based on information sensed by a sensing unit such as a FLASH LiDAR. In this case, the encoding units 7334A and 7334B may use encoding methods suitable for the sensing units.
[0147] Furthermore, instead of changing the encoding method, the encoding units 7334A and 7334B may use an encoding tool or encoding-related parameters suitable for the content or the sensing unit in the same encoding method.
[0148] The generated first coded data and second coded data are input to the multiplexing unit 7336. For example, a third sensor signal sensed by the sensing unit 7331C is data that does not need to be coded. In this case, point cloud data is not generated and coded, and the third sensor signal is directly input to the multiplexing unit 7336. Note that coding may not be performed for the purpose of low-latency transmission.
[0149] The synchronization unit 7335 has a function for synchronizing a plurality of sensing units. For example, the synchronization unit 7335 uses sensing time information, time stamp information, angle information, and the like as information related to synchronization. Such information related to synchronization may be multiplexed into an output signal as a synchronization signal that is common information. Alternatively, such information related to synchronization may be included in each sensor signal.
[0150] The multiplexing unit 7336 generates an output signal by multiplexing one or more of the coded data, the metadata, the raw data of the sensor signal, and the synchronization signal. The multiplexing unit 7336 also stores, in the output signal, information for identifying each data and information indicating the correspondence between each data.
[0151] FIG. 19 is a diagram showing a specific example of a three-dimensional data multiplexing device. As shown in FIG. 19, a beam LiDAR is used as the sensing unit 7331A, and a FLASH LiDAR is used as the sensing unit 7331B. The range and distance of the point cloud, as well as the resolution, vary depending on the characteristics of the LiDAR. FIG. 20 is a diagram showing an example of the sensor range of the beam LiDAR and the FLASH LiDAR. For example, the beam LiDAR detects all directions around the vehicle (sensor), and the FLASH LiDAR detects a range in one direction (for example, forward) of the vehicle.
[0152] The point cloud data generating unit 7333A generates first point cloud data based on distance information and reflectance information for the beam irradiation angle acquired from the beam LiDAR. The point cloud data generating unit 7333B generates second point cloud data based on two-dimensional distance information and reflectance acquired from the FLASH LiDAR. The point cloud data generating units 7333A and 7333B may further use two-dimensional color information acquired by the camera to generate point cloud data having both color information and reflectance.
[0153] In addition, an in-vehicle sensor that is a position information sensor such as a three-axis gyro sensor, a three-axis acceleration sensor, or a GPS is used as the sensing unit 7331C. These sensor information are sensor information that represent the overall state of the vehicle, and can also be said to be common information related to the first sensor signal and the second sensor signal. These common sensor information may be encoded and multiplexed, or may be multiplexed without being encoded. In addition, these pieces of information may be stored in the first sensor signal or the second sensor signal as additional information common to the point cloud data and encoded. Alternatively, the common sensor information may be stored in one of the first sensor signal and the second sensor signal. In this case, information indicating which sensor signal the common sensor information is stored in may be indicated, for example, in the other sensor signal or the synchronization signal.
[0154] In addition, as information regarding the time of acquiring the sensor, a time stamp based on reference time information such as NTP (Network Time Protocol) or PTP (Precision Time Protocol) is added to the first point cloud data based on the beam LiDAR and the second point cloud data based on the FLASH LiDAR. The time stamps of the sensors are synchronized with a common reference time and are encoded by the encoding units 7334A and 7334B.
[0155] Furthermore, reference time information indicating a common reference time may be multiplexed as a synchronization signal. The reference time information does not have to be multiplexed. The three-dimensional data demultiplexing device (three-dimensional data decoding device) acquires each timestamp from the encoded data of the multiple sensor signals. Since the timestamps are synchronized to the common reference time, the three-dimensional data demultiplexing device can synchronize between the multiple sensors by operating the decoded data of the multiple sensor signals based on each timestamp.
[0156] Note that time information corresponding to each beam LiDAR and FLASH LiDAR may be set. Also, a triaxial sensor may be provided for each beam LiDAR and FLASH LiDAR. In this case, a common time such as Internet time is used as the NTP for each. Also, each triaxial sensor is calibrated in advance, and multiple triaxial sensors that are synchronized in advance are used.
[0157] Fig. 21 is a diagram showing another example of the configuration of a three-dimensional data multiplexing device. As shown in Fig. 21, the three-dimensional data multiplexing device includes sensing units 7341A, 7341B, and 7341C, input / output units 7342A, 7342B, and 7342C, a point cloud data generating unit 7343, encoding units 7344A and 7344B, a synchronization unit 7345, and a multiplexing unit 7346.
[0158] The input / output unit 7342A acquires a first sensor signal generated by sensing in the sensing unit 7341A. The input / output unit 7342B acquires a second sensor signal generated by sensing in the sensing unit 7341B. The input / output unit 7342C acquires a third sensor signal generated by sensing in the sensing unit 7341C. Note that the input / output units 7342A, 7342B, and 7342C may have a memory that stores the acquired sensor signals.
[0159] The point cloud data generating unit 7343 generates first point cloud data from the first sensor signal. The encoding unit 7344A generates first encoded data by encoding the first point cloud data. The encoding unit 7344B generates second encoded data by encoding the second sensor information.
[0160] The synchronization unit 7345 has a function for synchronizing a plurality of sensing units. The multiplexing unit 7346 generates an output signal by multiplexing one or more of the coded data, the metadata, the raw data of the sensor signal, and the synchronization signal.
[0161] Thus, in the configuration shown in FIG. 21, point cloud data is not generated from the sensor signal (RAW data) obtained by the sensing unit 7341B, and the sensor signal is encoded as RAW data. For example, when the second sensor signal is two-dimensional information obtained by a CMOS sensor such as a FLASH LiDAR or a camera, the encoding unit 7344B encodes the second sensor signal using a video codec such as AVC or HEVC. This makes it possible to realize encoding with high encoding efficiency. In addition, by utilizing existing codecs, it is possible to build a low-cost system.
[0162] In this way, the three-dimensional data multiplexing device uses both a means for encoding point cloud data after conversion and a means for encoding RAW data without converting it to point cloud data, depending on the sensing unit, and multiplexes each of the encoded data.
[0163] Next, an example of a method for generating an output signal of a predetermined file format by multiplexing will be described. In the following, an example will be described in which the predetermined file format is ISOBMFF (ISO based media file format). Note that the file format is not limited to ISOBMFF, and other file formats may be used.
[0164] ISOBMFF is a file format standard defined in ISO / IEC 14496-12. ISOBMFF specifies a format that can store multiplexed data of various media such as video, audio, and text, and is a media-independent standard.
[0165] The method of storing each type of media in ISOBMFF is specified separately. For example, the method of storing AVC video and HEVC video is specified in ISO / IEC14496-15.
[0166] On the other hand, a method is required to store the encoded data of the data acquired from multiple sensor information (sensor signals) in ISOBMFF. Fig. 22 is a diagram showing a protocol for encoding multiple pieces of sensor information using various encoding methods and storing them in ISOBMFF.
[0167] Data1 to Data5 are sensor data (sensor signals) acquired from various types of sensors, such as RAW data. Data1 and Data2 are converted into a 3D point cloud format and encoded using the encoding method Codec1 or Codec2 for the 3D point cloud format. Data3 and Data4 are converted into a 2D data format such as an image, and encoded using the encoding method Codec3 or Codec4 for the 2D data format.
[0168] Each piece of encoded data is converted into an NAL unit by a predetermined method and stored in the ISOBMFF. The NAL unit may be in a common format for the 3D format and the 2D format, or may be in a different format. The NAL units of the different encoding methods may be in a common format, or may be in a different format. The format of the sensor data may be a 1D format or other format in addition to the 3D and 2D formats listed here.
[0169] Data5 is a case where sensor data acquired from a sensor is stored directly in ISOBMFF without being encoded.
[0170] By providing a format for storing any combination of these data in an integrated manner, it becomes easier to manage data in a system that handles multiple sensors, and various functions can be realized.
[0171] Next, the structure of ISOBMFF will be described. The three-dimensional data multiplexing device stores a plurality of sensor data in ISOBMFF. Fig. 23 is a diagram showing a structure example of input data to be multiplexed. Fig. 24 is a diagram showing a structure example of an NAL unit. Fig. 25 is a diagram showing a structure example of ISOBMFF. Fig. 26 is a diagram showing a structure example of moov and mdat.
[0172] The coded data included in the input data is roughly divided into coded data (Data) and metadata (Meta). Metadata includes metadata indicated in the header of each coded data and metadata stored in an independent NAL unit as a parameter set. Metadata may also be included in coded data. The 3D data multiplexing device stores these NAL units and RAW data for each of the different codecs in one ISOBMFF.
[0173] ISOBMFF is composed of a box structure. The boxes of ISOBMFF are "moov" and "meta" that mainly store metadata, and "mdat" that stores data.
[0174] The encoded data and raw data are stored in sample units in "mdat" in ISOBMFF. In addition, metadata in the input data is stored in a specific format in "trak" of "moov" in ISOMBFF for each encoded data. Metadata and synchronization information included in the encoded data are also stored in "moov".
[0175] Information for retrieving data from "mdat" (address information (offset information) of the data from the beginning of the file, data size, etc.) is stored in the metadata for each encoded data. In addition, "ftyp" indicates the file type of the following data, etc.
[0176] The formats and box names may be other than those listed here as long as they have the same functions.
[0177] In addition, in a use case such as real-time communication, units obtained by dividing a box such as "moov" and "mdat" may be transmitted at different times. Furthermore, data of the divided units may be interleaved.
[0178] The three-dimensional data multiplexing device defines a box indicating configuration information (hereinafter simply referred to as configuration information) and stores identification information of multiple data included in a file in the configuration information. The three-dimensional data multiplexing device also stores identification information that allows access to metadata of each data in the configuration information.
[0179] Fig. 27 is a diagram illustrating an example of the configuration information, and Fig. 28 is a diagram illustrating an example of the syntax of the configuration information.
[0180] The configuration information indicates information about the contents and components that make up the ISOBMFF file, sensor information when the original data of the components was acquired, format information, and the encoding method, etc.
[0181] 27, the configuration information includes overall configuration information and configuration information for each piece of encoded data. The multiple pieces of configuration information may be the same data structure or box, or may be different data structures or boxes.
[0182] The type in the mp4 box indicates that it is a configuration information box by 4CC such as "msuc". The overall configuration information (data()) indicates the configuration of multiple data with different encoding methods. data() includes num_of_data, data_type, and data_configuration.
[0183] num_of_data indicates the number of encoded data and RAW data that constitute the file. data_type indicates identification information for each data. In other words, data_type indicates the type of multiple data.
[0184] Specifically, data_type indicates whether the data is point cloud data or a sensor signal (e.g., RAW data). Data_type may also indicate whether the data is encoded. Data_type may also indicate the encoding method (encoding scheme) used to encode the encoded data. The encoding method is, for example, GPPC or VPPC. The encoding method may be Codec1 to 4 shown in FIG. 22. Data_type may also indicate information for identifying configuration information.
[0185] Furthermore, data_type may indicate the type of original data of the point cloud data. The type of original data is the type of sensor that generated the sensor signal when the original data is a sensor signal (e.g., 2D sensor or 3D sensor, etc.). Furthermore, data_type may include information indicating the data format of the sensor signal (e.g., 1D information, 2D information, 3D information, etc.).
[0186] For example, data_type=0 indicates PCC Codec 1, data_type=1 indicates PCC Codec 2, data_type=2 indicates Video Codec 3, and data_type=4 indicates 3D axis sensor RAW data. data_configuration indicates configuration information for each data.
[0187] The data_configuration() is configuration information for each piece of encoded data, and includes num_of_component, component_type, and component_id.
[0188] num_of_component indicates the number of components in the encoded data. component_type indicates the type of component. For example, in the case of PCC encoding, component_type indicates whether the component is geometry, attribute, or metadata.
[0189] The component_id indicates a unique identifier for linking the component with other metadata and data.
[0190] In addition, the encoding method may be an encoding method used for audio, text, applications, 360-degree images, etc., other than the video codec and PCC codec. The data may be processed data such as mesh or CAD. The encoding method may be the same codec, or may be an encoding method with a different profile, level, or tool, and any encoding method can be handled in an integrated manner.
[0191] In this way, by multiplexing the data necessary for utilizing the point cloud data decoded in the three-dimensional data demultiplexing device in an application into one file, file management and synchronization management handled by the application can be simplified.
[0192] 29 is a diagram showing an example of the structure of the data box "mdat." Each piece of coded data or RAW data is stored individually in a sample, which is the smallest unit of the data box.
[0193] Furthermore, synchronization information such as a timestamp for each piece of encoded data included in a file is set based on overall synchronization information such as a common reference time, etc. Furthermore, each piece of synchronization information is information that is synchronized.
[0194] Also, for example, the reference time, time resolution, and time interval in the timestamps of multiple encoded data may be aligned to make the synchronization information common to multiple encoded data. In this case, the synchronization information may be stored in one or more of the synchronization information for each encoded data and the common synchronization information. In this case, the metadata includes at least one of information indicating where the common time information is stored and information indicating that the synchronization information is common.
[0195] Furthermore, when the encoded data are synchronized with each other, the three-dimensional data multiplexing device may store the synchronized encoded data as one sample. On the other hand, when at least one of the reference time, the time resolution, and the time interval is not aligned among the encoded data, the three-dimensional data multiplexing device may derive difference information indicating the difference in the time stamps between the encoded data separately, and store the derived difference information in the output signal. Furthermore, the three-dimensional data multiplexing device may store a flag indicating whether or not the data are synchronized in the output signal.
[0196] The three-dimensional data demultiplexing device synchronizes between the encoded data by processing each sample at the time indicated by the timestamp indicated in the metadata, using the synchronization information of each encoded data and the overall synchronization information.
[0197] An example of application processing will be described below. Fig. 30 is a flowchart showing an example of application processing. When an application operation is started, the three-dimensional data demultiplexing device acquires an ISOBMFF file including point cloud data and a plurality of encoded data (S7301). For example, the three-dimensional data demultiplexing device may acquire the ISOBMFF file through communication, or may read it from accumulated data.
[0198] Next, the three-dimensional data demultiplexing device analyzes the overall configuration information in the ISOBMFF file and identifies data to be used for the application (S7302). For example, the three-dimensional data demultiplexing device acquires data to be used for processing and does not acquire data not to be used for processing.
[0199] Next, the three-dimensional data demultiplexing device extracts one or more pieces of data to be used for an application, and analyzes configuration information of the data (S7303).
[0200] If the type of data is coded data (coded data in S7304), the three-dimensional data demultiplexing device converts the ISOBMFF into a coded stream and extracts the time stamp (S7305). In addition, the three-dimensional data demultiplexing device may determine whether or not data is synchronized by, for example, referring to a flag indicating whether or not data is synchronized, and may perform synchronization processing if data is not synchronized.
[0201] Next, the three-dimensional data demultiplexing device decodes the data in a predetermined manner according to the timestamp and other instructions, and processes the decoded data (S7306).
[0202] On the other hand, if the type of data is coded data (RAW data in S7304), the three-dimensional data demultiplexing device extracts the data and the time stamp (S7307). The three-dimensional data demultiplexing device may also determine whether or not the data are synchronized by, for example, referring to a flag indicating whether or not the data are synchronized, and perform synchronization processing if they are not synchronized. Next, the three-dimensional data demultiplexing device processes the data according to the time stamp and other instructions (S7308).
[0203] For example, an example will be described in which the sensor signals acquired by the beam LiDAR, the FLASH LiDAR, and the camera are encoded and multiplexed using different encoding methods. Fig. 31 is a diagram showing an example of the sensor ranges of the beam LiDAR, the FLASH LiDAR, and the camera. For example, the beam LiDAR detects all directions around the vehicle (sensor), and the FLASH LiDAR and the camera detect a range in one direction (for example, forward) of the vehicle.
[0204] In the case of an application that handles LiDAR point clouds in an integrated manner, the 3D data demultiplexer refers to the overall configuration information to extract and decode the coded data of beam LiDAR and FLASH LiDAR. The 3D data demultiplexer does not extract camera images.
[0205] The 3D data demultiplexing device processes the encoded data of the same time stamp simultaneously according to the time stamps of the LiDAR and FLASH LiDAR.
[0206] For example, the three-dimensional data demultiplexing device may present the processed data on a presentation device, synthesize point cloud data from beam LiDAR and FLASH LiDAR, or perform processing such as rendering.
[0207] Additionally, in the case of applications that involve calibration between data, the three-dimensional data demultiplexer may extract sensor position information for use in the application.
[0208] For example, the three-dimensional data demultiplexing device may select whether to use beam LiDAR information or FLASH LiDAR in an application, and switch processing depending on the selection result.
[0209] In this way, data acquisition and encoding processes can be adaptively changed according to application processing, making it possible to reduce the amount of processing and power consumption.
[0210] A use case in autonomous driving will be described below. FIG. 32 is a diagram showing a configuration example of an autonomous driving system. This autonomous driving system includes a cloud server 7350 and an edge 7360 such as an in-vehicle device or a mobile device. The cloud server 7350 includes a demultiplexer 7351, decoders 7352A, 7352B, and 7355, a point cloud data synthesizer 7353, a large-scale data storage unit 7354, a comparator 7356, and an encoder 7357. The edge 7360 includes sensors 7361A and 7361B, point cloud data generation units 7362A and 7362B, a synchronization unit 7363, encoding units 7364A and 7364B, a multiplexing unit 7365, an update data storage unit 7366, a demultiplexing unit 7367, a decoding unit 7368, a filter 7369, a self-position estimation unit 7370, and a driving control unit 7371.
[0211] In this system, the edge 7360 downloads large-scale data, which is large-scale point cloud map data stored in the cloud server 7350. The edge 7360 performs self-location estimation processing of the edge 7360 (vehicle or terminal) by matching the large-scale data with sensor information obtained by the edge 7360. In addition, the edge 7360 uploads the obtained sensor information to the cloud server 7350 and updates the large-scale data to the latest map data.
[0212] Furthermore, in various applications that handle point cloud data within a system, point cloud data that is encoded using different methods is handled.
[0213] The cloud server 7350 encodes and multiplexes the large-scale data. Specifically, the encoding unit 7357 performs encoding using a third encoding method suitable for encoding a large-scale point cloud. The encoding unit 7357 also multiplexes the encoded data. The large-scale data storage unit 7354 stores the data encoded and multiplexed by the encoding unit 7357.
[0214] The edge 7360 performs sensing. Specifically, the point cloud data generation unit 7362A generates first point cloud data (position information (geometry) and attribute information) using sensing information acquired by the sensor 7361A. The point cloud data generation unit 7362B generates second point cloud data (position information and attribute information) using sensing information acquired by the sensor 7361B. The generated first point cloud data and second point cloud data are used for self-position estimation or vehicle control for autonomous driving, or map update. In each process, part of the information of the first point cloud data and the second point cloud data may be used.
[0215] The edge 7360 performs self-location estimation. Specifically, the edge 7360 downloads large-scale data from the cloud server 7350. The demultiplexer 7367 demultiplexes the large-scale data in a file format to obtain encoded data. The decoder 7368 decodes the obtained encoded data to obtain large-scale data that is large-scale point cloud map data.
[0216] The self-position estimation unit 7370 estimates the vehicle's self-position on the map by matching the acquired large-scale data with the first point cloud data and the second point cloud data generated by the point cloud data generation units 7362A and 7362B. The driving control unit 7371 uses the matching result or the self-position estimation result for driving control.
[0217] The self-position estimation unit 7370 and the driving control unit 7371 may extract specific information, such as position information, from the large-scale data and perform processing using the extracted information. The filter 7369 performs processing, such as correction or thinning, on the first point cloud data and the second point cloud data. The self-position estimation unit 7370 and the driving control unit 7371 may use the first point cloud data and the second point cloud data after the processing. The self-position estimation unit 7370 and the driving control unit 7371 may use sensor signals obtained by the sensors 7361A and 7361B.
[0218] The synchronization unit 7363 performs time synchronization and position correction between a plurality of sensor signals or a plurality of point cloud data. The synchronization unit 7363 may also correct the position information of the sensor signal or the point cloud data to match the large-scale data, based on position correction information between the large-scale data and the sensor data generated by the self-location estimation process.
[0219] Note that synchronization and position correction may be performed by the cloud server 7350, not by the edge 7360. In this case, the edge 7360 may multiplex the synchronization information and position information and transmit them to the cloud server 7350.
[0220] The edge 7360 encodes and multiplexes the sensor signals or the point cloud data. Specifically, the sensor signals or the point cloud data are encoded using a first encoding method or a second encoding method suitable for encoding the respective signals. For example, the encoding unit 7364A generates first encoded data by encoding the first point cloud data using the first encoding method. The encoding unit 7364B generates second encoded data by encoding the second point cloud data using the second encoding method.
[0221] The multiplexing unit 7365 generates a multiplexed signal by multiplexing the first encoded data, the second encoded data, synchronization information, etc. The update data storage unit 7366 stores the generated multiplexed signal. The update data storage unit 7366 also uploads the multiplexed signal to the cloud server 7350.
[0222] The cloud server 7350 synthesizes the point cloud data. Specifically, the demultiplexer 7351 obtains the first encoded data and the second encoded data by demultiplexing the multiplexed signal uploaded to the cloud server 7350. The decoder 7352A obtains the first point cloud data (or the sensor signal) by decoding the first encoded data. The decoder 7352B obtains the second point cloud data (or the sensor signal) by decoding the second encoded data.
[0223] The point cloud data synthesis unit 7353 synthesizes the first point cloud data and the second point cloud data in a predetermined manner. When synchronization information and position correction information are multiplexed in the multiplexed signal, the point cloud data synthesis unit 7353 may perform synthesis using the information.
[0224] The decoding unit 7355 demultiplexes and decodes the large scale data stored in the large scale data storage unit 7354. The comparing unit 7356 compares the point cloud data generated based on the sensor signal obtained by the edge 7360 with the large scale data held by the cloud server 7350, and determines the point cloud data that needs to be updated. The comparing unit 7356 updates the point cloud data that is determined to need to be updated, out of the large scale data, to the point cloud data obtained from the edge 7360.
[0225] The encoding unit 7357 encodes and multiplexes the updated large scale data, and stores the resulting data in the large scale data storage unit 7354.
[0226] As described above, depending on the purpose or application to be used, the signals to be handled may differ, and the signals to be multiplexed or the encoding method may differ. Even in such a case, flexible decoding and application processing is possible by multiplexing data of various encoding methods using this embodiment. Furthermore, even if the signal encoding method is different, various applications and systems can be constructed and flexible services can be provided by converting the encoding method that is more suitable for the demultiplexing, decoding, data conversion, encoding, and multiplexing processes.
[0227] As described above, the three-dimensional data multiplexing device according to this embodiment performs the process shown in Fig. 33. The three-dimensional data multiplexing device generates an output signal of a predetermined file structure (e.g., ISOBMFF) by multiplexing a plurality of types of data including point cloud data (S7311). Next, the three-dimensional data multiplexing device stores information (e.g., data_type) indicating the type of each of the plurality of data included in the output signal in metadata (control information) in the file structure (S7312).
[0228] According to this, the three-dimensional data multiplexing device stores information indicating the type of each of the multiple data included in the output signal in metadata in the file structure. This allows the three-dimensional data demultiplexing device receiving the output signal to easily determine the type of each data. In this way, the three-dimensional data multiplexing method can appropriately multiplex and transmit point cloud data.
[0229] For example, the information indicating the type of each of the multiple data items may indicate (1) the encoding method applied to the data, (2) the structure of the data, (3) the type of sensor that generated the data, or (4) the data format.
[0230] For example, the metadata in the file configuration includes synchronization information for synchronizing the times of multiple pieces of data included in the output signal, which allows the multiple pieces of data to be synchronized in a three-dimensional data demultiplexing device that receives the output signal.
[0231] For example, the synchronization information indicates the difference in time stamps between a plurality of pieces of data, which makes it possible to reduce the amount of data in the output signal.
[0232] For example, the three-dimensional data multiplexing device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.
[0233] Moreover, the three-dimensional data demultiplexing device according to this embodiment performs the process shown in FIG. 34. The three-dimensional data demultiplexing device acquires information (e.g., data_type) indicating the type of each of the multiple data included in the output signal, which is stored in the metadata in the file structure, from an output signal of a predetermined file structure (e.g., ISOBMFF) in which multiple types of data including point cloud data are multiplexed (S7321). The three-dimensional data demultiplexing device acquires multiple data from the output signal using the information indicating the type of each of the multiple data (S7322). For example, the three-dimensional data demultiplexing device selectively acquires necessary data from the output signal using the information indicating the type of each of the multiple data. This allows the three-dimensional data demultiplexing device to easily determine the type of each data.
[0234] For example, the information indicating the type of each of the multiple data items may indicate (1) the encoding method applied to the data, (2) the structure of the data, (3) the type of sensor that generated the data, or (4) the data format.
[0235] For example, the metadata in the file structure includes synchronization information for synchronizing the times of multiple pieces of data included in the output signal. For example, the three-dimensional data demultiplexing device synchronizes the multiple pieces of data using the synchronization information.
[0236] For example, the synchronization information indicates a difference in timestamps between the multiple data. For example, the synchronization information includes information indicating a timestamp of one of the multiple data, and the three-dimensional data demultiplexing device restores the timestamp of the other data among the multiple data by adding the difference indicated by the synchronization information to the timestamp of one of the multiple data. This makes it possible to reduce the amount of data in the output signal.
[0237] For example, the three-dimensional data demultiplexing device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.
[0238] (Embodiment 2) First, generation and synthesis of point cloud data will be described. Fig. 35 is a diagram showing an example of the configuration of a three-dimensional data encoding device (three-dimensional data multiplexing device) which is an example of a system for processing a plurality of sensor signals. The three-dimensional data encoding device includes sensing units 8601A, 8601B, and 8601C, input / output units 8602A, 8602B, and 8602C, a point cloud data generating unit 8603, an encoding unit 8604, a synchronization unit 8605, and a multiplexing unit 8606.
[0239] The input / output unit 8602A acquires a first sensor signal generated by sensing in the sensing unit 8601A. The input / output unit 8602B acquires a second sensor signal generated by sensing in the sensing unit 8601B. The input / output unit 8602C acquires a third sensor signal generated by sensing in the sensing unit 8601C. Note that the input / output units 8602A, 8602B, and 8602C may have a memory that stores the acquired sensor signals.
[0240] The point cloud data generation unit 8603 generates composite point cloud data by combining (also called merging or fusing) the first sensor signal and the second sensor signal. The composite point cloud data includes position information and attribute information of three-dimensional points obtained by the first sensing unit 8601A and the second sensing unit 8601B, and other sensor information. The point cloud data generation unit 8603 may combine the three-dimensional point cloud data based on a synchronization signal for synchronizing multiple sensing signals, such as a third sensor signal, a common sensor signal, and common time information.
[0241] The encoding unit 8604 generates encoded data by encoding the composite point cloud data. The synchronization unit 8605 has a function for synchronizing a plurality of sensing units. The multiplexing unit 8606 generates an output signal (bit stream) by multiplexing the encoded data, metadata, raw data of the sensor signal, and a synchronization signal.
[0242] 36 is a block diagram showing the configuration of the point cloud data generation unit 8603. The point cloud data generation unit 8603 includes conversion units 8611 and 8612, and a synthesis unit 8613.
[0243] The converter 8611 generates first point cloud data from the first sensor signal. The converter 8612 generates second point cloud data from the second sensor signal. The sensor signal is, for example, one-dimensional information consisting of laser angle and distance information, or two-dimensional distance information obtained by a FLASH LiDAR or a camera. The converters 8611 and 8612 convert the one-dimensional or two-dimensional position information (geometry information) into three-dimensional position information (for example, (x, y, z) coordinates).
[0244] A point (three-dimensional point) which is a component of point cloud data includes position information such as three-dimensional coordinates and attribute information for the position information. Figs. 37 to 39 are diagrams showing examples of information of three-dimensional points. In the example shown in Fig. 37, one attribute information exists for one piece of position information. In the examples shown in Figs. 38 and 39, two or more attribute information exists for one piece of position information. When multiple attribute information exists for one piece of position information, there are cases where multiple different attribute types exist, and cases where multiple attribute information of the same attribute type exists. The example shown in Fig. 37 is, for example, a case where one color information exists as attribute information. Also, the example shown in Fig. 38 is, for example, a case where color information and reflectance information exist. Also, the example shown in Fig. 39 is, for example, a case where three or more pieces of color information exist. Note that the position information has three values of (x, y, z). The color information has three values of (R, G, B) or (Y, Cb, Cr). The reflectance is a single value. In this way, the position information and the attribute information may have information for each dimension.
[0245] In addition, the attribute information based on the first sensor signal and the attribute information based on the second sensor signal may be attribute information of the same type (e.g., color) of different objects, or may be attribute information of different types (e.g., color and reflectance).
[0246] The synthesis of point cloud data files will be described below. Fig. 40 is a diagram showing an example of synthesizing a plurality of sensor signals to generate synthesized point cloud data. The figure shows an example of synthesizing sensor information X and sensor information Y having point-by-point information with sensor information Z having common information. Synthesized point cloud information, which is the point cloud information after synthesis, includes common information, point-by-point information, and configuration information.
[0247] An example will be described below in which sensor information X and sensor information Y have the same attribute information (e.g., color information or reflectance). For example, the configuration information includes a sensor ID and the sensor information of the original sensor. In the example shown in FIG. 40, the configuration information includes a sensor ID (ID=S1), sensor information (Sensor), the sensor version (Version), the sensor manufacturer name (Maker), the sensor installation information (Mount Info.), and the sensor position coordinates (World Coordinate) as information on the sensor X (SensorX) that generated the sensor information X. In addition, the configuration information includes a sensor ID (ID=S2) as information on the sensor Y (SensorY) that generated the sensor information Y. Note that these are merely examples, and some of the above information may be included as the sensor information.
[0248] Furthermore, the information on each point includes attribute information S indicating a sensor ID (S1 or S2). In this manner, identification information (sensor ID) indicating from which sensor the data for each point was acquired is indicated for each point.
[0249] Fig. 41 is a diagram showing an example in which identification information indicating which sensor acquired data for each point is shown in the configuration information. The configuration information shown in Fig. 41 includes the number of points (numPoints) included in the composite point cloud information for each ID.
[0250] Also, although the points obtained by a sensing means such as a sensor are described here, this is not limiting. For example, an artificially assigned point is not a point obtained by a sensing means, and may be defined as "sensor ID: S4...point obtained by means other than a sensing means", and S4 may be assigned to the attribute information S, or a fixed value meaning invalidation may be indicated in the attribute information S. This allows the three-dimensional data decoding device to identify points obtained by means other than a sensing means in the composite point cloud.
[0251] The configuration information may also include information identifying points obtained by means other than the sensing means, or may include a flag indicating that the composite point cloud includes points obtained by means other than the sensing means, thereby enabling the three-dimensional data decoding device to determine whether the composite point cloud includes points obtained by means other than the sensing means.
[0252] In addition, the example shown in FIG. 41 etc. is an example in which when the position information overlaps between sensor information X and sensor information Y (the first and fourth position information are the same at G1), the overlapping points are recorded as they are. The three-dimensional data encoding device may merge the overlapping points using a predetermined method, or may select one of them. When merging overlapping points, the three-dimensional data encoding device indicates in the composite point cloud information that the point is data in which information acquired from multiple sensors is merged. For example, sensor ID=S1 indicates data obtained by sensor X, sensor ID=S2 indicates data obtained by sensor Y, and sensor ID=S3 indicates that the data from sensor X and sensor Y is merged.
[0253] An example will be described below in which the sensor information X and the sensor information Y have different attribute information. Fig. 42 is a diagram for explaining the synthesis process in this case. As shown in Fig. 42, the three-dimensional data encoding device acquires different attribute information A1 (e.g., color information) and attribute information A2 (e.g., reflectance) from different sensor information, and synthesizes them. The configuration information includes information on the attribute information A1 and A2, and identification information.
[0254] Among the points obtained by sensor X and sensor Y, there are some points where the location information overlaps and some points where it does not overlap. When the location information overlaps, the combined location information has attribute information A1 and attribute information A2 for one piece of location information. When the location information does not overlap, the combined location information has either attribute information A1 or attribute information A2 for one piece of location information.
[0255] At this time, if there is no attribute information, information (N / A) indicating that the attribute information is invalid is displayed. Although N / A is written in FIG. 42, invalidity may be indicated by a fixed value such as 0 or -1. Information on points including invalid attribute information may not be displayed. Alternatively, invalid attribute information may be interpolated or estimated by a predetermined method. Also, a flag indicating whether or not invalid attribute information is included may be included in the configuration information.
[0256] Next, an example will be described in which point cloud information X has attribute information A and attribute information B, and point cloud information Y has attribute information B. Fig. 43 is a diagram for explaining the synthesis process in this case.
[0257] The point cloud information X is point cloud information including two or more pieces of attribute information obtained from one piece of sensor information. The point cloud information X may be point cloud information converted in advance from two or more pieces of sensor information. The point cloud information Y is point cloud information having one piece of attribute information.
[0258] In the point cloud information X, the position information G2 does not have valid attribute information A2. In addition, the position information G2 and the position information G3 overlap in the point cloud information X and the point cloud information Y.
[0259] The composite point cloud information (with overlapping points) is an example in which information on overlapping points is described individually. The number of points included in the composite point cloud information is equal to the total number of points in the point cloud information before synthesis. The configuration information also indicates that the attribute information S is identification information indicating which sensor was used to acquire the point information (position information and attribute information). The attribute information S is also added as information for each point.
[0260] The composite point cloud information (no overlapping points) is an example in which the information of overlapping points in the composite point cloud information (with overlapping points) is merged. All of the attribute information A1 is data acquired by sensor X, and the attribute information A2 is either data acquired by sensor X, data acquired by sensor Y, data based on both data, or N / A.
[0261] The configuration information indicates that the attribute information A1 was acquired by a sensor X. The attribute information indicates that the attribute information S is information indicating which sensor acquired the attribute information A2. Note that information indicating which sensor acquired the attribute information A1 may be provided as separate attribute information.
[0262] In addition, a flag indicating whether or not overlapping points have been merged in the synthesis of point cloud information may be included in the configuration information. In addition, a flag indicating whether or not attribute information has been merged in the merging of overlapping points may be included in the configuration information, and an identifier indicating a merging method may be included in the configuration information. Alternatively, if one of the attribute information is selected instead of merging, an identifier indicating the selection criterion may be included in the configuration information.
[0263] The attribute information S indicates which sensor acquired the data of each point in the attribute information A2. For example, a sensor ID=S1 indicates data acquired by a sensor X, a sensor ID=S2 indicates data acquired by a sensor Y, and a sensor ID=S3 indicates that the data from the sensors X and Y are merged.
[0264] 43, f(A2X3, A2Y2) indicates a value derived by a predetermined method using A2X3 and A2Y2. The predetermined method is an average value, a weighted average value, or a method of selecting one value from multiple values based on a predetermined reference value (e.g., the reliability of a sensor), etc.
[0265] As described above, by indicating in configuration information or attribute information which sensor information or which point cloud information the point cloud data or attribute information of the point cloud data is obtained from, efficient encoding or various flexible functions can be realized in the downstream encoding unit or decoding unit, multiplexing unit or demultiplexing unit, or application processing.
[0266] Although an example in which there are two pieces of attribute information has been shown here, there may be three or more pieces of attribute information, or the point group may not include any attribute information.
[0267] In addition, the attribute information S may not only be information indicating which sensor the point cloud information or attribute information was acquired by, but also information indicating which sensor the position information was acquired by, or any combination (class) of position information and attribute information may be defined and may indicate which class of data the combination of points belongs to.
[0268] Also, the configuration information shown in Fig. 43 may include at least one of the following in addition to the sensor ID: sensor information (Sensor), sensor version (Version), sensor manufacturer name (Maker), sensor installation information (Mount Info.), and sensor position coordinates (World Coordinate), in the same way as in Fig. 40. This allows the three-dimensional data decoding device to obtain information on various sensors from the configuration information.
[0269] Next, a method for encoding the composite point cloud data will be described. Fig. 44 is a block diagram of a three-dimensional data encoding / decoding system. The system shown in the figure includes a three-dimensional data encoding device 8620 and a three-dimensional data decoding device 8630. The three-dimensional data encoding device 8620 includes an encoding unit 8621 and a multiplexing unit 8622. The encoding unit 8621 generates encoded data by encoding the composite point cloud data (or sensor signal). The multiplexing unit 8622 generates a multiplexed signal (bit stream) by multiplexing the encoded data.
[0270] The three-dimensional data decoding device 8630 includes a demultiplexing unit 8631, a decoding unit 8632, and a reconstruction unit 8633. The demultiplexing unit 8631 generates encoded data by demultiplexing a multiplexed signal. The decoding unit 8632 generates decoded data by decoding the encoded data. The reconstruction unit 8633 restores the point cloud data (or the sensor signal) by reconstructing the decoded data.
[0271] 45 is a diagram showing the structure of a bitstream including metadata and encoded data. The SPS, which is the entire metadata, includes a list of attribute information included in the point cloud data. The list stores identification information (attribute_type) of the attribute information.
[0272] Attribute information (e.g., "sensor attribute") is defined to indicate which sensor was used to acquire the point cloud information or attribute information. If attribute information of attribute_type="sensor attribute" is included in the overall point cloud information, attribute_type="sensor attribute" is indicated in the list of attribute information in the overall metadata. It may be defined whether the sensor attribute information is one-dimensional or two-dimensional, and the number of dimensions may be indicated.
[0273] Moreover, if the entire sensor information (Z) is not essential for encoding, it is stored in the optional SEI, and if it is essential for encoding, it is stored in the SPS or APS. Furthermore, metadata related to sensor attributes obtained from the configuration information is stored in metadata APS3 related to sensor attributes. The position information G, attribute information A1, attribute information A2, and attribute information S indicating the sensor attributes are each compressed by entropy encoding and output as encoded data.
[0274] Also, as shown in Figure 45, SPS is metadata for the entire encoded data, GPS is metadata for location information, APS is metadata for each attribute information, SEI is optional metadata, G is encoded data for location information, and A1, A2, and A3 are encoded data for attribute information.
[0275] Next, a method for decoding composite point cloud data and an example of its use will be described. Fig. 46 is a flowchart of a decoding process in a three-dimensional data decoding device. First, the three-dimensional data decoding device analyzes metadata included in a bit stream, acquires position information and attribute information constituting the encoded data, and decodes each encoded data (S8601).
[0276] The three-dimensional data decoding device also determines whether the encoded data includes attribute information whose attribute_type is "sensor attribute" (S8602). If the encoded data includes attribute information of the sensor attribute (Yes in S8602), the three-dimensional data decoding device reconstructs point cloud data including the sensor attribute (S8603). If the encoded data does not include attribute information of the sensor attribute (No in S8602), the three-dimensional data decoding device reconstructs point cloud data not including the sensor attribute (S8604).
[0277] Fig. 47 is a flowchart showing a process of extracting a specific point from point cloud data including sensor attributes. Fig. 48 is a diagram showing an example of data extraction. First, the three-dimensional data decoding device determines to extract points related to sensor X from point cloud data including sensor attributes (S8611). Next, the three-dimensional data decoding device analyzes the configuration information and recognizes that attribute information A1 was acquired by sensor X and that attribute information S indicates which sensor acquired attribute information A2 (S8612).
[0278] Next, the three-dimensional data decoding device determines the type of attribute information (S8613). If the type of attribute information is A1 (A1 in S8613), the three-dimensional data decoding device determines that the data is data based on sensor X, and extracts the A1 data (S8614). Note that, if the data is N / A, the three-dimensional data decoding device does not extract the data.
[0279] If the type of attribute information is A2 or S (A2 or S in S8613), the three-dimensional data decoding device analyzes the sensor attribute of A2 based on the attribute information S (S8615). If the sensor attribute is S1 or S3 (S1 or S3 in S8615), the three-dimensional data decoding device determines that the data is data based on sensor X and extracts the A2 data (S8616). On the other hand, if the sensor attribute is S2 (S2 in S8615), the three-dimensional data decoding device determines that the data is data based on sensor Y and does not extract the A2 data (S8617).
[0280] When the sensor attribute is S3 in the analysis of the sensor attribute (S8615), the three-dimensional data decoding device may convert or correct the merged f(A2X3, A2Y2) in a predetermined manner. When extracting original data acquired by sensor X, the three-dimensional data decoding device does not need to extract data of S3 in the analysis of the sensor attribute.
[0281] Also, the configuration information shown in Fig. 48 may include at least one of the sensor information (Sensor), the sensor version (Version), the sensor manufacturer name (Maker), the sensor installation information (Mount Info.), and the sensor position coordinates (World Coordinate) in addition to the sensor ID, similarly to Fig. 40. This allows the three-dimensional data decoding device to obtain information on various sensors from the configuration information.
[0282] Next, a method for encoding the composite point cloud data when data division is performed will be described. Data division is effective for increasing speed through parallel processing and improving encoding efficiency by grouping attribute information having the same attribute, and the point cloud data is divided by a predetermined method.
[0283] 49 is a block diagram of a three-dimensional data encoding / decoding system. The system shown in the figure includes a three-dimensional data encoding device 8640 and a three-dimensional data decoding device 8650. The three-dimensional data encoding device 8640 includes a division unit 8641, a plurality of encoding units 8642, and a multiplexing unit 8643.
[0284] The division unit 8641 generates a plurality of divided data by dividing the point cloud data into a plurality of data units such as tiles or slices. Alternatively, the division unit 8641 generates a plurality of divided data by dividing attribute information included in the point cloud data for each piece of sensor information.
[0285] The multiple encoding units 8642 generate multiple pieces of encoded data by encoding the multiple pieces of divided data for each division unit. The multiplexing unit 8643 multiplexes the multiple pieces of encoded data to generate a multiplexed signal (bit stream).
[0286] The three-dimensional data decoding device 8650 includes a demultiplexing unit 8651, a plurality of decoding units 8652, and a reconstruction unit 8653. The demultiplexing unit 8651 generates a plurality of coded data by demultiplexing a multiplexed signal. The decoding unit 8652 generates a plurality of decoded data by decoding the plurality of coded data. The reconstruction unit 8653 reconstructs the plurality of decoded data to reconstruct point cloud data.
[0287] FIG. 50 is a flowchart of the encoding process when data is divided for each piece of sensor information. First, the three-dimensional data encoding device analyzes configuration information of the input point cloud data (S8621). Next, the three-dimensional data encoding device decides to divide and encode the point cloud data (position information and attribute information) for each piece of sensor information (S8622). For example, the three-dimensional data encoding device may decide to divide the data when it is determined that the encoding efficiency is improved by dividing the data based on the configuration information. Alternatively, the three-dimensional data encoding device may actually estimate the encoding efficiency and decide whether or not to divide the data based on the result. Alternatively, the three-dimensional data encoding device may decide whether or not to divide the data based on an external instruction such as an app.
[0288] Next, the three-dimensional data encoding device determines which sensor the point data is based on (S8623). If the sensor attribute is S1 and the point data is based on sensor X (S1 in S8624), the three-dimensional data encoding device encodes the data as split data based on sensor X (S8625). On the other hand, if the sensor attribute is S2 and the point data is based on sensor Y (S2 in S8624), the three-dimensional data encoding device encodes the data as split data based on sensor Y (S8626).
[0289] Next, the three-dimensional data encoding device divides the configuration information of the input data into overall configuration information and configuration information for each divided data, and stores the overall configuration information and the configuration information for each divided data in metadata. In addition, the three-dimensional data encoding device generates a multiplexed signal (bit stream) by multiplexing the metadata and the encoded data (S8627).
[0290] Fig. 51 is a diagram showing an example of this division process. As shown in Fig. 51, the first divided data has attribute information A1 and A2, and the second divided data has attribute information A2 but does not have attribute information A1.
[0291] At least one of the overall configuration information and the configuration information for each divided data shown in Fig. 51 may include at least one of the sensor information (Sensor), the sensor version (Version), the sensor manufacturer name (Maker), the sensor installation information (Mount Info.), and the sensor position coordinates (World Coordinate) in addition to the sensor ID, as in Fig. 40. This allows the three-dimensional data decoding device to obtain information on various sensors from the configuration information.
[0292] An example of decoding and application of the divided data will be described below. First, the information of the divided data will be described. Fig. 52 is a diagram showing an example of the configuration of a bit stream. The overall information of the divided data indicates the sensor ID (sensor_id) and data ID (data_id) of each divided data. The data ID is also indicated in the header of each encoded data.
[0293] Note that the overall information of the divided data shown in Fig. 52 may include at least one of the following in addition to the sensor ID: sensor information (Sensor), sensor version (Version), sensor manufacturer name (Maker), sensor installation information (Mount Info.), and sensor position coordinates (World Coordinate), in the same way as in Fig. 40. This allows the three-dimensional data decoding device to obtain information on various sensors from the configuration information.
[0294] The overall information of the divided data may be stored in the metadata SPS, GPS, or APS, or may be stored in the metadata SEI that is not essential for encoding. In addition, the three-dimensional data encoding device stores the SEI in an ISOBMFF file during multiplexing. The three-dimensional data decoding device can obtain the desired divided data based on the metadata.
[0295] In Figure 52, SPS is metadata for the entire encoded data, GPS is metadata for location information, APS is metadata for each attribute information, G is encoded data for location information for each split data, and A1 etc. are encoded data for attribute information for each split data.
[0296] Next, an example of an application for divided data will be described. An example of an application for selecting an arbitrary point cloud from point cloud data and presenting the selected point cloud will be described. Fig. 53 is a flowchart of the point cloud selection process executed by this application. Figs. 54 to 56 are diagrams showing example screens for the point cloud selection process.
[0297] 54, the three-dimensional data decoding device that executes the application has a UI unit that displays an input UI (user interface) 8661 for selecting an arbitrary point cloud, for example. The input UI 8661 has a presentation unit 8662 that presents the selected point cloud, and an operation unit (buttons 8663 and 8664) that accepts user operations. After the point cloud is selected in the UI 8661, the three-dimensional data decoding device acquires the desired data from the storage unit 8665.
[0298] First, based on the user's operation on the input UI 8661, the user selects the point cloud information that he or she wishes to display (S8631). Specifically, by selecting button 8663, a point cloud based on sensor 1 is selected. By selecting button 8664, a point cloud based on sensor 2 is selected. Alternatively, by selecting both buttons 8663 and 8664, both the point cloud based on sensor 1 and the point cloud based on sensor 2 are selected. Note that the method of selecting the point clouds is merely an example and is not limited thereto.
[0299] Next, the three-dimensional data decoding device analyzes the overall information of the divided data included in the multiplexed signal (bit stream) or the encoded data, and identifies the data ID (data_id) of the divided data constituting the selected point cloud from the sensor ID (sensor_id) of the selected sensor (S8632). Next, the three-dimensional data decoding device extracts encoded data including the identified desired data ID from the multiplexed signal, and decodes the extracted encoded data to decode the point cloud based on the selected sensor (S8633). Note that the three-dimensional data decoding device does not decode other encoded data.
[0300] Finally, the three-dimensional data decoding device presents (e.g., displays) the decoded point cloud (S8634). Fig. 55 shows an example in which the sensor 1 button 8663 is pressed, and the point cloud of sensor 1 is presented. Fig. 56 shows an example in which both the sensor 1 button 8663 and the sensor 2 button 8664 are pressed, and the point clouds of sensor 1 and sensor 2 are presented.
[0301] As described above, the three-dimensional data encoding device according to this embodiment performs the process shown in Fig. 57. The three-dimensional data encoding device generates encoded data by encoding point cloud data (S8641), and generates a bit stream including the encoded data (S8642). The bit stream includes first information (e.g., configuration information or attribute information S, etc.) indicating a sensor corresponding to the point cloud data.
[0302] This allows the three-dimensional data decoding device to extract point cloud data obtained by a desired sensor using the first information, and therefore the three-dimensional data encoding device can generate a bit stream from which the point cloud data can be appropriately extracted.
[0303] For example, as shown in FIG. 40 etc., the first information is generated for each of a plurality of three-dimensional points included in the point cloud data, and includes second information (eg, attribute information S) indicating a sensor corresponding to the corresponding three-dimensional point.
[0304] For example, as shown in FIG. 40, the point cloud data includes position information (e.g., G) of each of a plurality of three-dimensional points and one or more pieces of attribute information (e.g., A1), and the second information is included in the encoded data as one of the one or more pieces of attribute information (e.g., attribute information S).
[0305] For example, as shown in FIG. 40, the bit stream includes third information (e.g., configuration information) common to multiple three-dimensional points, the second information is a sensor identifier (e.g., S1, etc.), and the third information includes fourth information (e.g., ID, etc.) indicating the identification of the sensor and the correspondence with the sensor.
[0306] For example, as shown in FIG. 41, the bit stream includes third information common to multiple 3D points, and the third information indicates the number of 3D points for each sensor (eg, numPoints).
[0307] For example, as shown in FIG. 42, the point cloud data includes position information of each of a plurality of three-dimensional points and one or more pieces of attribute information, and the first information indicates a sensor corresponding to each of the one or more pieces of attribute information.
[0308] For example, as shown in FIG. 43, the three-dimensional data encoding device further generates point cloud data (e.g., synthesized point cloud information (without overlapping points)) by synthesizing first point cloud data based on a first sensor and second point cloud data based on a second sensor. In the synthesis, if the first point cloud data and the second point cloud data contain attribute information of a first point having the same position information, attribute information (e.g., f(A2X3, A2Y2) based on the attribute information of the first point contained in the first point cloud data and the attribute information of the first point contained in the second point cloud data is generated as attribute information after synthesis, and the first information indicates that the information of the first point contained in the point cloud data is based on the first sensor and the second sensor (e.g., S3).
[0309] For example, as shown in Fig. 51, in encoding point cloud data, the three-dimensional data encoding device divides the point cloud data into first point cloud data based on a first sensor and second point cloud data based on a second sensor, and generates encoded data by encoding the first point cloud data and the second point cloud data. The bit stream includes first control information for the first point cloud data (e.g., configuration information of divided data 1) and second control information for the second point cloud data (e.g., configuration information of divided data 2). The first information includes (1) information included in the first control information and indicating a sensor corresponding to the first point cloud data, and (2) information included in the second control information and indicating a sensor corresponding to the second point cloud data.
[0310] For example, as shown in Fig. 51, in encoding point cloud data, the three-dimensional data encoding device divides the point cloud data into first point cloud data based on a first sensor and second point cloud data based on a second sensor, and generates encoded data by encoding the first point cloud data and the second point cloud data. The bit stream includes control information (e.g., overall configuration information) common to the first point cloud data and the second point cloud data, and the first information is included in the control information.
[0311] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.
[0312] Moreover, the three-dimensional data decoding device according to this embodiment performs the process shown in Fig. 58. The three-dimensional data decoding device acquires the first information and the point cloud data by decoding a bit stream that includes encoded data generated by encoding the point cloud data and that includes first information indicating a sensor corresponding to the point cloud data (S8651). For example, the three-dimensional data decoding device acquires point cloud data corresponding to a desired sensor from the bit stream using the acquired first information.
[0313] According to this, the three-dimensional data decoding device can obtain point cloud data by decoding a bit stream from which point cloud data can be appropriately extracted.
[0314] For example, as shown in FIG. 40 etc., the first information is generated for each of a plurality of three-dimensional points included in the point cloud data, and includes second information (eg, attribute information S) indicating a sensor corresponding to the corresponding three-dimensional point.
[0315] For example, as shown in FIG. 40, the point cloud data includes position information (e.g., G) of each of a plurality of three-dimensional points and one or more pieces of attribute information (e.g., A1), and the second information is included in the encoded data as one of the one or more pieces of attribute information (e.g., attribute information S).
[0316] For example, as shown in FIG. 40, the bit stream includes third information (e.g., configuration information) common to multiple three-dimensional points, the second information is a sensor identifier (e.g., S1, etc.), and the third information includes fourth information (e.g., ID, etc.) indicating the identification of the sensor and the correspondence with the sensor.
[0317] For example, as shown in FIG. 41, the bit stream includes third information common to multiple 3D points, and the third information indicates the number of 3D points for each sensor (eg, numPoints).
[0318] For example, as shown in FIG. 42, the point cloud data includes position information of each of a plurality of three-dimensional points and one or more pieces of attribute information, and the first information indicates a sensor corresponding to each of the one or more pieces of attribute information.
[0319] For example, as shown in Fig. 43, point cloud data (e.g., synthesized point cloud information (no overlapping points)) is generated by synthesizing first point cloud data based on a first sensor and second point cloud data based on a second sensor. In the synthesis, when the first point cloud data and the second point cloud data contain attribute information of a first point having the same position information, attribute information (e.g., f(A2X3, A2Y2)) based on the attribute information of the first point contained in the first point cloud data and the attribute information of the first point contained in the second point cloud data is generated as attribute information after synthesis, and the first information indicates that the information of the first point contained in the point cloud data is based on the first sensor and the second sensor (e.g., S3).
[0320] For example, as shown in Fig. 51, the encoded data is generated by dividing the point cloud data into first point cloud data based on a first sensor and second point cloud data based on a second sensor, and encoding the first point cloud data and the second point cloud data. The bit stream includes first control information for the first point cloud data (e.g., configuration information of divided data 1) and second control information for the second point cloud data (e.g., configuration information of divided data 2). The first information includes (1) information included in the first control information and indicating a sensor corresponding to the first point cloud data, and (2) information included in the second control information and indicating a sensor corresponding to the second point cloud data.
[0321] For example, as shown in Fig. 51, the encoded data is generated by dividing the point cloud data into first point cloud data based on a first sensor and second point cloud data based on a second sensor, and encoding the first point cloud data and the second point cloud data. The bit stream includes control information (e.g., overall configuration information) common to the first point cloud data and the second point cloud data, and the first information is included in the control information.
[0322] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.
[0323] Although the three-dimensional data encoding device and the three-dimensional data decoding device according to the embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments.
[0324] Furthermore, each processing unit included in the three-dimensional data encoding device and the three-dimensional data decoding device according to the above-mentioned embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be included in a single chip.
[0325] The integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. A field programmable gate array (FPGA) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of the circuit cells inside the LSI may also be used.
[0326] In each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.
[0327] Furthermore, the present disclosure may be realized as a three-dimensional data encoding method or a three-dimensional data decoding method executed by a three-dimensional data encoding device, a three-dimensional data decoding device, or the like.
[0328] In addition, the division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, or some functions may be transferred to another functional block. Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.
[0329] In addition, the order in which each step is performed in the flowchart is merely an example for specifically explaining the present disclosure, and the steps may be performed in an order other than the above. In addition, some of the steps may be performed simultaneously (in parallel) with other steps.
[0330] Although the three-dimensional data encoding device and the three-dimensional data decoding device according to one or more aspects have been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceived by a person skilled in the art to the present embodiment and forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects. [Industrial Applicability]
[0331] The present disclosure is applicable to a three-dimensional data encoding device and a three-dimensional data decoding device. [Explanation of symbols]
[0332] 4601 Three-dimensional data encoding system 4602 Three-dimensional data decoding system 4603 Sensor terminal 4604 External connection part 4611 Point Cloud Data Generation System 4612 Presentation section 4613 Encoding section 4614 Multiplexer 4615 Input / output section 4616 Control Unit 4617 Sensor information acquisition unit 4618 Point Cloud Data Generation Unit 4621 Sensor Information Acquisition Unit 4622 Input / output section 4623 Demultiplexer 4624 Decoding section 4625 Presentation section 4626 User Interface 4627 Control Unit 4630 First Encoding Section 4631 Location information encoder 4632 Attribute information encoder 4633 Additional Information Encoding Unit 4634 Multiplexer 4640 First Decoding Unit 4641 Demultiplexer 4642 Location Information Decoding Unit 4643 Attribute Information Decoding Unit 4644 Additional Information Decoding Unit 4650 Second Encoding Section 4651 Additional Information Generation Unit 4652 Position image generation unit 4653 Attribute Image Generation Unit 4654 Video Encoding Unit 4655 Additional Information Encoding Unit 4656 Multiplexer 4660 Second Decryption Unit 4661 Demultiplexer 4662 Video Decoding Unit 4663 Additional Information Decoding Unit 4664 Location information generator 4665 Attribute information generation section 7301, 7301A, 7301B, 7301C Sensing Devices 7302, 7302A, 7302B, 7302C Sensor information input section 7303, 7303A, 7303C Point Cloud Data Generation Unit 7304A, 7304B Sensing section 7305 Merge Department 7310 First Device 7311, 7321 Sensing unit 7312, 7322 Output I / F 7320 Second Device 7323 Input I / F 7324 Processing section 7325 Point Cloud Data Generation Unit 7326 Encoder 7331A, 7331B, 7331C Sensing section 7332A, 7332B, 7332C Sensor information input section 7333A, 7333B Point Cloud Data Generation Unit 7334A, 7334B encoder 7335 Synchronization Unit 7336 Multiplexer 7341A, 7341B, 7341C Sensing section 7342A, 7342B, 7342C input / output section 7343 Point Cloud Data Generation Unit 7344A, 7344B encoder 7345 Synchronization Unit 7346 Multiplexer 7350 Cloud Server 7351 Demultiplexer 7352A, 7352B Decoding Unit 7353 Point Cloud Data Synthesis Section 7354 Large-Scale Data Storage Department 7355 Decryption Unit 7356 Comparison Section 7357 Encoder 7360 Edge 7361A, 7361B Sensors 7362A, 7362B Point Cloud Data Generation Unit 7363 Synchronization section 7364A, 7364B encoder 7365 Multiplexer 7366 Update Data Storage Unit 7367 Demultiplexer 7368 Decryption section 7369 Filter 7370 Self-position estimation part 7371 Operation control unit 8601A, 8601B, 8601C Sensing section 8602A, 8602B, 8602C input / output section 8603 Point Cloud Data Generation Unit 8604 Encoding section 8605 Synchronization section 8606 Multiplexer 8611, 8612 conversion unit 8613 Synthesis Department 8620, 8640 3D data encoding device 8621, 8642 Encoding section 8622, 8643 Multiplexer 8630, 8650 3D data decoding device 8631, 8651 Demultiplexer 8632, 8652 Decoding section 8633, 8653 Reconstruction part 8641 Split section 8661 Input UI 8662 Presentation section 8663, 8664 Buttons 8665 Storage unit
Claims
1. An encoding method performed by an encoding device, comprising: generating encoded data by encoding point cloud data composed of a plurality of three-dimensional points; generating a bitstream including the encoded data and control information; the encoded data includes a plurality of point data each corresponding to one of the plurality of three-dimensional points; When the plurality of point data includes one or more point data to which sensor identification information is assigned, the control information indicates that the encoded data includes the sensor identification information. Three-dimensional data encoding method.
2. Each of the plurality of point data includes one or more pieces of attribute information; the control information includes attribute identification information for identifying a type of each of the one or more pieces of attribute information, When the one or more pieces of attribute information include the sensor identification information, the control information includes the attribute identification information indicating the sensor identification information.
2. The three-dimensional data encoding method according to claim 1.
3. The point cloud data is acquired by one or more sensors; The sensor identification information indicates which of the one or more sensors was used to acquire information included in the point data to which the sensor identification information is assigned.
2. The three-dimensional data encoding method according to claim 1.
4. the encoded data includes position information and one or more pieces of attribute information for each of the plurality of three-dimensional points; The sensor identification information is included in the encoded data as one piece of the one or more pieces of attribute information.
2. The three-dimensional data encoding method according to claim 1.
5. A decoding method performed by a decoding device, comprising: Obtaining coded data and control information from the bitstream; Decoding the encoded data to obtain point cloud data composed of a plurality of three-dimensional points; the encoded data includes a plurality of point data each corresponding to one of the plurality of three-dimensional points; When the plurality of point data includes one or more point data to which sensor identification information is assigned, the control information indicates that the encoded data includes the sensor identification information. A method for decoding three-dimensional data.
6. Each of the plurality of point data includes one or more pieces of attribute information; the control information includes attribute identification information for identifying a type of each of the one or more pieces of attribute information, When the one or more pieces of attribute information include the sensor identification information, the control information includes the attribute identification information indicating the sensor identification information.
6. The three-dimensional data decoding method according to claim 5.
7. The point cloud data is acquired by one or more sensors; The sensor identification information indicates which of the one or more sensors was used to acquire information included in the point data to which the sensor identification information is assigned.
6. The three-dimensional data decoding method according to claim 5.
8. the encoded data includes position information and one or more pieces of attribute information for each of the plurality of three-dimensional points; The sensor identification information is included in the encoded data as one piece of the one or more pieces of attribute information.
6. The three-dimensional data decoding method according to claim 5.
9. A processor; A memory. The processor uses the memory to: generating encoded data by encoding point cloud data composed of a plurality of three-dimensional points; generating a bitstream including the encoded data and control information; the encoded data includes a plurality of point data each corresponding to one of the plurality of three-dimensional points; When the plurality of point data includes one or more point data to which sensor identification information is assigned, the control information indicates that the encoded data includes the sensor identification information. Three-dimensional data encoding device.
10. A processor; A memory. The processor uses the memory to: Obtaining coded data and control information from the bitstream; Decoding the encoded data to obtain point cloud data composed of a plurality of three-dimensional points; the encoded data includes a plurality of point data each corresponding to one of the plurality of three-dimensional points; When the plurality of point data includes one or more point data to which sensor identification information is assigned, the control information indicates that the encoded data includes the sensor identification information. Three-dimensional data decoding device.
Citation Information
Patent Citations
Three-dimensional data coding method, three-dimensional data decoding method, three-dimensional data coding device, and three-dimensional data decoding device
WO2017104115A1
Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device
WO2019078292A1
Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device
WO2019082837A1
Map display device
WO2014020663A1
