Decoding device, decoding method, encoding device, encoding method, and device

Abstract

A decoding device (1320) comprises a circuit (1321) and a memory (1322) connected to the circuit (1321). During operation, the circuit (1321): restores a plurality of first vertices on the basis of encoded data (S1311); connects the plurality of restored first vertices to generate first face information that includes a first face (S1313); restores a plurality of second vertices on the basis of the encoded data and the first face (S1314); connects a point group that includes the plurality of restored first vertices and the plurality of restored second vertices to generate second face information that includes a second face (S1315); and outputs the plurality of first vertices, the plurality of second vertices, the first face information, and the second face information (S1316).

Description

Decoding device, decoding method, encoding device, encoding method, and device

[0001] The present disclosure relates to a decoding device, a decoding method, a coding device, a coding method, and an apparatus.

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a representation called a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. In a point cloud, the position and color of the points are stored. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just as with two-dimensional moving images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] Furthermore, compression of point clouds is partially supported by a public library (Point Cloud Library) that performs processing related to point clouds.

[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1).

[0006] International Publication No. 2014 / 020663

[0007] The present disclosure aims to provide a decoding device and the like that can output three-dimensional data with different resolutions.

[0008] A decoding device according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit, and in operation, the circuit acquires encoded data, reconstructs a plurality of first vertices based on the encoded data, connects the reconstructed plurality of first vertices to generate first surface information including a first surface, reconstructs a plurality of second vertices based on the encoded data and the first surface, connects points including the reconstructed plurality of first vertices and the plurality of second vertices to generate second surface information including a second surface, and outputs the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information.

[0009] An encoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, determines a plurality of first vertices of a three-dimensional object, connects the plurality of first vertices to generate first surface information including a first surface, determines a plurality of second vertices based on the first surface, connects a point cloud including the plurality of first vertices and the plurality of second vertices to generate second surface information including a second surface, and generates encoded data including the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information.

[0010] An apparatus according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit. In operation, the circuit determines a type of three-dimensional data to be decoded and extracts three-dimensional data of the determined type from encoded data. The encoded data includes (1) a plurality of first vertices of a three-dimensional object, (2) first surface information including a first surface generated by connecting the plurality of first vertices, (3) a plurality of second vertices determined based on the first surface, and (4) a second surface generated by connecting a point cloud including the plurality of first vertices and the plurality of second vertices. The types of the extracted three-dimensional data include a first format and a second format. If the type of three-dimensional data to be decoded is determined to be the first format, first encoded data including the plurality of first vertices and the first surface information is extracted. If the type of three-dimensional data to be decoded is determined to be the second format, second encoded data including extended information for decoding the first encoded data and the plurality of second vertices and the second surface is extracted.

[0011] These comprehensive or specific aspects may be realized as a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0012] The decoding device and the like of the present disclosure can output three-dimensional data at different resolutions.

[0013] FIG. 1 is a diagram showing an example of the configuration of a 3D data encoding / decoding system according to the first embodiment. FIG. 2 is a diagram showing the configuration of point cloud data according to the first embodiment. FIG. 3 is a diagram showing an example of the configuration of a data file in which information about point cloud data is described according to the first embodiment. FIG. 4 is a diagram showing the configuration of 3D mesh data according to the first embodiment. FIG. 5 is a diagram showing an example of the configuration of a data file in which information about 3D mesh data is described according to the first embodiment. FIG. 6 is a diagram for explaining a 3D model according to the first embodiment. FIG. 7 is a diagram showing types of 3D data according to the first embodiment. FIG. 8 is a diagram for explaining the encoding process of 3D data according to the first embodiment. FIG. 9 is a diagram for explaining the decoding process of 3D data according to the first embodiment. FIG. 10 is a diagram showing a two-dimensional schematic diagram of tiles and slices of 3D data according to the first embodiment. FIG. 11 is a diagram showing an example of a terminal presentation screen that can be switched in response to a user request according to the first embodiment. FIG. 12 is a diagram showing an example of a terminal presentation screen that can be automatically switched in response to a user operation according to the first embodiment. FIG. 13 is a block diagram showing an example of the functional configuration of a server and a terminal according to the first embodiment. Fig. 14 is a block diagram showing another example of the data generation unit of the server in Embodiment 1. Fig. 15 is a diagram illustrating the coordinate system synchronization process in Embodiment 1. Fig. 16 is a diagram illustrating the coordinate system synchronization process in Embodiment 1. Fig. 17 is a diagram illustrating the relationship between three-dimensional space and coded data in Embodiment 1. Fig. 18 is a diagram illustrating an example of the syntax of a coding method unit in Embodiment 1. Fig. 19 is a diagram illustrating an example of the syntax of a coded point group in Embodiment 1. Fig. 20 is a diagram illustrating an example of the syntax of a coded mesh in Embodiment 1. Fig. 21 is a diagram illustrating an example of the syntax of a coded three-dimensional model in Embodiment 1. Fig. 22 is a diagram illustrating an example of the syntax of three-dimensional data information in Embodiment 1. Fig. 23 is a diagram illustrating the data structure of a coded point group in Embodiment 1.FIG. 24 is a diagram illustrating the data structure of an encoded mesh in Embodiment 1. FIG. 25 is a diagram illustrating the data structure of an encoded 3D model in Embodiment 1. FIG. 26 is a two-dimensional diagram illustrating an example of a plurality of 3D spaces in Embodiment 1. FIG. 27 is a diagram illustrating an example of a bounding box in Embodiment 1. FIG. 28 is a diagram illustrating an example of the syntax of 3D space information in Embodiment 1. FIG. 29 is a flowchart illustrating an example of partial decoding in Embodiment 1. FIG. 30 is a diagram illustrating an example of a 3D space region to be subjected to partial decoding in Embodiment 1. FIG. 31 is a diagram illustrating an example of the data structure of an encoded point group to be partially decoded in Embodiment 1. FIG. 32 is a diagram illustrating an example of the data structure of an encoded mesh to be partially decoded in Embodiment 1. FIG. 33 is a diagram illustrating an example of the data structure of an encoded 3D model to be partially decoded in Embodiment 1. FIG. 34 is a diagram illustrating an example of coordinate systems for different types of 3D data that are not spatially synchronized in Embodiment 1. FIG. 35 is a diagram illustrating an example of the syntax of 3D data information in Embodiment 1. FIG. 36 is a diagram showing an example of the syntax of three-dimensional space information in Embodiment 1. FIG. 37 is a diagram showing an example of the functional configuration of a terminal in Embodiment 1. FIG. 38 is a flowchart showing an example of spatial synchronization processing in Embodiment 1. FIG. 39 is a diagram showing an example of the configuration of a decoding device in Embodiment 1. FIG. 40 is a flowchart showing an example of a decoding method by the decoding device in Embodiment 1. FIG. 41 is a flowchart showing another example of a decoding method by the decoding device. FIG. 42 is a diagram showing an example of the configuration of an encoding device. FIG. 43 is a flowchart showing an example of an encoding method by the encoding device. FIG. 44 is a diagram illustrating partial decoding of part of three-dimensional data in Embodiment 2. FIG. 45 is a diagram illustrating an example of extracting and decoding part of three-dimensional data in Embodiment 2. FIG. 46 is a block diagram showing an example of the functional configuration of a server and a terminal in Embodiment 2. FIG. 47 is a flowchart showing an example of processing by the terminal in Embodiment 2.FIG. 48 is a diagram showing an example of an original point cloud in the second embodiment. FIG. 49 is a diagram showing an example of a pruned octree in the second embodiment. FIG. 50 is a diagram showing an example of a two-dimensional display of a leaf node in the second embodiment. FIG. 51 is a diagram for explaining a method of generating a centroid vertex in the second embodiment. FIG. 52 is a diagram for explaining a method of generating a centroid vertex in the second embodiment. FIG. 53 is a diagram showing an example of vertex information in the second embodiment. FIG. 54 is a diagram showing an example of a Trithorpe surface in the second embodiment. FIG. 55 is a diagram for explaining a point cloud restoration process in the second embodiment. FIG. 56 is a diagram showing an example of the configuration of a decoding unit in the second embodiment. FIG. 57 is a diagram showing an example of the syntax of metadata (SEI) in an outputtable format in the second embodiment. FIG. 58 is a diagram showing an example of format_type in the second embodiment. FIG. 59 is a diagram showing another example of format_type in the second embodiment. FIG. 60 is a flowchart showing an example of the details of the process of decoding 3D data in a format specified by a predetermined method in the second embodiment. FIG. 61 is a block diagram showing another example of the functional configuration of a server and a terminal according to the second embodiment. FIG. 62 is a flowchart showing another example of processing by a terminal according to the second embodiment. FIG. 63 is a diagram showing an example of a data unit structure according to the second embodiment. FIG. 64 is a diagram showing an example of the syntax of a coding scheme unit according to the second embodiment. FIG. 65 is a diagram showing an example of type information according to the second embodiment. FIG. 66 is a diagram showing an example of a data unit structure for partial transmission according to the second embodiment. FIG. 67 is a diagram showing an example of a data unit structure for partial transmission according to the second embodiment. FIG. 68 is a diagram showing an example of a data unit structure for partial transmission according to the second embodiment. FIG. 69 is a diagram showing an example of the syntax of a coding scheme unit for partial transmission according to the second embodiment. FIG. 70 is a diagram showing an example of type information according to the second embodiment for partial transmission according to the second embodiment. FIG. 71 is a diagram for explaining a group ID according to the second embodiment.Fig. 72 is a diagram showing an example of the configuration of a decoding device in Embodiment 2. Fig. 73 is a flowchart showing an example of a decoding method by the decoding device in Embodiment 2. Fig. 74 is a diagram showing an example of the configuration of a coding device in Embodiment 2. Fig. 75 is a flowchart showing an example of a coding method by the coding device in Embodiment 2. Fig. 76 is a flowchart showing an example of a method by the decoding device or coding device in Embodiment 2.

[0014] A decoding device according to a first aspect of the present disclosure includes a circuit and a memory connected to the circuit, and in operation, the circuit acquires encoded data, reconstructs a plurality of first vertices based on the encoded data, connects the reconstructed first vertices to generate first surface information including a first surface, reconstructs a plurality of second vertices based on the encoded data and the first surface, connects points including the reconstructed first vertices and the plurality of second vertices to generate second surface information including a second surface, and outputs the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information.

[0015] This allows low-resolution information including multiple first vertices and first surface information, and high-resolution information including multiple second vertices and second surface information to be generated and output in two stages, making it possible, for example, to output low-resolution three-dimensional data based on the low-resolution information, or to output high-resolution three-dimensional data with a higher resolution than the low-resolution three-dimensional data based on the low-resolution information and high-resolution information.

[0016] A decoding device according to a second aspect of the present disclosure is a decoding device according to the first aspect, wherein the circuit further generates point cloud data including a plurality of points based on the second surface and outputs the point cloud data.

[0017] A decoding device according to a third aspect of the present disclosure is a decoding device according to the first or second aspect, wherein the encoded data includes first encoded data including the plurality of first vertices and the first face information, and second encoded data including extension information for decoding the plurality of second vertices and the second face.

[0018] A decoding device according to a fourth aspect of the present disclosure is a decoding device according to the third aspect, wherein the encoded data includes a header and a data unit, and the data unit includes the first encoded data and the second encoded data in that order.

[0019] A decoding device according to a fifth aspect of the present disclosure is a decoding device according to the third aspect, wherein the encoded data includes a first data unit including a first header and the first encoded data, and a second data unit including a second header and the second encoded data.

[0020] A decoding device according to a sixth aspect of the present disclosure is a decoding device according to the third aspect, wherein the circuit further decodes a portion of the first encoded data and the second encoded data, and outputs either (1) the first encoded data or (2) the first encoded data or the second encoded data.

[0021] A decoding device according to a seventh aspect of the present disclosure is a decoding device according to the sixth aspect, wherein the circuit further acquires numerical information indicating the number of formats that can be output, and determines the data to output based on the numerical information.

[0022] A decoding device according to an eighth aspect of the present disclosure is a decoding device according to the sixth aspect, wherein the circuit further acquires type information indicating the type of format that can be output, and determines the type information and the data to be output.

[0023] A decoding device according to a ninth aspect of the present disclosure is a decoding device according to the third aspect, wherein the circuit further acquires (1) the first encoded data and (2) either the first encoded data or the second encoded data, and decodes and outputs the acquired (1) the first encoded data or (2) the first encoded data and the second encoded data.

[0024] A coding device according to a tenth aspect of the present disclosure includes a circuit and a memory connected to the circuit, wherein the circuit, in operation, determines a plurality of first vertices of a three-dimensional object, connects the plurality of first vertices to generate first surface information including a first surface, determines a plurality of second vertices based on the first surface, connects a point cloud including the plurality of first vertices and the plurality of second vertices to generate second surface information including the second surface, and generates coded data including the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information.

[0025] According to this, encoded data obtained by generating low-resolution information including a plurality of first vertices and first surface information and high-resolution information including a plurality of second vertices and second surface information in two stages is output, and therefore a decoding device that acquires the encoded data can, for example, output low-resolution three-dimensional data based on the low-resolution information, or output high-resolution three-dimensional data with a higher resolution than the low-resolution three-dimensional data based on the low-resolution information and high-resolution information.

[0026] An encoding device according to an eleventh aspect of the present disclosure is an encoding device according to the tenth aspect, wherein the circuit further generates point cloud data including a plurality of points based on the second surface, and the encoded data further includes the point cloud data.

[0027] A coding device according to a twelfth aspect of the present disclosure is a coding device according to the tenth aspect, wherein the circuit further generates first coded data including the plurality of first vertices and the first surface information, and generates second coded data including the plurality of second vertices and the second surface information, and the coded data includes the first coded data and the second coded data.

[0028] Therefore, a decoding device that acquires the encoded data can restore a plurality of three-dimensional data sets with different resolutions.

[0029] A coding device according to a thirteenth aspect of the present disclosure is a coding device according to the twelfth aspect, wherein the coded data includes a header and a data unit, and the data unit includes the first coded data and the second coded data in that order.

[0030] A coding device according to a fourteenth aspect of the present disclosure is a coding device according to the twelfth aspect, wherein the coded data includes a first data unit including a first header and the first coded data, and a second data unit including a second header and the second coded data.

[0031] A coding device according to a fifteenth aspect of the present disclosure is the coding device according to the fourteenth aspect, wherein the coded data includes number information indicating the number of formats that can be output.

[0032] A coding device according to a sixteenth aspect of the present disclosure is the coding device according to the fourteenth aspect, wherein the coded data includes type information indicating types of formats that can be output.

[0033] A coding device according to a seventeenth aspect of the present disclosure is a coding device according to the tenth aspect, wherein the circuit further outputs (1) the first coded data and (2) either the first coded data or the second coded data.

[0034] A decoding method according to an eighteenth aspect of the present disclosure reconstructs a plurality of first vertices based on encoded data, connects the reconstructed plurality of first vertices to generate first surface information including a first surface, reconstructs a plurality of second vertices based on the encoded data and the first surface, connects a point cloud including the reconstructed plurality of first vertices and the plurality of second vertices to generate second surface information including a second surface, and outputs the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information.

[0035] This allows low-resolution information including multiple first vertices and first surface information, and high-resolution information including multiple second vertices and second surface information to be generated and output in two stages, making it possible, for example, to output low-resolution three-dimensional data based on the low-resolution information, or to output high-resolution three-dimensional data with a higher resolution than the low-resolution three-dimensional data based on the low-resolution information and high-resolution information.

[0036] An encoding method according to a nineteenth aspect of the present disclosure determines a plurality of first vertices of a three-dimensional object, connects the plurality of first vertices to generate first surface information including a first surface, determines a plurality of second vertices based on the first surface, connects a point cloud including the plurality of first vertices and the plurality of second vertices to generate second surface information including the second surface, and generates encoded data including the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information.

[0037] According to this, encoded data obtained by generating low-resolution information including a plurality of first vertices and first surface information and high-resolution information including a plurality of second vertices and second surface information in two stages is output, and therefore a decoding device that acquires the encoded data can, for example, output low-resolution three-dimensional data based on the low-resolution information, or output high-resolution three-dimensional data with a higher resolution than the low-resolution three-dimensional data based on the low-resolution information and high-resolution information.

[0038] An apparatus according to a twentieth aspect of the present disclosure includes a circuit and a memory connected to the circuit. In operation, the circuit determines a type of three-dimensional data to be decoded and extracts three-dimensional data of the determined type from the encoded data. The encoded data includes (1) a plurality of first vertices of a three-dimensional object, (2) first surface information including a first surface generated by connecting the plurality of first vertices, (3) a plurality of second vertices determined based on the first surface, and (4) a second surface generated by connecting a point cloud including the plurality of first vertices and the plurality of second vertices. The types of the extracted three-dimensional data include a first format and a second format. When the type of three-dimensional data to be decoded is determined to be the first format, first encoded data including the plurality of first vertices and the first surface information is extracted. When the type of three-dimensional data to be decoded is determined to be the second format, second encoded data including extended information for decoding the first encoded data and the plurality of second vertices and the second surface is extracted.

[0039] Therefore, the encoded data necessary for decoding can be extracted based on the determined format. Therefore, when a format requiring a small amount of data for decoding is selected, the amount of data to be extracted can be reduced, which can reduce, for example, the amount of transmission required for transmitting the encoded data and the processing load of the decoding process by the decoding device that has acquired the encoded data.

[0040] These comprehensive or specific aspects may be realized as a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0041] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that represents a superordinate concept will be described as optional components.

[0042] (Embodiment 1) The configuration of a three-dimensional data encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 1001, a three-dimensional data decoding system 1002, a sensor terminal 1003, and an external connection unit 1004.

[0043] The three-dimensional data encoding system 1001 generates encoded data or multiplexed data by encoding three-dimensional data. Note that the three-dimensional data encoding system 1001 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 1001.

[0044] The three-dimensional data encoding system 1001 includes a three-dimensional data generation system 1011, a presentation unit 1012, an encoding unit 1013, a multiplexing unit 1014, an input / output unit 1015, and a control unit 1016. The three-dimensional data generation system 1011 includes a sensor information acquisition unit 1017 and a three-dimensional data generation unit 1018.

[0045] The sensor information acquisition unit 1017 acquires a sensor signal from the sensor terminal 1003 and outputs the sensor signal to the three-dimensional data generation unit 1018. The three-dimensional data generation unit 1018 generates three-dimensional data from the sensor signal and outputs the three-dimensional data to the encoding unit 1013.

[0046] The presentation unit 1012 presents the sensor signal or the three-dimensional data to the user. For example, the presentation unit 1012 displays information or an image based on the sensor signal or the three-dimensional data.

[0047] The encoding unit 1013 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 1014. The additional information includes, for example, a sensor signal.

[0048] The multiplexing unit 1014 generates multiplexed data by multiplexing the coded data input from the coding unit 1013, the control information, and the additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0049] The input / output unit 1015 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 1016 (or the application execution unit) controls each processing unit. That is, the control unit 1016 controls encoding, multiplexing, etc. The control unit 1016 may also control demultiplexing, decoding, or presentation.

[0050] The sensor signal may be input to the encoding unit 1013 or the multiplexing unit 1014. The input / output unit 1015 may output the three-dimensional data or the encoded data directly to the outside.

[0051] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 1001 is input to the three-dimensional data decoding system 1002 via the external connection unit 1004 .

[0052] The three-dimensional data decoding system 1002 generates three-dimensional data by decoding encoded data or multiplexed data. Note that the three-dimensional data decoding system 1002 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 1002.

[0053] The three-dimensional data decoding system 1002 includes a sensor information acquisition unit 1021 , an input / output unit 1022 , a demultiplexing unit 1023 , a decoding unit 1024 , a presentation unit 1025 , a user interface 1026 , and a control unit 1027 .

[0054] The sensor information acquisition unit 1021 acquires a sensor signal from the sensor terminal 1003 .

[0055] The input / output unit 1022 acquires a transmission signal, decodes multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 1023 .

[0056] The demultiplexing unit 1023 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoding unit 1024 .

[0057] The decoding unit 1024 reconstructs point cloud data by decoding the encoded data.

[0058] The presentation unit 1025 presents the point cloud data to the user. For example, the presentation unit 1025 displays information or an image based on the point cloud data. The user interface 1026 acquires instructions based on user operations. The control unit 1027 (or the application execution unit) controls each processing unit. That is, the control unit 1027 controls demultiplexing, decoding, presentation, etc.

[0059] The input / output unit 1022 may acquire point cloud data or encoded data directly from the outside. The presentation unit 1025 may acquire additional information such as a sensor signal and present information based on the additional information. The presentation unit 1025 may also present information based on a user instruction acquired by the user interface 1026.

[0060] The sensor terminal 1003 generates a sensor signal, which is information obtained by a sensor. The sensor terminal 1003 is a terminal equipped with a sensor or a camera, and may be, for example, a mobile object such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

[0061] The sensor signals obtainable by the sensor terminal 1003 include, for example, (1) a signal indicating the distance between the sensor terminal 1003 and an object or the reflectance of the object obtained from a LIDAR, millimeter-wave radar, or infrared sensor, and (2) a signal indicating the distance between a camera and an object or the reflectance of the object obtained from multiple monocular camera images or stereo camera images. The sensor signals may also include the attitude, orientation, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor signals may also include temperature, air pressure, humidity, magnetism, etc.

[0062] The external connection unit 1004 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0063] Next, point cloud data will be described. Fig. 2 is a diagram showing the structure of point cloud data. Fig. 3 is a diagram showing an example of the structure of a data file in which information about point cloud data is written.

[0064] Point cloud data includes data on multiple points. Each point includes location information (three-dimensional coordinates) and attribute information for that location. A collection of multiple points is called a point cloud. For example, a point cloud can represent the three-dimensional shape of an object.

[0065] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.

[0066] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0067] The data file configuration example shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.

[0068] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0069] Next, the three-dimensional mesh data will be described. Fig. 4 is a diagram showing the structure of the three-dimensional mesh data. Fig. 5 is a diagram showing an example of the structure of a data file in which information about the three-dimensional mesh data is written.

[0070] Three-dimensional mesh data is a data format used in computer graphics (CG) and represents the three-dimensional shape of an object using a collection of multiple pieces of surface information. Each piece of surface information represents a polygon such as a triangle or a quadrangle. Three-dimensional mesh data is also called a polygon or polygon mesh.

[0071] The components are a set of three-dimensional point clouds, vertices that are multiple three-dimensional points of the three-dimensional point cloud, edges connecting two vertices at multiple three-dimensional points, and faces surrounded by multiple edges. A three-dimensional point cloud is a set of points that include position information in three-dimensional space and attribute information corresponding to the position information. Note that three-dimensional points may simply be referred to as points.

[0072] A vertex may have attribute information such as color information, reflectance, and normal vector for a three-dimensional point. The relationship between the vertices that make up an edge or a face may be indicated by information called connectivity. Note that a vertex may be expressed as a position. The front and back of a face may be expressed by the direction of the normal vector for the three-dimensional point. Also, a vertex may have attribute information for the face.

[0073] An example of a mesh data file format is an object file. In a mesh data file such as that shown in Fig. 5, the position information G(1) to G(N) of the N vertices that make up the mesh and the attribute information A(1) to A(N) of the vertices are indicated as vertex information. In a mesh data file, the vertex information does not need to include attribute information.

[0074] Furthermore, the attribute information does not have to correspond one-to-one to the vertices. The mesh data file in Fig. 5 shows an example in which the three-dimensional mesh data has M pieces of attribute information A2.

[0075] Face information is represented by a combination of vertex indices, where n[1,3,4] indicates a triangular face made up of three vertices: vertex n=1, vertex n=3, and vertex n=4.

[0076] Furthermore, m[2,4,6] indicates that the attribute information of m=2, the attribute information of m=4, and the attribute information of m=6 in the attribute information A2 correspond to three vertices, respectively. Note that although an example in which a face is formed by three vertices has been shown here, the number of vertices of a face is not limited to three, as long as it is an integer equal to or greater than three. For example, if the face is a quadrangle, the number of vertices is four, and if the face is a polygon, the number of vertices is equal to the number of vertices that form the polygon.

[0077] The attribute information A2 may be represented in a file separate from the mesh data file, or may include pointer information thereto. For example, the attribute information may be stored in a two-dimensional attribute map file, and the attribute map file name and two-dimensional coordinates in the attribute map may be represented by the attribute information A2 of the mesh data file. In this way, the attribute information A2 may be included in the mesh data file or represented in a file separate from the mesh data file. Either method can be used to specify attribute information for a three-dimensional point.

[0078] Next, the three-dimensional model will be described with reference to Fig. 6.

[0079] A three-dimensional model is a model generated based on two-dimensional or three-dimensional data.

[0080] The three-dimensional model learning unit 1031 learns, for example, two-dimensional data (two-dimensional images) or three-dimensional data (point clouds or meshes) to generate a three-dimensional model, which is a network model in which three-dimensional shapes and attribute information corresponding to the three-dimensional shapes are learned using a neural network or the like.

[0081] The three-dimensional model learning unit 1031 may generate a three-dimensional model by learning using Neural Radiance Fields (NeRF) based on two-dimensional images. The three-dimensional model learning unit 1031 may generate a three-dimensional model after converting the two-dimensional images into three-dimensional data by performing photogrammetry using the two-dimensional images. The three-dimensional model may be generated using three-dimensional data acquired by a sensor (distance sensor).

[0082] The three-dimensional model data is the elements that make up the three-dimensional model, and includes information indicating the structure of the network model, feature quantities, etc. The three-dimensional model data includes, for example, information about the components of the neural network. The information about the components includes, for example, multiple layers such as an input layer, an intermediate layer, and an output layer, nodes in each layer, weight coefficients for the nodes, and transformation functions for the nodes.

[0083] The three-dimensional model encoding unit 1032 may encode the three-dimensional model data and transmit the encoded three-dimensional model data.

[0084] The three-dimensional model decoding unit 1033 receives the transmitted encoded three-dimensional model data and decodes the three-dimensional model based on the encoded three-dimensional model data.

[0085] The rendering reconstruction unit 1034 reconstructs (generates) two-dimensional data (two-dimensional image) or three-dimensional data (point cloud or mesh) based on the decoded three-dimensional model. For example, when using a three-dimensional model modeled using NeRF, the rendering reconstruction unit 1034 acquires viewpoint position or line-of-sight vector information, generates rendered two-dimensional data (two-dimensional image) based on the three-dimensional model and the viewpoint position or line-of-sight vector, and outputs the two-dimensional data. The generated two-dimensional data represents a three-dimensional object as seen from the viewpoint position, or a two-dimensional image of the three-dimensional object as seen from the line of sight indicated by the line-of-sight vector. The three-dimensional object is the three-dimensional object of the subject that was the source of the two-dimensional data or three-dimensional data input to the three-dimensional model learning unit 1031.

[0086] Next, the types of three-dimensional data will be explained. Fig. 7 shows the types of three-dimensional data. As shown in Fig. 7, three-dimensional data includes static objects and dynamic objects.

[0087] A static object is three-dimensional data at any time (a certain time). A dynamic object is three-dimensional data that changes over time. Hereinafter, point cloud data at a certain time will be referred to as a PCC frame or frame. Mesh data at a certain time will be referred to as a mesh frame or frame.

[0088] The object may be three-dimensional data with a certain area restriction, such as ordinary video data, or may be three-dimensional data with no area restriction, such as map information.

[0089] There may also be sparse point cloud data (sparse mesh data) and dense point cloud data (dense mesh data) with points of various densities.

[0090] Each processing unit will be described in detail below. Sensor information is acquired by various methods, such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The three-dimensional data generation unit 1018 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 1017. The three-dimensional data generation unit 1018 generates position information (geometry information) as the point cloud data, and adds attribute information for the position information to the position information.

[0091] The three-dimensional data generation unit 1018 may process the point cloud data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 1018 may reduce the amount of data by deleting point clouds with overlapping positions. Furthermore, the three-dimensional data generation unit 1018 may convert the position information (such as by shifting, rotating, or normalizing) or process the point cloud data to generate mesh data. Furthermore, the three-dimensional data generation unit 1018 may render the attribute information.

[0092] In FIG. 1, the three-dimensional data generation system 1011 is included in the three-dimensional data encoding system 1001, but it may be provided independently outside the three-dimensional data encoding system 1001.

[0093] The encoding unit 1013 generates encoded data by encoding the three-dimensional data based on a predetermined encoding method. The encoding methods include G-PCC (an encoding method using position information), V-PCC (an encoding method using a video codec), Draco (a mesh encoding method), and V-DMC (a mesh encoding method). The encoding method is not limited to these methods, and may be, for example, a method of encoding a dynamic mesh, or another method that combines these methods.

[0094] The decoding unit 1024 decodes the coded data based on a predetermined coding method to decode the three-dimensional data.

[0095] The multiplexing unit 1014 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the encoded data of the three-dimensional data, the multiplexing unit 1014 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 1014 may also multiplex attribute information related to sensor information or point cloud data.

[0096] Multiplexing methods or file formats include ISOBMFF, and ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0097] The demultiplexing unit 1023 extracts the coded data of the three-dimensional data, other media, time information, and the like from the multiplexed data.

[0098] The input / output unit 1015 transmits the multiplexed data using a method suited to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 1015 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0099] The communication protocol used may be http, ftp, TCP, UDP, etc. A pull-type communication method or a push-type communication method may be used.

[0100] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter waves, etc. are used.

[0101] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0102] Next, a process of dividing three-dimensional data into one or more three-dimensional data will be described. Fig. 8 is a diagram for explaining a process of encoding three-dimensional data. Fig. 9 is a diagram for explaining a process of decoding three-dimensional data.

[0103] 8 , the data dividing unit 1041 divides the three-dimensional data into one or more three-dimensional spaces and generates one or more divided three-dimensional data (i.e., one or more divided three-dimensional data). The encoding unit 1042 may encode one or more divided three-dimensional data to generate encoded data. The data dividing unit 1041 and the encoding unit 1042 may be included in a single encoding device as components of the single encoding device, or may be included in separate devices.

[0104] Each of the one or more three-dimensional spaces may be referred to as a tile or a space. The three-dimensional space may be, for example, a bounding box. Furthermore, the three-dimensional data included in each of the divided three-dimensional spaces may be referred to as a slice. A slice is divided three-dimensional data and includes any of a point cloud, a mesh, and a three-dimensional model having position information (geometry) or attribute information (attribute). Each of the multiple slices is coded by the coding unit 1042 for each component element and output as coded data. The coded data includes the coded multiple slices.

[0105] As shown in FIG. 9 , in the decoding process, the decoding unit 1051 decodes one or more pieces of divided three-dimensional data (one or more slices) based on the encoded data. The data combining unit 1052 combines the one or more pieces of divided three-dimensional data to restore (generate) three-dimensional data. The decoding unit 1051 and the data combining unit 1052 may be included in a single decoding device as components of the single decoding device, or may be included in separate devices. The one or more pieces of divided three-dimensional data decoded by the decoding unit 1051 do not need to be combined. The decoding unit 1051 may decode some of the one or more pieces of divided three-dimensional data based on part of the encoded data and output the decoded some of the divided three-dimensional data. In this case, the decoding device does not need to have the data combining unit 1052.

[0106] FIG. 10 is a two-dimensional schematic diagram showing tiles and slices of three-dimensional data.

[0107] When encoding multiple slices, the encoding device may encode them using a dependency relationship between the multiple slices, or may encode them without using the dependency relationship. When encoding without using the dependency relationship, the encoding device can encode each slice independently, and can reduce processing time by encoding multiple slices using parallel processing. Furthermore, when multiple slices are encoded without using the dependency relationship, the decoding device can decode each slice independently, and can reduce processing time by decoding multiple slices using parallel processing. Furthermore, the decoding device can reduce the amount of processing by performing partial decoding, which decodes some of the multiple slices.

[0108] When encoding using a dependency relationship, the encoding device signals an identifier indicating the dependency relationship and encodes the data in order starting with the dependent data. When multiple slices are encoded using a dependency relationship, the decoding device decodes the data in order starting with the dependent data based on the identifier.

[0109] The division of the three-dimensional data may be performed using any number of divisions and any division method. The division of the three-dimensional data may involve determining the shape of an object and dividing each object into multiple three-dimensional points. The division of the three-dimensional data may also be performed based on the number of three-dimensional points included in a slice. In other words, an upper limit on the number of three-dimensional points included in one slice may be set. The three-dimensional data may also be divided based on whether or not it is included in the three-dimensional space (tile information) using map information or location information. Multiple tile shapes may overlap.

[0110] By dividing the three-dimensional data into a plurality of divided three-dimensional data in this way, adaptive encoding according to the content or object and parallel processing in decoding become possible.

[0111] Next, a method for selecting three-dimensional data to be presented or transmitted from among a plurality of three-dimensional data will be described.

[0112] The server stores multiple pieces of three-dimensional data for the same space. The server stores, for example, point cloud data and mesh data for the same space. The server is an example of an encoding device. The terminal switches the three-dimensional data acquired from the server based on the purpose of the terminal and presents the switched three-dimensional data. The terminal may be, for example, a terminal that analyzes three-dimensional data. In this case, the terminal may switch the three-dimensional data to be presented based on the purpose of analysis or presentation, or user operation. The terminal is an example of a decoding device.

[0113] The switching of three-dimensional data may involve switching between presenting a point cloud or a mesh as the three-dimensional data. Furthermore, the switching of three-dimensional data may involve switching between transmitting a point cloud or a mesh as the three-dimensional data. For example, the terminal may transmit a user's selection result to a server, receive (download) three-dimensional data based on the selection result from the server, and present the received three-dimensional data. The three-dimensional data (point cloud or mesh) may or may not be encoded by the server. If the three-dimensional data is encoded, the terminal may receive the encoded three-dimensional data from the server, decode the three-dimensional data based on the received encoded three-dimensional data, and present the decoded three-dimensional data.

[0114] FIG. 11 is a diagram showing an example of a terminal presentation screen that can be switched in response to a user request.

[0115] As shown in FIG. 11 , the terminal presentation screen 1061 may be switched in response to a user request. The terminal presentation screen includes a point cloud button 1061a and a mesh button 1061b as UIs for accepting user requests. The point cloud button 1061a is a button for accepting a request to present an encoded point cloud (encoded point cloud data). The mesh button 1061b is a button for accepting a request to present an encoded mesh (encoded mesh data). When the user inputs the point cloud button 1061a, the terminal accepts the request to present the encoded point cloud and notifies (transmits) the request as a result of the selection of presentation data to the server. When the user inputs the mesh button 1061b, the terminal accepts the request to present an encoded mesh and notifies (transmits) the request as a result of the selection of presentation data to the server. Note that presentation may also be expressed as display.

[0116] When the server receives the request as the selection result, it transmits three-dimensional data according to the selection result to the terminal. If the received request is a request to present an encoded point cloud, the server transmits the encoded point cloud to the terminal. If the received request is a request to present an encoded mesh, the server transmits the encoded mesh to the terminal.

[0117] Meshes are suitable for presentation because they require a small processing load. On the other hand, point clouds are suitable for measurement because they provide highly accurate positional information of 3D models represented by 3D point clouds. For example, by selecting the type of 3D data appropriate for the required purpose, such as selecting a mesh when observing a 3D model and a point cloud when measuring, a user can reduce the presentation processing load or use the data for accurate measurements.

[0118] FIG. 12 is a diagram showing an example of a terminal presentation screen that is automatically switched in response to a user operation.

[0119] The terminal performs a process of enlarging the three-dimensional data by accepting a user operation. The terminal may present mesh data as shown on a terminal presentation screen 1062 at a magnification smaller than a predetermined magnification, and may present point cloud data as shown on a terminal presentation screen 1063 at a magnification equal to or larger than the predetermined magnification. For example, the terminal may present mesh data at a magnification smaller than the predetermined magnification, and when the mesh data is enlarged to the predetermined magnification in response to a user operation and presented, download point cloud data from a server corresponding to a portion of the three-dimensional data that is highlighted by the enlargement (presented by the enlargement), and decode and present the downloaded point cloud data.

[0120] Although the above example illustrates switching the type of 3D data presented in response to an operation to enlarge or reduce the 3D data, other operations may also be used to switch the type of 3D data presented. For example, when a terminal receives a user's tap on a portion of presented mesh data, the terminal may switch to and present point cloud data corresponding to the portion of the mesh data selected by the tap. The terminal may present the point cloud data superimposed on the mesh data, or may present the point cloud data independently from the mesh data (i.e., without presenting the mesh data). The terminal may predict the enlarged portion before presentation at a predetermined magnification, download the point cloud data corresponding to the enlarged portion, and decode and present the downloaded point cloud data. Furthermore, if the point cloud data and the mesh data have been spatially synchronized (coordinate alignment) in advance, the terminal may present the downloaded point cloud data as is without spatial synchronization. If the point cloud data and the mesh data have not been spatially synchronized, the terminal may align the point cloud data with the mesh data (or the coordinate system of the mesh data) based on synchronization information related to spatial synchronization and present the point cloud data. The terminal may also present the point cloud data without alignment. Although the above describes an example in which point cloud data is spatially synchronized with mesh data, the terminal may also present mesh data in spatial synchronization with point cloud data.

[0121] In this way, the terminal downloads only the necessary point cloud data, decodes the downloaded point cloud data, and presents it. This is expected to reduce the communication volume between the server and the terminal, reduce the processing load on the terminal, and reduce the presentation delay.

[0122] Generally, mesh data has a small data size, while point cloud data has a large data size. In other words, when comparing mesh data and point cloud data of the same 3D object in terms of data size, the mesh data is smaller than the point cloud data. Therefore, by first downloading the mesh data and displaying it initially, the terminal can reduce the delay until initial display.

[0123] In addition, by displaying mesh data, the terminal can reduce the processing load required for observing the 3D model. Also, by displaying mesh data and point cloud data simultaneously, the terminal can improve visibility and make measurements easier.

[0124] The 3D data to be presented may be switched based on the communication network bandwidth. The terminal may present mesh data when the bandwidth is narrow and the network speed is slower than a predetermined speed, and present point cloud data when the network speed is equal to or faster than the predetermined speed.

[0125] Furthermore, the 3D data to be presented may be switched depending on the capabilities of the terminal. For example, the 3D data to be presented may be switched based on the processing performance of the terminal or the usage rate of the CPU of the terminal. While presenting point cloud data, the terminal may switch from presenting point cloud data to presenting mesh data if the terminal's processing performance or CPU resources are insufficient to process the point cloud data. While presenting mesh data, the terminal may switch from presenting mesh data to presenting point cloud data if the terminal's processing performance or CPU resources are determined to be sufficient to process the mesh data.

[0126] Next, another example of switching three-dimensional data will be described.

[0127] When presenting three-dimensional data used at a construction site, the terminal may switch the type of three-dimensional data presented, such as presenting the exterior of a building or map as mesh data, and presenting point cloud data when the data includes areas that require measurement, such as distortions, cracks, and warping of the building.

[0128] The terminal can switch the type of three-dimensional data it presents, such as presenting mesh data when it wants to show the appearance of spaces such as stadiums, halls, and factories, including lighting brightness, color, and atmosphere, and presenting point cloud data when it wants to measure studio shape, equipment placement, aisle clearance, and so on.

[0129] Also, three-dimensional data with different resolutions may be switched and presented.

[0130] When rendering and presenting a 3D model, the terminal may switch between multiple pieces of point cloud data with different resolutions, multiple pieces of mesh data with different resolutions, and the 3D model. For example, the terminal may present low-resolution mesh data when presenting 3D data far from the viewpoint, and switch to high-resolution mesh data when the viewpoint is closer. This can improve the accuracy of the 3D data display.

[0131] When analyzing the movements of athletes in sports, the terminal may present mesh data until a specific movement or location is identified, and then switch to presenting point cloud data of the corresponding part after the specific movement or location is identified, and use the point cloud data to measure the three-dimensional shape.

[0132] The terminal may switch between 3D models with different resolutions in a video game or the like. For example, the terminal may use high-resolution 3D data to present 3D data of important parts and low-resolution 3D data to present 3D data of unimportant parts. This can improve overall processing performance. Note that whether each 3D model is important is determined, for example, by whether it is related to the action of the video game (e.g., in a shooting game, the player, target, gun, etc. are important) or whether it is related to the player's selection (e.g., product information is important in a purchasing scene). Note that whether each 3D model is important may be set in advance for each video game.

[0133] When displaying the terrain data as a three-dimensional model, the terminal may switch between terrain data with different resolutions, thereby enabling more detailed terrain information to be displayed as needed.

[0134] For example, when displaying thumbnails on the web, the terminal may present low-resolution three-dimensional data, and when measuring or presenting in an application using three-dimensional data selected from the thumbnail, high-resolution three-dimensional data may be used for measurement or presentation. For example, when displayed on a two-dimensional display terminal, the three-dimensional data is presented as two-dimensional data of a three-dimensional object (subject) viewed from a specific viewpoint in a specific direction. In other words, the three-dimensional data may be divided into camera information indicating a specific viewpoint and a specific direction and two-dimensional data of a three-dimensional object viewed from the specific viewpoint in the specific direction.

[0135] When display is the priority, the terminal presents mesh data, which requires less processing, and when measuring, replaces it with point cloud data suitable for measurement, making it possible to realize applications with less processing.

[0136] Next, a description will be given of the configuration of the server 1070 and the terminal 1090. Fig. 13 is a block diagram showing an example of the functional configuration of the server and the terminal.

[0137] The server 1070 includes a data generation unit 1071 , a synchronization unit 1075 , a point cloud encoding unit 1076 , a mesh encoding unit 1077 , a model encoding unit 1078 , a multiplexing unit 1079 , and a data extraction unit 1080 .

[0138] The data generation unit 1071 generates three-dimensional data based on at least one of two-dimensional data and three-dimensional data. The generated three-dimensional data includes at least two of point cloud data, mesh data, and three-dimensional model data. The data generation unit 1071 includes a point cloud generation unit 1072, a mesh generation unit 1073, and a model generation unit 1074. The data generation unit 1071 may include at least two of the point cloud generation unit 1072, the mesh generation unit 1073, and the model generation unit 1074. The point cloud generation unit 1072 generates point cloud data based on at least one of the two-dimensional data and the three-dimensional data. The mesh generation unit 1073 generates mesh data based on at least one of the two-dimensional data and the three-dimensional data. The model generation unit 1074 generates three-dimensional model data by performing machine learning based on at least one of the two-dimensional data and the three-dimensional data.

[0139] The two-dimensional data input to the data generation unit 1071 may be a two-dimensional image acquired by a camera. The three-dimensional data input to the data generation unit 1071 may be point cloud data acquired by a sensor such as LiDAR of a space such as a construction site, factory, or office. The data generation unit 1071 may generate color information corresponding to each point included in the point cloud data of the three-dimensional data as attribute information using a two-dimensional image of the two-dimensional data. The three-dimensional data generated by the data generation unit 1071 may be divided into any space. The point cloud data, mesh data, and three-dimensional model data may each be divided into any space.

[0140] The synchronization unit 1075 synchronizes the spatial positions of the point cloud data, mesh data, and 3D model data generated by the data generation unit 1071 or the time of each data (playback time, decoding time, acquisition time, etc.). The time of each data is the playback time, decoding time, acquisition time, etc. The synchronization unit 1075 may generate synchronization information for synchronization without synchronizing the point cloud data, mesh data, and 3D model data. The synchronization unit 1075 may synchronize at least two types of 3D data among the point cloud data, mesh data, and 3D model data generated by the data generation unit 1071, or may perform processing to generate synchronization information (synchronization signal) for synchronization, and may not necessarily perform processing to synchronize three types of 3D data (synchronization processing).

[0141] The point cloud encoding unit 1076 encodes the point cloud data after synchronization processing has been performed by the synchronization unit 1075. Note that the point cloud encoding unit 1076 does not necessarily have to encode the point cloud data. The point cloud data may be encoded in advance, or may be encoded in response to a request from the terminal 1090.

[0142] The mesh encoding unit 1077 encodes the mesh data after the synchronization process has been performed by the synchronization unit 1075 .

[0143] The model encoding unit 1078 encodes the three-dimensional model data after the synchronization process has been performed by the synchronization unit 1075 .

[0144] The multiplexing unit 1079 multiplexes the encoded point cloud data (encoded point cloud), the encoded mesh data (encoded mesh data), the encoded 3D model data, and the synchronization information using a predetermined format or a predetermined multiplexing method. Note that multiplexing by the multiplexing unit 1079 does not have to be performed. In this case, the server 1070 does not have to be equipped with the multiplexing unit 1079.

[0145] The data extraction unit 1080 extracts a portion of the multiplexed three-dimensional data in response to a request from the terminal 1090, and transmits the extracted portion of the three-dimensional data to the terminal 1090. Note that data extraction by the data extraction unit 1080 does not have to be performed. In this case, the server 1070 does not have to be equipped with the data extraction unit 1080. If data extraction by the data extraction unit 1080 is not performed, the server 1070 may transmit to the terminal 1090 the three-dimensional data multiplexed by the multiplexing unit 1079. Furthermore, if multiplexing by the multiplexing unit 1079 is not performed, the server 1070 may transmit to the terminal 1090 the coded point cloud data (coded point cloud), coded mesh data (coded mesh), coded three-dimensional model data (coded three-dimensional model), and synchronization information, or may transmit to the terminal 1090 a bitstream including the coded point cloud data (coded point cloud), coded mesh data (coded mesh), coded three-dimensional model data (coded three-dimensional model), and synchronization information.

[0146] The terminal 1090 includes a control unit 1091 , a decoding unit 1092 , and a presentation unit 1093 .

[0147] The control unit 1091 transmits a request for the part of the three-dimensional data to be presented to the server 1070. The control unit 1091 may accept an operation by the user to specify the part of the three-dimensional data.

[0148] The decoding unit 1092 decodes a portion of the three-dimensional data based on the bit stream (encoded data) acquired from the server 1070 .

[0149] The presentation unit 1093 renders and presents a portion of the decoded three-dimensional data.

[0150] The data generation unit 1071 in Fig. 13 may be realized by a data generation unit 1110 shown in Fig. 14. Fig. 14 is a block diagram showing another example of the data generation unit of the server.

[0151] The data generation unit 1110 includes a point cloud generation unit 1111 , a mesh generation unit 1112 , and a model generation unit 1113 .

[0152] The point cloud generation unit 1111 has the same function as the point cloud generation unit 1072. The point cloud generation unit 1111 acquires point cloud data obtained from the point cloud sensor 1101 and a two-dimensional image obtained from the camera 1102, and generates point cloud data based on the point cloud data and the two-dimensional image. The point cloud data generated by the point cloud generation unit 1111 includes position information of each point and attribute information corresponding to each point indicated by the position information, and includes attribute information (such as color information) extracted from the two-dimensional image.

[0153] The mesh generation unit 1112 generates mesh data based on the point cloud data generated by the point cloud generation unit 1111 .

[0154] The model generation unit 1113 has the same functions as the model generation unit 1074. The model generation unit 1113 acquires point cloud data obtained from the point cloud sensor 1101 and two-dimensional images obtained from the camera 1102, and performs machine learning based on the point cloud data and the two-dimensional images to generate three-dimensional model data.

[0155] The point cloud data, mesh data, and three-dimensional model data may be data generated independently of each other, as described in Fig. 13. The mesh data may be generated from the point cloud data, as described in Fig. 14. Note that the point cloud data may be generated from the mesh data.

[0156] A mesh may be generated from a point cloud, or a point cloud may be generated from a mesh.

[0157] The point cloud data, mesh data, and three-dimensional model data may be generated by the server 1070, or may be generated by a sensor or a terminal 1090 equipped with a sensor. The sensor is, for example, a point cloud sensor 1101 and a camera 1102.

[0158] Next, a process for synchronizing the coordinate systems of a plurality of three-dimensional data will be described. Figures 15 and 16 are diagrams for explaining the process for synchronizing the coordinate systems.

[0159] Even when the three-dimensional data is for the same space, the origin position (origin coordinates) of the local coordinate system handled by the system may differ from the actual coordinates due to differences in the system or coordinate system being handled.

[0160] For example, if mesh data is generated from point cloud data, the origin position of the mesh data and the point cloud data will be the same, but if the mesh data and the point cloud data are generated using different systems, the origin positions of the mesh data and the point cloud data may be different.

[0161] 15 and 16, the origin coordinates of the coordinate system of the mesh data and the origin coordinates of the coordinate system of the point cloud data are both, for example, (x1, y1, z1) in world coordinates, and the coordinate system of the mesh data and the coordinate system of the point cloud data are the same. Note that if these origin coordinates are not the same, they may be corrected to be the same.

[0162] Next, a case where divided three-dimensional space information is synchronized will be described.

[0163] 15 shows a case where the three-dimensional spatial regions for dividing the mesh data and the three-dimensional spatial regions for dividing the point cloud data are the same. Specifically, the number, size, and positions of the three-dimensional spatial regions are the same between the multiple three-dimensional spatial regions in the mesh data coordinate system and the multiple three-dimensional spatial regions in the point cloud coordinate system.

[0164] Focusing on the three-dimensional space area indicated by the thick dotted line, the positions of the origin (black triangle mark) of the bounding box representing the three-dimensional space and the maximum value point (black square mark) of the bounding box are the same in the coordinate system of the mesh data and the coordinate system of the point cloud data.

[0165] In this case, the vertical, horizontal, and depth sizes of this bounding box are the same in the coordinate system of the mesh data and the coordinate system of the point cloud data.

[0166] 16 , the number, size, and positions of the multiple three-dimensional spatial regions in the mesh data coordinate system and the multiple three-dimensional spatial regions in the point cloud coordinate system do not have to be completely identical. The three-dimensional spatial regions in the mesh data coordinate system may be divided into larger regions, and the three-dimensional spatial regions in the point cloud data coordinate system may be further divided into the three-dimensional spatial regions in the mesh data coordinate system. In this example, the combined unit of the multiple three-dimensional spatial regions in the point cloud data coordinate system is identical to the three-dimensional spatial region in the mesh data coordinate system.

[0167] Next, the relationship between the three-dimensional space and the coded data will be described with reference to Fig. 17.

[0168] As described above, the three-dimensional data includes, for example, any one of point cloud data, mesh data, and a three-dimensional model.

[0169] As shown in Fig. 17, when three-dimensional data is divided into three pieces of three-dimensional data in three three-dimensional spaces (tiles or spaces), the encoding device encodes each of the three divided pieces of three-dimensional data, attaches a header to each piece, and creates a data unit. The header signals (assigns) an identifier (Space_ID) of the space to which the encoded data of the data unit belongs, and an identifier (DataUnit_ID) of the data unit.

[0170] The data units are further given a header including an identifier of the data unit or length information of the data unit, and are then unitized to generate coding scheme units.

[0171] Next, the syntax of the coding method unit will be described. Fig. 18 is a diagram showing an example of the syntax of the coding method unit. Fig. 19 is a diagram showing an example of the syntax of the coding point group. Fig. 20 is a diagram showing an example of the syntax of the coding mesh. Fig. 21 is a diagram showing an example of the syntax of the coding 3D model.

[0172] The unit_type indicates the type of data unit stored in the encoding method unit, thereby specifying the type of data unit stored in the encoding method unit.

[0173] The length indicates the length of the data unit.

[0174] data() indicates the body of the data unit.

[0175] In Fig. 19, when unit_type indicates 0, it indicates that the data unit is position information (geometry) of the coded point group. When unit_type indicates 1, it indicates that the data unit is attribute information of the coded point group. When unit_type indicates 2, it indicates that the data unit is metadata of the coded point group.

[0176] In Fig. 20, when unit_type indicates 0, it indicates that the data unit is position information (geometry) of the coded mesh. When unit_type indicates 1, it indicates that the data unit is attribute information of the coded mesh. When unit_type indicates 2, it indicates that the data unit is metadata of the coded mesh.

[0177] In Fig. 21, when unit_type indicates 0, it indicates that the data unit is element 1 of the encoded 3D model. When unit_type indicates 1, it indicates that the data unit is element 2 of the encoded 3D model. When unit_type indicates 2, it indicates that the data unit is metadata of the encoded 3D model.

[0178] 19 to 21 are merely examples and are not limited to the above configurations. These syntaxes may use a partial configuration of the syntax, or may use a type (classification) not described above, or the order of the syntax components may be changed. For example, the syntax of the encoding method unit may have a configuration of the encoding method unit common to multiple encoding methods as shown in FIG. 18, and may include the unit_type, length, and data() shown in FIGS. 19 to 21.

[0179] The coding method unit may further include a header indicating the type of the coding method unit. The coding unit type may include, for example, point_cloud_codec_unit indicating point cloud data, mesh_codec_unit indicating mesh data, and model_codec_unit indicating 3D model data. This allows multiple coding methods to be handled in an integrated manner.

[0180] FIG. 22 is a diagram showing an example of the syntax of three-dimensional data information.

[0181] In the syntax, when multiple encoding methods are stored in one format, the number of 3D data included in the format (number_of_3Dformat) and the type of 3D data (format_type) are indicated, and data of each format may be stored. This makes it possible to handle multiple encoding methods or 3D data in an integrated manner, and also to identify multiple encoding methods or 3D data.

[0182] 3Ddata_info indicates format structure information for storing multiple three-dimensional data.

[0183] number_of_3Dformat indicates the number of 3D formats used.

[0184] format_type indicates the type of format of the three-dimensional data to be stored. For example, the number of format_type and the format corresponding to that number may be defined as follows: When format_type indicates 0, it indicates that the format of the three-dimensional data to be stored is point cloud data. When format_type indicates 1, it indicates that the format of the three-dimensional data to be stored is mesh data. When format_type indicates 2, it indicates that the format of the three-dimensional data to be stored is G-PCC data (g-pcc). When format_type indicates 3, it indicates that the format of the three-dimensional data to be stored is V-DMC data (v-dmc). When format_type indicates 4, it indicates that the format of the three-dimensional data to be stored is three-dimensional model data (3Dmodel).

[0185] Next, the data structure of encoded data of multiple pieces of three-dimensional data will be described for each type of three-dimensional data. Fig. 23 is a diagram for explaining the data structure of an encoded point group. Fig. 24 is a diagram for explaining the data structure of an encoded mesh. Fig. 25 is a diagram for explaining the data structure of an encoded three-dimensional model.

[0186] The encoding device divides each of the multiple types of three-dimensional data into multiple pieces of three-dimensional data for each of multiple spatial regions, encodes each of the multiple pieces of divided three-dimensional data (i.e., multiple divided three-dimensional data), and generates encoded data.

[0187] A header is attached to each piece of encoded data, and at least one of a data_unit_id and a space_id is stored.

[0188] Here, data_unit_id is an identifier that identifies a data unit in the coded data and is unique within the coded data. Furthermore, space_id indicates identification information for a spatial region. If data_unit_id or space_id is common to multiple pieces of three-dimensional data, the same value is indicated in the multiple pieces of three-dimensional data.

[0189] 23 to 25, the data unit with data_unit_id=0 in the encoded point group, the data unit with data_unit_id=3 in the encoded mesh, and the data unit with data_unit_id=0 in the encoded 3D model are all assigned space_id=1, which means that they are 3D data contained in a common 3D space indicated by Space_ID #1.

[0190] The data and headers may be included in a bitstream structure such as a data unit or encoding method, or may be stored in a predetermined file format such as each box of ISOBMFF.

[0191] Next, three-dimensional space information will be described. Fig. 26 is a diagram showing an example of a plurality of three-dimensional spaces in two dimensions. Fig. 27 is a diagram showing an example of a bounding box. Fig. 28 is a diagram showing an example of the syntax of three-dimensional space information.

[0192] In the syntax of the three-dimensional space information, 3Dspace_info is information indicating the divided three-dimensional space. 3Dspace_info can be used for partial decoding.

[0193] The number_of_space indicates the number of divided three-dimensional spaces.

[0194] The space_id indicates the identifier of the divided three-dimensional space.

[0195] The three-dimensional space information includes bounding box information as information for defining the bounding box shown in FIG.

[0196] The bounding box information includes bounding_box_xyz and bounding_box_whd.

[0197] bounding_box_xyz indicates the coordinates of the reference point of the bounding box, and in the example of Fig. 27, is expressed as x, y, and z coordinate values ​​(x0, y0, z0).

[0198] The bounding_box_whd field indicates the size of the bounding box, which in the example of Fig. 27 is expressed as width w, height h, and depth d (w0, h0, d0).

[0199] The three-dimensional spatial information may also include an identifier of a data unit for each piece of coded data. However, the three-dimensional spatial information does not necessarily have to include the identifier. In other words, the identifier does not necessarily have to be signaled.

[0200] The pointcloud_id indicates the identifier of the data unit of the encoded point cloud of the space corresponding to the space_id.

[0201] mesh_id indicates the identifier of the data unit of the encoding mesh of the space corresponding to space_id.

[0202] model_id indicates the identifier of the data unit of the coded three-dimensional model of the space corresponding to space_id.

[0203] When the data unit does not indicate a space_id but indicates a data_unit_id, an identifier of the data unit for each piece of coded data may be stored in the information indicating each space of the three-dimensional space information, thereby allowing the three-dimensional space information to be associated with the divided three-dimensional coded data.

[0204] Furthermore, when a space_id is indicated in a data unit, the three-dimensional spatial information may be associated with an identifier of the data unit for each encoded data by the space_id. In this case, the identifier of the data unit for each encoded data does not need to be stored.

[0205] The division method, the origin of each divided space, and the bounding box size may be the same for mesh data and point cloud data, thereby sharing the three-dimensional spatial information of the point cloud data and the mesh data. Furthermore, the same three-dimensional spatial information may be used for both point cloud data and mesh data. In this way, the three-dimensional spatial information may be shared or the same three-dimensional spatial information may be used between multiple different types of three-dimensional data. Sharing the three-dimensional spatial information facilitates switching between different types of three-dimensional data (e.g., switching between presentations or transmissions). Furthermore, in a format that handles multiple three-dimensional data in an integrated manner, three-dimensional spatial information does not need to be provided for each piece of three-dimensional data; instead, one piece of three-dimensional spatial information can be used for each piece of three-dimensional data, thereby reducing the amount of three-dimensional spatial information.

[0206] In addition to the point cloud data and mesh data, the three-dimensional spatial information of the three-dimensional model may be synchronized with other types of three-dimensional data, or may be made common with the three-dimensional spatial information of other types of three-dimensional data.

[0207] Next, the relationship between the data structure of three-dimensional data and partial decoding will be described. Fig. 29 is a flowchart showing an example of partial decoding. Fig. 30 is a diagram showing an example of a three-dimensional spatial region that is the target of partial decoding. Fig. 31 is a diagram showing an example of the data structure of an encoded point group that is partially decoded. Fig. 32 is a diagram showing an example of the data structure of an encoded mesh that is partially decoded. Fig. 33 is a diagram showing an example of the data structure of an encoded three-dimensional model that is partially decoded.

[0208] In partial decoding, the decoding device first determines a three-dimensional spatial region to be partially decoded (S1001).

[0209] Next, the decoding device uses the three-dimensional spatial information (3Dspace_info) to identify an area that overlaps with the target three-dimensional spatial area from the bounding box information of multiple three-dimensional spatial areas, and obtains the space_id corresponding to the identified area (S1002).

[0210] Next, the decoding device obtains and decodes the data unit having the obtained space_id from the encoded data (S1003). As a result, the decoding device performs partial decoding, which decodes a part of the three-dimensional data. In partial decoding, the decoding device decodes only a part of the three-dimensional data without decoding the entire three-dimensional data.

[0211] For example, as shown in FIG. 30, when the three-dimensional space area to be partially decoded is the area indicated by the thick line, the space_id of the three-dimensional space to be acquired is determined to be #2 from the three-dimensional space information.

[0212] Then, as shown in FIGS. 31 to 33, the data unit associated with Space_id=#2 from the coded data of the multiple types of three-dimensional data is obtained and decoded.

[0213] The decoding device may also obtain a data unit ID instead of space_id from the three-dimensional spatial information, and obtain and partially decode a data unit having the obtained data unit ID.

[0214] Next, a case where spatial synchronization is not achieved will be described. Fig. 34 is a diagram showing an example of coordinate systems of different types of three-dimensional data where spatial synchronization is not achieved. Fig. 35 is a diagram showing an example of the syntax of three-dimensional data information. Fig. 36 is a diagram showing an example of the syntax of three-dimensional space information.

[0215] As shown in Fig. 34, when the origin coordinates of data in the same space are different due to differences in coordinate systems, the encoding device may align these origin coordinates by calculating and correcting the relative coordinate values ​​(x1-x2, y1-y2, z1-z2). The encoding device may notify (transmit) the calculated relative coordinate values ​​to the decoding device (terminal) as synchronization information.

[0216] The synchronization information may be indicated by the relative position of the point cloud data (origin) relative to the position of the mesh data (origin), or may be indicated by the relative position of the mesh data relative to the position of the point cloud data.

[0217] The same applies to three-dimensional model data. In this way, the synchronization information may be indicated by the relative position of the positions (origins) of different types of three-dimensional data. Note that when there are three or more types of three-dimensional data, the relative position is calculated based on any one type of three-dimensional data.

[0218] The three-dimensional data information may include space_sync_information indicating synchronization information, as shown in Fig. 35. The space_sync_information indicates synchronization information of the three-dimensional space, and indicates, for example, the amount of deviation in the three-dimensional space (the difference between the reference coordinates and the current coordinates, i.e., a relative value).

[0219] In the three-dimensional data information, synchronization information (space_sync_information) may be stored for each of a plurality of pieces of format information. In this case, the data format that serves as the synchronization reference may be placed at the beginning of a loop, and synchronization information (relative position information) from the beginning format may be stored in the second loop and thereafter.

[0220] Furthermore, the three-dimensional space information may include space_sync_information indicating synchronization information, as shown in Fig. 36. When the spatial position is shifted for each three-dimensional space, the synchronization information may be stored in a loop for each three-dimensional space.

[0221] FIG. 37 is a diagram illustrating an example of a functional configuration of a terminal.

[0222] The terminal 1120 includes a decoding unit 1121 and a synchronous presentation unit 1122 .

[0223] The decoding unit 1121 decodes the synchronization information based on the three-dimensional data information or the three-dimensional space information. The synchronization presentation unit 1122 aligns the three-dimensional data based on the synchronization information and presents it.

[0224] Next, spatial synchronization will be described with reference to a flowchart shown in FIG.

[0225] First, the system including the encoding device (server) and the decoding device (terminal) determines whether spatial synchronization is required between the point cloud data and the mesh data (S1011).

[0226] If the system determines that spatial synchronization is necessary (Yes in S1011), it executes step S1012, and if the system determines that spatial synchronization is not necessary (No in S1011), it executes step S1013.

[0227] In step S1012, the system determines whether spatial synchronization is achieved between the point cloud data and the mesh data (S1012).

[0228] If the system determines that spatial synchronization has been achieved (Yes in S1012), it executes step S1014, and if the system determines that spatial synchronization has not been achieved (No in S1012), it executes step S1015.

[0229] In step S1013, the system presents the point cloud data and mesh data without spatial synchronization (S1013).

[0230] In step S1014, the system presents the point cloud data and mesh data in spatial synchronization based on the synchronization information (S1014).

[0231] In step S1015, the system presents the point cloud data and mesh data as they are (S1015).

[0232] The processes of steps S1011 to S1012 may be performed by the encoding device or the decoding device, and the processes of steps S1013 to S1015 may be performed by the decoding device.

[0233] Although the above flowchart describes the spatial synchronization process between point cloud data and mesh data, it may also be performed for 3D model data. The spatial synchronization process may be performed between different types of 3D data, and the type of 3D data that is the subject of the spatial synchronization process is not limited.

[0234] Whether spatial synchronization is necessary may be switched depending on the application or use. For example, when three-dimensional data is used for measurement purposes, accurate positioning is required, so it may be determined that spatial synchronization is necessary.

[0235] Also, when displaying and presenting to the user, accurate alignment may not be necessary, and it may be determined that spatial synchronization is not necessary.

[0236] Note that a level of alignment may be defined, and in this case, the system (encoding device or decoding device) may change the accuracy of synchronization based on the level of alignment.

[0237] (Others) In the above embodiment, an example of synchronizing multiple pieces of three-dimensional data has been described using spatial synchronization as an example, but other than that, time synchronization may be performed to synchronize the presentation time, decoding time, or acquisition time. At least one of spatial synchronization and time synchronization may be performed.

[0238] Furthermore, in order to seamlessly switch the color representation presented when switching between three-dimensional data, parameters such as the color matrix, the number of color bits, and HDR may be set to be the same. In this way, attribute information may be synchronized between multiple three-dimensional data.

[0239] In the above embodiment, switching between three-dimensional data has been described using examples of different types of three-dimensional data, such as point cloud data and mesh data. However, the multiple three-dimensional data to be switched are not limited to different types of three-dimensional data. The multiple three-dimensional data may be, for example, three-dimensional data with different resolutions, multiple point cloud data with different numbers of points, or multiple mesh data with different numbers of points or faces. Furthermore, the multiple three-dimensional data to be switched may be three or more three-dimensional data.

[0240] The multiple pieces of 3D data to be switched may be multiple pieces of point cloud data acquired at different times. For example, the multiple pieces of 3D data to be switched may include point cloud data before construction, point cloud data after construction, point cloud data 10 years from now, and modeled mesh data at a construction site.

[0241] In addition to three-dimensional data, three-dimensional model data such as NeRF may also be used. The three-dimensional model data is a model for presenting three-dimensional data and may or may not be encoded. Multiple three-dimensional model data for the same space may be switched, or multiple three-dimensional model data may be switched. In this case, the spatial synchronization method described above can be used. Furthermore, by adding a three-dimensional model data identifier to a data format that handles point cloud data and mesh data in an integrated manner, it becomes possible to handle the three-dimensional data and the three-dimensional model data in an integrated manner. Furthermore, by making the spatial information (origin, bounding box, division method, etc.) of the three-dimensional data and the three-dimensional model data the same, it is possible to share area information of the three-dimensional space, facilitating partial decoding.

[0242] In addition, the three-dimensional data to be processed (three-dimensional data to be presented or transmitted) may be switched in the following order: mesh data lower than the first resolution, mesh data higher than the first resolution, point cloud data lower than the second resolution, and point cloud data higher than the second resolution.

[0243] Regarding the timing of data transmission or encoding, the server may store data in a pre-encoded state of multiple 3D models and extract the requested 3D data based on a request from the terminal. The server may also encode the requested 3D data at the timing of the request from the terminal. The terminal may request the 3D data to be processed in advance and download the requested 3D data from the server. This reduces the time it takes for the terminal to present the data.

[0244] Furthermore, for each of the plurality of divided 3D data, it may be possible to select whether to present point cloud data, mesh data, or 3D model data, or to present two or more of these 3D data in a superimposed manner. For example, the terminal may present divided data corresponding to the avatar's torso as a mesh and divided data corresponding to the face as a point cloud, thereby enabling more accurate presentation.

[0245] In the above-described embodiment, point cloud data, mesh data, and three-dimensional model data are exemplified as three-dimensional data representing a three-dimensional object, but the present invention is not limited to these. For example, a three-dimensional object may be represented by a plurality of sets, each of which includes line-of-sight information indicating a line of sight and a two-dimensional image of the three-dimensional object as viewed from the line of sight. In other words, data including the plurality of sets may be treated as a type of three-dimensional data. Furthermore, the three-dimensional data may be data in another format, such as Gaussian splatting data.

[0246] Fig. 39 is a diagram showing an example of the configuration of a decoding device, and Fig. 40 is a flowchart showing an example of a decoding method by the decoding device.

[0247] The decoding device 1130 includes a circuit 1131 and a memory 1132 connected to the circuit 1131 .

[0248] The circuit 1131 performs the following operations.

[0249] The circuit 1131 acquires encoded data including first data representing a three-dimensional object, encoding format information (format) indicating one of encoding formats including second data representing the three-dimensional object, and identification information indicating a three-dimensional space including the three-dimensional object (S1021). Next, the circuit 1131 decodes the first data and the second data corresponding to the three-dimensional space based on the encoded data (S1022). Next, the circuit 1131 renders the first data to generate first presentation data for presentation (S1023). Next, the circuit 1131 renders the second data to generate second presentation data for presentation (S1024). Next, the circuit 1131 switches from the generated second presentation data to the first presentation data and presents it (S1025). Note that the first presentation data and the second presentation data are, for example, two-dimensional data or three-dimensional data generated by the rendering reconstruction unit 1034.

[0250] According to this, the first presentation data and the second presentation data are generated based on the first data and the second data corresponding to the three-dimensional space, and the first presentation data is switched to be presented, so that the two pieces of data representing the three-dimensional object can be presented without causing a spatial misalignment when they are switched, thereby making it possible to appropriately present the first presentation data and the second presentation data.

[0251] For example, the first data is point cloud data representing the three-dimensional object.

[0252] Therefore, the second presentation data is switched to the first presentation data based on the point cloud data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0253] For example, the second data is mesh data representing the three-dimensional object.

[0254] Therefore, the second presentation data based on mesh data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0255] For example, the second data is three-dimensional model data representing the three-dimensional object, and the three-dimensional model data represents a machine learning model obtained by machine learning a plurality of sets of gazes and two-dimensional images.

[0256] Therefore, the second presentation data based on the three-dimensional model data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0257] For example, the second data is a two-dimensional image of the three-dimensional object when viewed from a predetermined line of sight.

[0258] Therefore, the second presentation data based on a two-dimensional image is switched to the first presentation data, so that the two pieces of data representing a three-dimensional object can be switched and presented without causing any spatial misalignment.

[0259] For example, the circuit may further receive a request to switch presentation data from a user, and in the presentation, the circuit may switch from the second presentation data to the first presentation data in response to the request to switch presentation data.

[0260] Therefore, the switching can be performed at a timing designated by the user.

[0261] For example, the circuit may further receive an operation from a user to change the presentation mode, and the circuit may change the presentation mode in response to the operation, and switch from the second presentation data to the first presentation data in response to the change.

[0262] Therefore, switching can be performed at a timing according to the user's operation.

[0263] For example, the circuit may acquire the encoded data from an encoding device via a communication network during the acquiring step, and may switch from the second presentation data to the first presentation data during the presenting step in accordance with a bandwidth of the communication network.

[0264] Therefore, switching can be performed according to the bandwidth of the communication network; for example, when the bandwidth of the communication network changes from less than a specified bandwidth to greater than or equal to the specified bandwidth, the second presentation data can be switched to the first presentation data and presented.

[0265] For example, the circuit switches from the second presentation data to the first presentation data in accordance with the available capabilities of the circuit during the presentation.

[0266] Therefore, switching can be performed according to the capacity of the available circuit; for example, when the capacity of the available circuit changes from less than a predetermined capacity to greater than or equal to the predetermined capacity, the second presentation data can be switched to the first presentation data and presented.

[0267] For example, the encoded data includes synchronization information for synchronizing a coordinate system of the first data and a coordinate system of the second data, and the circuit, during presentation, presents the first presentation data and the second presentation data based on the synchronization information.

[0268] Therefore, the first presentation data and the second presentation data can be switched to the first presentation data after the coordinate systems of the first presentation data and the second presentation data are aligned, so that the two pieces of data representing the three-dimensional object can be switched and presented with less spatial misalignment.

[0269] For example, the circuit further determines whether to synchronize a coordinate system of the first data with a coordinate system of the second data. If the circuit determines that the coordinate system of the first data is synchronized with the coordinate system of the second data, the circuit presents the first presentation data and the second presentation data based on the synchronization information during the presentation.

[0270] This allows synchronization processing to be performed when necessary and can be skipped when not necessary, potentially reducing the processing load.

[0271] For example, the first data and the second data each have a common structure.

[0272] This reduces the amount of coded data, thereby reducing communication capacity.

[0273] For example, the encoded data includes spatial information for identifying the three-dimensional space in which the three-dimensional object is located. The circuit further acquires a target area indicating a portion of the three-dimensional space. The circuit identifies first duplicate data that is a portion of the first data and overlaps with the target area based on the spatial information. In the decoding process, the circuit decodes the identified first duplicate data.

[0274] Therefore, for example, by acquiring only the first duplicate data, the volume of data to be acquired can be reduced, thereby reducing communication capacity. Also, for example, it is possible to decrypt only the first duplicate data, thereby reducing processing load.

[0275] Alternatively, the circuit 1131 may operate according to the decoding method shown in the flowchart of Fig. 41. Fig. 41 is a flowchart showing another example of the decoding method performed by the decoding device.

[0276] The circuit 1131 decodes encoding information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data (S1031). The circuit 1131 decodes second data in the second encoding method indicated by the encoding information (S1032). The second data is used to generate second presentation data for presentation.

[0277] This allows the second data of the second encoding method indicated by the encoding method information obtained by decoding to be decoded, thereby obtaining second data for generating appropriate second presentation data for presentation.

[0278] Fig. 42 is a diagram showing an example of the configuration of an encoding device, and Fig. 43 is a flowchart showing an example of an encoding method performed by the encoding device.

[0279] The encoding device 1140 includes a circuit 1141 and a memory 1142 connected to the circuit 1141 .

[0280] The circuit 1141 performs the following operations.

[0281] The circuit 1141 generates encoding information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data (S1041). The circuit 1141 generates second data in the second encoding method indicated by the encoding information (S1042). The circuit 1141 generates a bitstream including the encoding information and the second data (S1043). The second data is used to generate second presentation data for presentation.

[0282] This allows a bitstream including encoding method information and second data to be generated, and a decoding device that acquires the bitstream can obtain the second data for generating second presentation data for appropriate presentation.

[0283] (Embodiment 2) In this embodiment, a method for switching three-dimensional data, three-dimensional data formats, and encoding methods depending on the use case of an application that handles three-dimensional data including, for example, point cloud data, mesh data, and three-dimensional model data will be described.

[0284] When the point cloud data, mesh data, and three-dimensional model data are simultaneously encoded using the same encoding method, the decoding device may decode the desired type of three-dimensional data from the encoded data.

[0285] Furthermore, if three-dimensional data of different resolutions is encoded for each type of point cloud data, mesh data, and three-dimensional model data, the three-dimensional data of the desired resolution may be extracted from the encoded data and decoded.

[0286] FIG. 44 is a diagram illustrating partial decoding for decoding part of three-dimensional data in the second embodiment.

[0287] The control unit 1202 designates a portion of the three-dimensional data to be decoded.

[0288] The partial decoding unit 1201 decodes a portion of the three-dimensional data specified by the control unit 1202. For example, the partial decoding unit 1201 acquires three-dimensional data from an encoding device and decodes a portion of the acquired three-dimensional data.

[0289] FIG. 45 is a diagram illustrating an example of extracting and decoding a part of three-dimensional data in the second embodiment.

[0290] The control unit 1212 specifies the three-dimensional data to be extracted from the three-dimensional data.

[0291] The data extraction unit 1211 extracts a portion of the three-dimensional data specified by the control unit 1212. For example, the data extraction unit 1211 extracts a portion of the three-dimensional data from the three-dimensional data, and transmits the extracted portion of the three-dimensional data to, for example, a decoding device.

[0292] This allows the decoding device to decode the specified portion of three-dimensional data.

[0293] The coded data may have a hierarchical structure, and hierarchical data indicating the hierarchical structure may be output.

[0294] Although the following mainly describes a switching method using point cloud data or mesh data, it is equally applicable to a three-dimensional model data model. In other words, the switching method may be performed using two or more of three-dimensional data selected from point cloud data, mesh data, and three-dimensional model data.

[0295] Next, the configurations of the server 1220 and the terminal 1230 will be described. Fig. 46 is a block diagram showing an example of the functional configuration of the server and the terminal in embodiment 2. Note that the server 1220 is, for example, an encoding system that encodes three-dimensional data, and the terminal 1230 is a decoding system that decodes and generates three-dimensional data.

[0296] The server 1220 includes a point cloud generation unit 1221 , a mesh generation unit 1222 , a point cloud encoding unit 1223 , and a multiplexing unit 1224 .

[0297] The point cloud generator 1221 generates point cloud data based on at least one of two-dimensional data and three-dimensional data.

[0298] The mesh generation unit 1222 generates mesh data based on the point cloud data generated by the point cloud generation unit 1221 .

[0299] The point cloud encoding unit 1223 encodes three-dimensional data including the point cloud data generated by the point cloud generating unit 1221 and the mesh data generated by the mesh generating unit 1222 .

[0300] The multiplexing unit 1224 multiplexes the encoded three-dimensional data using a predetermined format or a predetermined multiplexing method. Note that multiplexing by the multiplexing unit 1224 may not be performed. In this case, the server 1220 may not be equipped with the multiplexing unit 1224. The multiplexed encoded data is transmitted (sent) to the terminal 1230.

[0301] The terminal 1230 includes a control unit (application execution unit) 1231 , a decoding unit 1232 , and a presentation unit 1233 .

[0302] The control unit 1231 specifies a portion of the three-dimensional data to be decoded by the decoding unit 1232. The specified portion of the three-dimensional data is, for example, the type (format) of the three-dimensional data. The control unit 1231 specifies whether to decode point cloud data or mesh data. The control unit 1231 may also specify the resolution of the three-dimensional data.

[0303] The decoding unit 1232 decodes a portion of the three-dimensional data specified by the control unit 1231 from the encoded data received from the server 1220. The decoding unit 1232 decodes the point cloud data if the control unit 1231 specifies that the point cloud data be decoded, and decodes the mesh data if the control unit 1231 specifies that the mesh data be decoded. Alternatively, the decoding unit 1232 decodes three-dimensional data at a specified resolution.

[0304] The presentation unit 1233 presents a portion of the three-dimensional data decoded by the decoding unit 1232. The presentation unit 1233 presents the point cloud data if the point cloud data has been decoded by the decoding unit 1232, and presents the mesh data if the mesh data has been decoded by the decoding unit 1232. Alternatively, the decoding unit 1232 presents the three-dimensional data at a specified resolution.

[0305] Here, the point cloud data, mesh data, and three-dimensional model data have different resolutions, different display appearances, and different granularity. Three-dimensional data that includes at least two of the point cloud data, mesh data, and three-dimensional model data can also be said to be hierarchical data.

[0306] FIG. 47 is a flowchart showing an example of processing by a terminal in the second embodiment.

[0307] The control unit 1231 determines the format of the three-dimensional data to be used based on a predetermined process (S1201).

[0308] The control unit 1231 instructs (specifies) the format of the three-dimensional data to be output (presented) to the decoding unit 1232 (S1202).

[0309] The decoding unit 1232 decodes the three-dimensional data in the specified format using a predetermined method and outputs it (S1203).

[0310] The presentation unit 1233 uses the three-dimensional data in the specified format in the application using a predetermined method (S1204). For example, the presentation unit presents the three-dimensional data.

[0311] Next, the Trisorp method, which is an example of a method for encoding position information by an encoding device, will be described.

[0312] The Trithorpe method is a lossy compression method used in the G-PCC encoding method to encode the positional information of point cloud data. The Trithorpe method replaces the original point cloud to be processed with a set of triangles, and then approximates the point cloud on that plane. Specifically, the original point cloud is replaced with vertex information within a node, and the vertices are connected to generate a set of triangles. The vertex information used to generate the triangles is then stored in a bitstream and sent to a decoding device.

[0313] First, the encoding process using the Trithorpe method will be described. Fig. 48 is a diagram showing an example of an original point cloud in Embodiment 2. As shown in Fig. 48, a point cloud 1242 of an object is contained in an object space 1241 and includes a plurality of points 1243.

[0314] First, the encoding device divides the source point cloud into an octree to a predetermined depth. In the octree division, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. Furthermore, the node including the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated up to a predetermined hierarchy.

[0315] In normal octree coding, division is repeated until the number of points included in a node is reduced to one or a threshold value. On the other hand, in the Trithorpe method, octree division is performed up to an intermediate layer, but not to layers below that layer. Such an octree up to an intermediate layer is called a pruned octree.

[0316] Fig. 49 is a diagram showing an example of a pruned octree in Embodiment 2. As shown in Fig. 49, a point cloud 1242 is divided into a plurality of leaf nodes 1244 (nodes in the lowest layer) of the pruned octree.

[0317] Next, the encoding device performs the following process on each leaf node 1244 of the pruned octree. Note that hereinafter, a leaf node may also be simply referred to as a node. The encoding device generates a vertex on the edge as a representative point of a group of points close to the edge of the node. This vertex is called an edge vertex. For example, an edge vertex is generated for each of multiple edges (e.g., four parallel sides).

[0318] Figure 50 shows an example of a two-dimensional display of a leaf node 1244 in embodiment 2, for example, a diagram showing an xy plane as viewed from the z direction shown in Figure 50. As shown in Figure 50, an edge vertex 1252 is generated on the edge based on a point close to the edge among multiple points 1251 in the leaf node 1244.

[0319] In Fig. 50, the dotted line on the periphery of the leaf node 1244 is the edge. In this example, the edge vertex 1252 is generated at the weighted average position of the positions of points whose distance from the edge is within 1 (points included in the range 1253 in Fig. 50). The unit of distance is, for example, the resolution of the point cloud, but is not limited to this. In this example, this distance (threshold) is 1, but it may be a value other than 1 and may be variable.

[0320] Next, the encoding device generates a vertex inside the node based on a group of points existing in the normal direction of the plane containing the edge vertices. This vertex is called a centroid vertex.

[0321] 51 and 52 are diagrams illustrating a method for generating a centroid vertex in the second embodiment. First, the encoding device selects, for example, four points as representative points from a group of edge vertices. In the example shown in FIG. 51, edge vertices v1 to v4 are selected. Next, the encoding device calculates an approximate plane 1261 that passes through the four points. Next, the encoding device calculates a normal n to the approximate plane 1261 and an average coordinate M of the four points. Next, the encoding device generates a centroid vertex C at the weighted average coordinate of one or more points (for example, points included in the range 1262 shown in FIG. 52) that are close to a half-line extending from the average coordinate M in the direction of the normal n.

[0322] Next, the encoding device entropy-encodes the vertex information, which is information on the edge vertices and centroid vertices, and stores the encoded vertex information in a Geometry Data Unit (GDU) included in the bitstream. Note that the GDU includes information indicating the pruned octree in addition to the vertex information.

[0323] 53 is a diagram showing an example of vertex information in Embodiment 2. Through the above processing, the point group 1242 is converted into vertex information 1263 as shown in FIG.

[0324] Next, the decoding process of the bitstream generated above will be described. First, the decoding device decodes the GDU from the bitstream to obtain vertex information. Next, the decoding device connects the vertices to generate a TriSoup-Surface, which is a group of triangles.

[0325] Figure 54 is a diagram showing an example of a Trithorpe surface in embodiment 2. In the example shown in Figure 54, four edge vertices v1 to v4 and a centroid vertex C are generated based on the vertex information. In addition, a triangle 1271 (Trithorpe surface) is generated whose vertices are the centroid vertex C and two edge vertices. For example, pairs of two edge vertices on two adjacent edges are each selected, and a triangle 1271 is generated whose vertices are the selected pair and the centroid vertex.

[0326] Fig. 55 is a diagram for explaining the point cloud restoration process in embodiment 2. By performing the above process for each leaf node, a three-dimensional model is generated in which the target object is represented by a plurality of triangles 1271, as shown in Fig. 55.

[0327] Next, the decoding device generates points 1272 (face vertices) at regular intervals on the surface of the triangle 1271 to restore the position information of the point group 1273 .

[0328] In the Trisoap method, in addition to outputting point cloud data, a decoding unit capable of decoding and outputting point cloud data or mesh data of a desired resolution according to a use case that enables output of point cloud data or mesh data of different resolutions will be described. The following describes an example of decoding data encoded by the Trisoap method in G-PCC, which is an example of a compression method for point cloud data, but the present invention can also be applied to decoding compressed mesh data such as V-DMC and Draco.

[0329] FIG. 56 is a diagram showing an example of the configuration of a decoding unit in the second embodiment.

[0330] The decoding unit 1232 includes a mesh vertex generation unit 1281, a high-resolution additional information decoding unit 1282, an additional vertex generation / correction unit 1283, an additional vertex generation / correction unit 1284, a first connection information generation unit 1285, an additional information decoding unit for point cloud generation 1286, a point generation unit 1287, a second connection information generation unit 1288, a third connection information generation unit 1289, and a point generation unit 1290.

[0331] The mesh vertex generation unit 1281 decodes the bit stream and obtains information about the vertices (edge ​​vertices) that make up the surface.

[0332] The high-resolution additional information decoding unit 1282 decodes, from the bitstream, additional information for restoring additional vertices for increasing the resolution of a surface. For example, in a triangle soup, the vertices that make up a surface are the edge vertices of each node, and the additional vertices are the center of gravity (centroid vertices) of each node or points located on the surface of the node (face vertices).

[0333] The additional vertex generation / correction unit 1283 restores (generates) additional vertices that make up a surface. The additional vertex generation / correction unit 1283 restores (generates) points within a node, such as the center of gravity (centroid vertex) for each node, based on, for example, edge vertices and extension information for restoring the center of gravity. In this way, the additional vertex generation / correction unit 1283 increases the resolution of the surface.

[0334] The additional vertex generating / correcting unit 1284 further restores (generates) additional vertices that make up the face based on the edge vertices, the center of gravity, and the extension information for restoring the face vertices. The additional vertex generating / correcting unit 1284 restores (generates) points (face vertices) located on the face of the node, for example.

[0335] After restoring the points, the additional vertex generating / correcting units 1283 and 1284 may correct the restored points using a predetermined method.

[0336] The first connection information generation unit 1285 connects pairs of restoration points among the plurality of restoration points using a predetermined method to form a surface, thereby generating first connection information that indicates the connection relationship between the restoration points that form the surface.

[0337] The point cloud generation additional information decoding unit 1286 decodes the extended information for restoring the point cloud from the bit stream.

[0338] The point generation unit 1287 generates and outputs points using a predetermined method based on the surface constructed by the first connection information generation unit 1285 and the extended information decoded by the point cloud generation additional information decoding unit 1286. This outputs a point cloud.

[0339] Here, the edge vertices generated by the mesh vertex generation unit 1281 are points that make up a low-resolution surface. The points within the nodes generated by the additional vertex generation / correction unit 1283 are points that make up a medium-resolution surface. The face vertices generated by the additional vertex generation / correction unit 1284 are points that make up a high-resolution surface.

[0340] The decoding unit 1232 may output high-resolution surface information (points that constitute the surface and first connection information) generated by processing by the additional vertex generation / correction unit 1284 and processing by the first connection information generation unit 1285.

[0341] The second connection information generation unit 1288 generates second connection information indicating the connection relationship of the restoration points by connecting the restoration points using a predetermined method, based on the points that constitute the medium-resolution surface output by the additional vertex generation / correction unit 1283. The decoding unit 1232 may output the medium-resolution surface information (the points that constitute the surface and the second connection information) generated by the processing by the additional vertex generation / correction unit 1283 and the processing by the second connection information generation unit 1288.

[0342] The third connection information generation unit 1289 generates third connection information indicating the connection relationships of the restoration points by connecting the restoration points using a predetermined method, based on the points that constitute the low-resolution surface output by the mesh vertex generation unit 1281. The decoding unit 1232 may output the low-resolution surface information (the points that constitute the surface and the third connection information) generated by the processing by the mesh vertex generation unit 1281 and the processing by the third connection information generation unit 1289.

[0343] The fourth connection information generation unit 1291 performs processing to further remove some points from the points that constitute the low-resolution surface output by the mesh vertex generation unit 1281, thereby connecting the restoration points in a predetermined manner based on the points that constitute a surface with an even lower resolution than the low-resolution surface, and generates fourth connection information that indicates the connection relationships of the restoration points. The decoding unit 1232 may output the even lower-resolution surface information (parent node mesh) generated by the processing by the mesh vertex generation unit 1281 and the processing by the fourth connection information generation unit 1291.

[0344] The point generation unit 1290 generates and outputs points using a predetermined method based on the low-resolution surfaces output by the mesh vertex generation unit 1281. This outputs a point cloud. The point cloud output by the point generation unit 1290 is a low-resolution point cloud with a lower resolution than the point cloud output by the point generation unit 1287.

[0345] Note that the decoding unit 1232 may have a point generation unit that restores points for surfaces with different resolutions, in addition to the point generation unit 1287 and the point generation unit 1290. In this way, the decoding unit 1232 may output point groups with different resolutions.

[0346] In this way, the decoding unit 1232 may increase the number of points that make up the surface to be restored during decoding and generate surface information that makes up a high-resolution surface, or may generate low-resolution surface information that connects the surfaces using the few points obtained through decoding.

[0347] The decoding unit 1232 may output point cloud data at a plurality of different resolutions or mesh data at a plurality of different resolutions. Then, the point cloud data or mesh data at a desired resolution may be selected and used in an application. Alternatively, the control unit 1231 may specify a resolution or a type of three-dimensional data (point cloud data, mesh data, or three-dimensional model data), and the decoding unit 1232 may decode three-dimensional data of the specified type. In this case, processing for decoding or reproducing data other than the specified data may be skipped. This reduces the amount of processing.

[0348] Note that multiple pieces of three-dimensional data may be superimposed and used by an application, or the decoding unit 1232 may output data in which multiple pieces of three-dimensional data are superimposed. Here, the multiple pieces of three-dimensional data may be three-dimensional data of different types or three-dimensional data with different resolutions.

[0349] FIG. 57 is a diagram showing an example of the syntax of metadata (SEI) in an outputtable format in the second embodiment.

[0350] The encoding unit transmits metadata (SEI) indicating the format included in the encoded data, i.e., the format that the decoding unit can output, in a bitstream. The decoding unit can learn the format that the decoding unit can output by analyzing the metadata. The decoding unit 1232 may also output metadata indicating the format that can be output to the control unit (application execution unit) 1231. For example, in the case of encoded data for each partition space, the metadata may indicate the format that can be output for each partition space.

[0351] num_output_format indicates the number of formats that can be output. format_type indicates the type of format that can be output.

[0352] Note that the metadata may not indicate the default output, but may indicate only the formats that can be output as options.

[0353] The control unit (application execution unit) 1231 refers to the metadata, checks the formats that can be output, determines the format to be output from the formats that can be output, and instructs the decoding device on the determined format.

[0354] Note that, when the decoding device is capable of superimposing and outputting multiple pieces of three-dimensional data, a flag indicating whether the decoding device is capable of superimposing and outputting multiple pieces of three-dimensional data, or the format type of the multiple pieces of three-dimensional data to be superimposed and output may be signaled. That is, the metadata may indicate the flag or the format type. Furthermore, the metadata may indicate a format type indicating a combination of formats that can be output.

[0355] Furthermore, format_type may be a flag indicating whether a predetermined format can be output for each bit.

[0356] Fig. 58 is a diagram showing an example of format_type in Embodiment 2. Fig. 59 is a diagram showing another example of format_type in Embodiment 2.

[0357] Note that format_type is an example and is not limited to this. Only some of the examples in Figures 58 and 59 may be used, types not shown may be used, or the order may be changed.

[0358] Next, a detailed description will be given of the process of step S1203 described with reference to Fig. 47. Fig. 60 is a flowchart showing an example of the details of the process of decoding three-dimensional data in a format specified by a predetermined method in the second embodiment.

[0359] The decoding unit 1232 determines the output format instructed by the control unit 1231 (S1211). The decoding unit 1232 determines whether the instructed output format is low-resolution mesh data, high-resolution mesh data, or point cloud data.

[0360] If the output format is low-resolution mesh data, the decoding unit 1232 decodes the edge vertices (S1212). For example, the decoding unit 1232 restores (generates) faces (tri-node meshes) based on the point cloud data, and restores (generates) the vertices (edge ​​vertices) that make up the faces.

[0361] The decoding unit 1232 generates low-resolution mesh data by generating connection information based on the edge vertices (S1213). For example, the decoding unit 1232 connects restoration points using a predetermined method based on the points that make up the low-resolution surface, and generates third connection information that indicates the connection relationship between the restoration points, thereby generating low-resolution mesh data (low-resolution surface information).

[0362] The decoding unit 1232 outputs the generated low-resolution mesh data (S1214).

[0363] If the output format is high-resolution mesh data, the decoding unit 1232 decodes the edge vertices (S1215). For example, the decoding unit 1232 restores (generates) faces (tri-node meshes) based on the point cloud data, and restores (generates) the vertices (edge ​​vertices) that make up the faces.

[0364] The decoding unit 1232 decodes the center of gravity (centroid vertices) of each node or the face vertices (face vertices) located on the face of the node, and generates connection information based on the decoded vertices (edge ​​vertices, center of gravity, and face vertices), thereby generating high-resolution mesh data (S1216). For example, the decoding unit 1232 connects restoration points using a predetermined method based on the points that make up the high-resolution face, and generates first connection information that indicates the connection relationship between the restoration points, thereby generating high-resolution mesh data (high-resolution face information).

[0365] The decoding unit 1232 outputs the generated high-resolution mesh data (S1217).

[0366] If the output format is point cloud data, the decoding unit 1232 decodes the edge vertices (S1218). For example, the decoding unit 1232 restores (generates) faces (tri-node meshes) based on the point cloud data, and restores (generates) the vertices (edge ​​vertices) that make up the faces.

[0367] The decoding unit 1232 decodes the center of gravity (centroid vertices) of each node or the face vertices (face vertices) located on the face of the node, and generates connection information based on the decoded vertices (edge ​​vertices, center of gravity, and face vertices), thereby generating high-resolution mesh data (S1219). For example, the decoding unit 1232 connects restoration points using a predetermined method based on the points that make up the high-resolution face, and generates first connection information that indicates the connection relationship between the restoration points, thereby generating high-resolution mesh data (high-resolution face information).

[0368] The decoding unit 1232 generates point cloud data based on the high-resolution mesh data (S1220).

[0369] The decoding unit 1232 outputs the generated point cloud data (S1221).

[0370] Next, a description will be given of another configuration of the server 1300 and the terminal 1310. Fig. 61 is a block diagram showing another example of the functional configuration of the server and the terminal in the second embodiment.

[0371] The server 1300 includes a point cloud generation unit 1301 , a mesh generation unit 1302 , a point cloud encoding unit 1303 , a multiplexing unit 1304 , and a data extraction unit 1305 .

[0372] The point cloud generation unit 1301 generates point cloud data based on at least one of two-dimensional data and three-dimensional data.

[0373] The mesh generation unit 1302 generates mesh data based on the point cloud data generated by the point cloud generation unit 1301 .

[0374] The point cloud encoding unit 1303 encodes three-dimensional data including the point cloud data generated by the point cloud generating unit 1301 and the mesh data generated by the mesh generating unit 1302 .

[0375] The multiplexing unit 1304 multiplexes the encoded three-dimensional data using a predetermined format or a predetermined multiplexing method. Note that multiplexing by the multiplexing unit 1304 may not be performed. In this case, the server 1300 may not include the multiplexing unit 1304.

[0376] The data extraction unit 1305 extracts a portion of the three-dimensional data specified by the terminal 1310 from the encoded data generated by the multiplexing unit 1304, and transmits (sends) the extracted portion of the three-dimensional data or three-dimensional data re-multiplexed using the extracted portion of the three-dimensional data to the terminal 1310.

[0377] The terminal 1310 includes a control unit (application execution unit) 1311 , a decoding unit 1312 , and a presentation unit 1313 .

[0378] The control unit 1311 specifies a portion of the three-dimensional data to be transmitted by the server 1300. The specified portion of the three-dimensional data is, for example, the type (format) of the three-dimensional data. The control unit 1311 specifies whether to extract point cloud data or mesh data. The control unit 1311 may also specify the resolution of the three-dimensional data.

[0379] The decoding unit 1312 decodes a portion of the three-dimensional data based on the encoded data received from the server 1300. The decoding unit 1312 may receive only a portion of the three-dimensional data from the server 1300, or may receive all of the three-dimensional data. When the decoding unit 1312 receives all of the three-dimensional data, it decodes a portion of the three-dimensional data. In other words, the server 1300 may transmit only a portion of the three-dimensional data to the terminal 1310, or may transmit all of the three-dimensional data to the terminal 1310.

[0380] The presentation unit 1313 presents a portion of the three-dimensional data decoded by the decoding unit 1312 .

[0381] Here, the point cloud data, mesh data, and three-dimensional model data have different resolutions, different display appearances, and different granularity. Three-dimensional data that includes at least two of the point cloud data, mesh data, and three-dimensional model data can also be said to be hierarchical data.

[0382] FIG. 62 is a flowchart showing another example of processing by the terminal in the second embodiment.

[0383] The control unit 1311 determines the format of the three-dimensional data to be used based on a predetermined process, and requests the server to switch to the determined format (S1301).

[0384] The server 1300 extracts and transmits the three-dimensional data in the requested format (S1302).

[0385] The decoding unit 1312 decodes the three-dimensional data in the transmitted format and outputs it (S1303).

[0386] The presentation unit 1313 uses the decoded three-dimensional data in an application (S1304). For example, the presentation unit presents the three-dimensional data.

[0387] Next, a data unit structure for storing encoded data for each resolution, which is used to realize partial decoding of 3D data at multiple resolutions and switching of decoded output, will be described. Fig. 63 is a diagram showing an example of a data unit structure in the second embodiment.

[0388] The data unit contains coded data in order of resolution. The low-resolution data includes points constituting the low-resolution mesh data and data or metadata for restoring connectivity information. The high-resolution data includes points constituting the high-resolution mesh data and data or metadata for restoring connectivity information. The point information includes data or metadata for restoring points constituting the point cloud data.

[0389] In the example of Figure 63, "low resolution" indicates coded data for decoding low-resolution three-dimensional data. For example, in the case of Trithorpe, "low resolution" is coded data for restoring edge vertices for each node. "high resolution" indicates extended information (additional information for high resolution) for restoring high-resolution three-dimensional data. For example, in the case of Trithorpe, "high resolution" is coded data for restoring additional vertices. "point information" indicates extended information (additional information for generating point cloud) for restoring point information. For example, in the case of Trithorpe, "point information" is coded data for restoring points.

[0390] For example, when outputting only low-resolution data, the decoding unit 1232 reads and decodes the data unit header and the low-resolution data in the data unit from the encoded data. In this case, the decoding unit 1232 does not need to read the high-resolution data and the point information data.

[0391] This configuration eliminates the need for subsequent data readout processing, such as high resolution and point information, which shortens the time until decoding begins and enables low-latency decoding. Furthermore, since subsequent processing is no longer necessary, processing speed can be improved.

[0392] Similarly, when outputting high-resolution data, the decoding unit 1232 reads the data unit header, the low-resolution data in the data unit, and the high-resolution data in the data unit from the encoded data, and performs decoding using both the low-resolution data and the high-resolution data. In other words, the decoding unit 1232 cannot decode using only the high-resolution data without the low-resolution data. In this case, reading of the point information is unnecessary, and the same effect is achieved, that is, since the subsequent data reading process is unnecessary, the time until decoding starts can be shortened and low-latency decoding can be achieved. Furthermore, since the subsequent process is unnecessary, the processing speed can be improved.

[0393] Next, the coding method unit will be described. Fig. 64 is a diagram showing an example of the syntax of the coding method unit according to the second embodiment. Fig. 65 is a diagram showing an example of type information according to the second embodiment.

[0394] To enable the configuration or processing described in Fig. 63, the header of the encoding method unit stores type information (unit_type) for identifying the type of data unit. The type information indicates, for example, the type that indicates the resolution of three-dimensional data.

[0395] By analyzing the unit_type in the encoding method unit header, the decoding device can identify the resolution of the encoded data and determine the decoding process.

[0396] The type for identifying the resolution may be indicated by an identifier or may be stored in a header of a unit other than the encoding method unit header, for example, in the header or metadata of a data unit.

[0397] Note that the unit_types illustrated in Fig. 65 are merely examples and are not limited to these. Only some of the examples in Fig. 65 may be used, types not shown may be used, or the order may be changed.

[0398] Next, the data unit structure for partial transmission will be described. Figures 66 to 68 are diagrams showing examples of data unit structures for partial transmission in the second embodiment.

[0399] In the data unit structure shown in the examples of Figures 67 and 68, data corresponding to each resolution is stored in a data unit consisting of a header and encoded data in resolution order, making it possible to extract only the desired data unit.

[0400] In addition, since it is possible to extract only the desired data units and construct a file as shown in Fig. 66, there is no need to transmit subsequent data units, which has the effect of reducing the amount of transmission. In addition, in this case, there is no need for decoding, which has the effect of reducing the amount of processing.

[0401] For example, when high-resolution data is to be decoded, a low-resolution data unit and a high-resolution extended information data unit are extracted and transmitted to the decoding device, as shown in Figure 67. When high-resolution point cloud data is to be decoded, a low-resolution data unit, a high-resolution data unit, and a point information data unit are extracted and transmitted to the decoding device.

[0402] Next, the coding method unit for partial transmission will be described. Figure 69 is a diagram showing an example of the syntax of the coding method unit for partial transmission in Embodiment 2. Figure 70 is a diagram showing an example of type information in Embodiment 2 for partial transmission.

[0403] To enable the configurations or processes described in Figures 66 to 68, the header of the encoding method unit stores type information (unit_type) for identifying the type of data unit. The type information indicates, for example, the type that indicates the resolution of three-dimensional data.

[0404] The decoding device determines the decoding method based on the data type. For example, the decoding device determines the decoding method based on a type indicating the resolution of the three-dimensional data, or, if the data is layered, a type indicating that the data is additional information. The type may be indicated by a flag indicating whether the data is layered or not. The decoding device can identify that the data is coded data of the same three-dimensional data based on the type information (unit_type), and can decode the three-dimensional data.

[0405] Note that the unit_types illustrated in Fig. 70 are merely examples and are not limited to these. Only some of the examples in Fig. 70 may be used, types not shown may be used, or the order may be changed.

[0406] Next, the group ID will be described. Fig. 71 is a diagram illustrating the group ID according to the second embodiment.

[0407] The group ID indicates an identifier for identifying that the hierarchical data has been generated from the same 3D point cloud. The group ID may be stored in a coding unit header or a data unit header. This allows the encoding device to determine which data unit to combine with the hierarchical data for decoding when encoding using the hierarchical data.

[0408] The group ID may indicate that the data units belong to the same group, or the ID of the corresponding or referencing low-resolution data unit may be indicated within the high-resolution data unit. Similar effects can be expected by using a method of indicating correspondence.

[0409] For example, when decoding a high-resolution three-dimensional point cloud with groupID=1, the decoding device can analyze the header to extract data units with unit_type=6 (complete encoded low-resolution data geometry) and groupID=1, and data units with unit_type=9 (encoded geometry information for high-resolution extension) and groupID=1, and can decode the extracted data units.

[0410] Fig. 72 is a diagram showing an example of a configuration of a decoding device according to Embodiment 2. Fig. 73 is a flowchart showing an example of a decoding method performed by the decoding device according to Embodiment 2.

[0411] The decoding device 1320 includes a circuit 1321 and a memory 1322 connected to the circuit 1321 .

[0412] The circuit 1321 performs the following operations.

[0413] The circuit 1321 acquires coded data (S1311). The circuit 1321 reconstructs a plurality of first vertices based on the coded data (S1312). The circuit 1321 connects the reconstructed first vertices to generate first surface information (fourth connection information) including a first surface (S1313). The circuit 1321 reconstructs a plurality of second vertices based on the coded data and the first surface (S1314). The circuit 1321 connects a point cloud including the reconstructed first vertices and the reconstructed second vertices to generate second surface information (first connection information) including a second surface (S1315). The circuit 1321 outputs the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information (S1316).

[0414] This allows low-resolution information including multiple first vertices and first surface information, and high-resolution information including multiple second vertices and second surface information to be generated and output in two stages, making it possible, for example, to output low-resolution three-dimensional data based on the low-resolution information, or to output high-resolution three-dimensional data with a higher resolution than the low-resolution three-dimensional data based on the low-resolution information and high-resolution information.

[0415] For example, the circuit 1321 further generates point cloud data including a plurality of points based on the second surface, and outputs the point cloud data.

[0416] For example, the encoded data includes first encoded data including the plurality of first vertices and the first face information, and second encoded data including extension information for decoding the plurality of second vertices and the second face.

[0417] For example, the encoded data includes a header and a data unit, and the data unit includes the first encoded data and the second encoded data in that order.

[0418] For example, the encoded data includes a first data unit including a first header and the first encoded data, and a second data unit including a second header and the second encoded data.

[0419] For example, circuit 1321 further decodes a portion of the first encoded data and the second encoded data, and outputs either (1) the first encoded data or (2) the first encoded data or the second encoded data.

[0420] For example, the circuit 1321 further acquires numerical information indicating the number of formats that can be output, and determines the data to be output based on the numerical information.

[0421] For example, the circuit 1321 further acquires type information indicating the type of format that can be output, and determines the type information and the data to be output.

[0422] For example, the circuit 1321 further acquires (1) the first encoded data, or (2) either the first encoded data or the second encoded data, and then decodes and outputs the acquired (1) the first encoded data or (2) the first encoded data and the second encoded data.

[0423] Fig. 74 is a diagram showing an example of a configuration of a coding device in Embodiment 2. Fig. 75 is a flowchart showing an example of a coding method performed by the coding device in Embodiment 2.

[0424] The encoding device 1330 includes a circuit 1331 and a memory 1332 connected to the circuit 1331 .

[0425] The circuit 1331 performs the following operations.

[0426] The circuit 1331 determines a plurality of first vertices of the three-dimensional object (S1321). The circuit 1331 connects the plurality of first vertices to generate first surface information including a first surface (S1322). The circuit 1331 determines a plurality of second vertices based on the first surface (S1323). The circuit 1331 connects a point cloud including the plurality of first vertices and the plurality of second vertices to generate second surface information including a second surface (S1324). The circuit 1331 generates coded data including the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information (S1325).

[0427] According to this, encoded data obtained by generating low-resolution information including a plurality of first vertices and first surface information and high-resolution information including a plurality of second vertices and second surface information in two stages is output, and therefore a decoding device that acquires the encoded data can, for example, output low-resolution three-dimensional data based on the low-resolution information, or output high-resolution three-dimensional data with a higher resolution than the low-resolution three-dimensional data based on the low-resolution information and high-resolution information.

[0428] For example, the circuit 1331 further generates point cloud data including a plurality of points based on the second surface. The encoded data further includes the point cloud data.

[0429] For example, the circuit 1331 further generates first encoded data including the plurality of first vertices and the first plane information. The circuit 1331 further generates second encoded data including the plurality of second vertices and the second plane information. The encoded data includes the first encoded data and the second encoded data. Therefore, a decoding device that acquires the encoded data can restore multiple pieces of three-dimensional data with different resolutions.

[0430] For example, the encoded data includes a header and a data unit, and the data unit includes the first encoded data and the second encoded data in that order. That is, the data unit includes the first encoded data and the second encoded data in that order.

[0431] For example, the encoded data includes a first data unit including a first header and the first encoded data, and a second data unit including a second header and the second encoded data.

[0432] For example, the coded data includes number information indicating the number of formats that can be output.

[0433] For example, the encoded data includes type information indicating the type of format that can be output.

[0434] For example, the circuit 1331 further outputs (1) the first encoded data, and (2) either the first encoded data or the second encoded data.

[0435] Furthermore, the circuit 1321 of the decoding device 1320 or the circuit 1331 of the encoding device 1330 may operate according to the method shown in the flowchart of Fig. 76. Fig. 76 is a flowchart showing an example of a method performed by the decoding device or the encoding device according to the second embodiment.

[0436] The circuit 1321 or the circuit 1331 determines the type of three-dimensional data to be decoded (S1331). The circuit 1321 or the circuit 1331 extracts three-dimensional data of the determined type from the encoded data (S1332). The encoded data includes (1) a plurality of first vertices of a three-dimensional object, (2) first surface information including a first surface generated by connecting the plurality of first vertices, (3) a plurality of second vertices determined based on the first surface, and (4) a second surface generated by connecting a point cloud including the plurality of first vertices and the plurality of second vertices. The type of the extracted three-dimensional data includes a first format and a second format. If the type of three-dimensional data to be decoded is determined to be the first format, first encoded data including the plurality of first vertices and the first surface information is extracted. If the type of three-dimensional data to be decoded is determined to be the second format, second encoded data including the first encoded data and extension information for decoding the plurality of second vertices and the second surface is extracted.

[0437] Therefore, the encoded data necessary for decoding can be extracted based on the determined format. Therefore, when a format requiring a small amount of data for decoding is selected, the amount of data to be extracted can be reduced, which can reduce, for example, the amount of transmission required for transmitting the encoded data and the processing load of the decoding process by the decoding device that has acquired the encoded data.

[0438] Although the three-dimensional data encoding device and the three-dimensional data decoding device according to the embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments.

[0439] Furthermore, each processing unit included in the encoding device, decoding device, server, terminal, etc. according to the above-described embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0440] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.

[0441] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0442] The present disclosure may also be realized as a decoding method or the like executed by a decoding device or the like.

[0443] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0444] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0445] While the encoding device and decoding device according to one or more aspects have been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0446] The present disclosure is applicable to a decoding device and a decoding method.

[0447] 1001 Three-dimensional data encoding system 1002 Three-dimensional data decoding system 1003 Sensor terminal 1004 External connection unit 1011 Three-dimensional data generation system 1012 Presentation unit 1013 Encoding unit 1014 Multiplexing unit 1015 Input / output unit 1016 Control unit 1017 Sensor information acquisition unit 1018 Three-dimensional data generation unit 1021 Sensor information acquisition unit 1022 Input / output unit 1023 Demultiplexing unit 1024 Decoding unit 1025 Presentation unit 1026 User interface 1027 Control unit 1031 Three-dimensional model learning unit 1032 Three-dimensional model encoding unit 1033 Three-dimensional model decoding unit 1034 Rendering reconstruction unit 1041 Data division unit 1042 Encoding unit 1051 Decoding unit 1052 Data combination unit 1061 Terminal presentation screen 1061a Point cloud button 1061b Mesh button 1062 Terminal presentation screen 1063 Terminal presentation screen 1070 Server 1071 Data generation unit 1072 Point cloud generation unit 1073 Mesh generation unit 1074 Model generation unit 1075 Synchronization unit 1076 Point cloud encoding unit 1077 Mesh encoding unit 1078 Model encoding unit 1079 Multiplexing unit 1080 Data extraction unit 1090 Terminal 1091 Control unit 1092 Decoding unit 1093 Presentation unit 1101 Point cloud sensor 1102 Camera 1110 Data generation unit 1111 Point cloud generation unit 1112 Mesh generation unit 1113 Model generation unit 1120 Terminal 1121 Decoding unit 1122 Synchronous presentation unit 1130 Decoding device 1131 Circuit 1132 Memory 1140 Encoding device 1141 Circuit 1142 Memory 1201 Partial decoding unit 1202 Control unit 1211 Data extraction unit 1212 Control unit 1220 Server 1221 Point cloud generation unit 1222 Mesh generation unit 1223 Point cloud encoding unit 1224 Multiplexing unit 1230 Terminal 1231 Control unit (application execution unit) 1232 Decoding unit 1233 Presentation unit 1241 Object space 1242 Point cloud 1243 Point 1244 Leaf node 1251 Point 1252 Edge vertex 1253 Range 1261 Approximate plane 1262 Range 1263 Vertex information 1271 Triangle 1272 Point 1273 Point cloud 1281 Mesh vertex generation unit 1282 High resolution additional information decoding unit1283 Correction unit 1284 Correction unit 1285 First connection information generation unit 1286 Point cloud generation additional information decoding unit 1287 Point generation unit 1288 Second connection information generation unit 1289 Third connection information generation unit 1290 Point generation unit 1291 Fourth connection information generation unit 1300 Server 1301 Point cloud generation unit 1302 Mesh generation unit 1303 Point cloud encoding unit 1304 Multiplexing unit 1305 Data extraction unit 1310 Terminal 1311 Control unit (application execution unit) 1312 Decoding unit 1313 Presentation unit 1320 Decoding device 1321 Circuit 1322 Memory 1330 Encoding device 1331 Circuit 1332 Memory

Claims

1. A decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in its operation, acquires encoded data; restores a plurality of first vertices based on the encoded data; connects the restored plurality of first vertices to generate first face information including a first face; restores a plurality of second vertices based on the encoded data and the first face; connects points including the restored plurality of first vertices and the plurality of second vertices to generate second face information including the second face; and outputs the plurality of first vertices, the plurality of second vertices, the first face information, and the second face information.

2. The decoding device according to claim 1, wherein the circuit further generates point cloud data including a plurality of points based on the second surface, and outputs the point cloud data.

3. The decoding device of claim 1, wherein the encoded data includes first encoded data including the plurality of first vertices and the first face information, and second encoded data including extension information for decoding the plurality of second vertices and the second face.

4. The decoding device according to claim 3, wherein the encoded data includes a header and a data unit, and the data unit includes the first encoded data and the second encoded data in that order.

5. The decoding device according to claim 3, wherein the encoded data includes a first data unit including a first header and the first encoded data, and a second data unit including a second header and the second encoded data.

6. The decoding device according to claim 3, wherein the circuit further decodes a portion of the first encoded data and the second encoded data, and outputs either (1) the first encoded data or (2) the first encoded data and the second encoded data.

7. The decoding device according to claim 6, wherein said circuit further obtains number information indicating the number of formats that can be output, and determines the data to be output based on said number information.

8. The decoding device according to claim 6, wherein said circuit further acquires type information indicating a type of format that can be output, and determines said type information and data to be output.

9. The decoding device according to claim 3, wherein the circuit further acquires (1) the first encoded data and (2) either the first encoded data or the second encoded data, and decodes and outputs the acquired (1) the first encoded data or (2) the first encoded data and the second encoded data.

10. An encoding device comprising: a circuit; and a memory connected to the circuit, the circuit performing, in an operation, the following: determining a plurality of first vertices of a three-dimensional object; connecting the plurality of first vertices to generate first surface information including a first surface; determining a plurality of second vertices based on the first surface; connecting a point cloud including the plurality of first vertices and the plurality of second vertices to generate second surface information including a second surface; and generating encoded data including the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information.

11. The encoding device according to claim 10, wherein the circuitry further generates point cloud data including a plurality of points based on the second surface, and the encoded data further includes the point cloud data.

12. The encoding device of claim 10, wherein the circuit further generates first encoded data including the plurality of first vertices and the first face information, and generates second encoded data including the plurality of second vertices and the second face information, the encoded data including the first encoded data and the second encoded data.

13. The encoding device according to claim 12, wherein the encoded data includes a header and a data unit, and the data unit includes the first encoded data and the second encoded data in that order.

14. The encoding device of claim 12, wherein the encoded data includes a first data unit including a first header and the first encoded data, and a second data unit including a second header and the second encoded data.

15. The encoding device according to claim 14, wherein the encoded data includes number information indicating the number of formats that can be output.

16. The encoding device according to claim 14, wherein the encoded data includes type information indicating a type of format that can be output.

17. The encoding device according to claim 12, wherein the circuit further outputs: (1) the first encoded data; and (2) either the first encoded data or the second encoded data.

18. A decoding method comprising: restoring a plurality of first vertices based on encoded data; connecting the restored plurality of first vertices to generate first face information including a first face; restoring a plurality of second vertices based on the encoded data and the first face; connecting a point cloud including the restored plurality of first vertices and the plurality of second vertices to generate second face information including the second face; and outputting the plurality of first vertices, the plurality of second vertices, the first face information, and the second face information.

19. An encoding method comprising: determining a plurality of first vertices of a three-dimensional object; connecting the plurality of first vertices to generate first surface information including a first surface; determining a plurality of second vertices based on the first surface; connecting a point cloud including the plurality of first vertices and the plurality of second vertices to generate second surface information including the second surface; and generating encoded data including the plurality of first vertices, the plurality of second vertices, the first surface information, and the second surface information.

20. An apparatus comprising: a circuit; and a memory connected to the circuit, the circuit, in an operation, determining a type of three-dimensional data to be decoded; and extracting three-dimensional data of the determined type from the encoded data, the encoded data including (1) a plurality of first vertices of a three-dimensional object; (2) first surface information including a first surface generated by connecting the plurality of first vertices; (3) a plurality of second vertices determined based on the first surface; and (4) a second surface generated by connecting a point cloud including the plurality of first vertices and the plurality of second vertices, the type of the extracted three-dimensional data including a first format and a second format, when the type of three-dimensional data to be decoded is determined to be the first format, first encoded data including the plurality of first vertices and the first surface information is extracted, and when the type of three-dimensional data to be decoded is determined to be the second format, second encoded data including extension information for decoding the first encoded data and the plurality of second vertices and the second surface is extracted.

Images