Decoding device, decoding method, and encoding device

Abstract

A decoding device (1130) is provided with a circuit (1131) and a memory (1132) connected to the circuit (1131). The circuit (1131) in operation: acquires encoded data including encoding system information and identification information, the encoding system information including first data representing a three-dimensional object and second data representing the three-dimensional object and indicating one encoding method, and the identification information indicating a three-dimensional space including the three-dimensional object; decodes the first data and the second data on the basis of the encoded data; renders the first data to generate first presentation data for presentation; renders the second data to generate second presentation data for presentation; and makes a presentation after switching from the generated second presentation data to the first presentation data.

Description

Decoding device, decoding method, and encoding device

[0001] The present disclosure relates to a decoding device, a decoding method, and an encoding device.

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a representation called a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. In a point cloud, the position and color of the points are stored. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just as with two-dimensional moving images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] Furthermore, compression of point clouds is partially supported by a public library (Point Cloud Library) that performs processing related to point clouds.

[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1).

[0006] International Publication No. 2014 / 020663

[0007] An object of the present disclosure is to provide a decoding device and the like that can appropriately present first presentation data and second presentation data based on first data and second data.

[0008] A decoding device according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit, and in operation, the circuit acquires encoded data including first data representing a three-dimensional object, encoding method information indicating one of encoding methods including second data representing the three-dimensional object, and identification information indicating a three-dimensional space in which the three-dimensional object is included, decodes the first data and the second data corresponding to the three-dimensional space based on the encoded data, renders the first data to generate first presentation data for presentation, renders the second data to generate second presentation data for presentation, and switches from the generated second presentation data to the first presentation data for presentation.

[0009] A decoding device according to one aspect of the present disclosure is a decoding device that decodes first data representing a three-dimensional object, the decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, decodes encoding method information that represents the three-dimensional object and indicates a second encoding method that is different from the first encoding method of the first data; decodes second data of the second encoding method indicated by the encoding method information; and the second data is used to generate second presentation data for presentation.

[0010] A coding device according to a sixteenth aspect of the present disclosure is a coding device for encoding first data representing a three-dimensional object, comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, generates coding method information representing the three-dimensional object and indicating a second coding method different from the first coding method of the first data, generates second data of the second coding method indicated by the coding method information, generates a bitstream including the coding method information and the second data, and the second data is used to generate second presentation data for presentation.

[0011] These comprehensive or specific aspects may be realized as a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0012] The decoding device and the like of the present disclosure can appropriately present the first presentation data and the second presentation data based on the first data and the second data.

[0013] FIG. 1 is a diagram illustrating an example of the configuration of a three-dimensional data encoding / decoding system according to an embodiment. FIG. 2 is a diagram illustrating the configuration of point cloud data. FIG. 3 is a diagram illustrating an example of the configuration of a data file in which information about point cloud data is described. FIG. 4 is a diagram illustrating the configuration of three-dimensional mesh data. FIG. 5 is a diagram illustrating an example of the configuration of a data file in which information about three-dimensional mesh data is described. FIG. 6 is a diagram illustrating a three-dimensional model. FIG. 7 is a diagram illustrating types of three-dimensional data. FIG. 8 is a diagram illustrating a process for encoding three-dimensional data. FIG. 9 is a diagram illustrating a process for decoding three-dimensional data. FIG. 10 is a diagram illustrating tiles and slices of three-dimensional data in a two-dimensional manner. FIG. 11 is a diagram illustrating an example of a terminal presentation screen that can be switched in response to a user request. FIG. 12 is a diagram illustrating an example of a terminal presentation screen that can be automatically switched in response to a user operation. FIG. 13 is a block diagram illustrating an example of the functional configuration of a server and a terminal. FIG. 14 is a block diagram illustrating another example of a data generation unit of the server. FIG. 15 is a diagram illustrating a process for synchronizing coordinate systems. FIG. 16 is a diagram illustrating a process for synchronizing coordinate systems. FIG. 17 is a diagram illustrating the relationship between three-dimensional space and encoded data. FIG. 18 is a diagram showing an example of the syntax of a coding scheme unit. FIG. 19 is a diagram showing an example of the syntax of a coded point group. FIG. 20 is a diagram showing an example of the syntax of a coded mesh. FIG. 21 is a diagram showing an example of the syntax of a coded 3D model. FIG. 22 is a diagram showing an example of the syntax of 3D data information. FIG. 23 is a diagram for explaining the data structure of a coded point group. FIG. 24 is a diagram for explaining the data structure of a coded mesh. FIG. 25 is a diagram for explaining the data structure of a coded 3D model. FIG. 26 is a diagram showing an example of a plurality of 3D spaces in two dimensions. FIG. 27 is a diagram showing an example of a bounding box. FIG. 28 is a diagram showing an example of the syntax of 3D space information. FIG. 29 is a flowchart showing an example of partial decoding. FIG. 30 is a diagram showing an example of a 3D space region to be partially decoded. FIG. 31 is a diagram showing an example of the data structure of a coded point group to be partially decoded.FIG. 32 is a diagram showing an example of the data structure of a partially decoded encoded mesh. FIG. 33 is a diagram showing an example of the data structure of a partially decoded encoded 3D model. FIG. 34 is a diagram showing an example of coordinate systems for different types of 3D data that are not spatially synchronized. FIG. 35 is a diagram showing an example of the syntax of 3D data information. FIG. 36 is a diagram showing an example of the syntax of 3D space information. FIG. 37 is a diagram showing an example of the functional configuration of a terminal. FIG. 38 is a flowchart showing an example of spatial synchronization processing. FIG. 39 is a diagram showing an example of the configuration of a decoding device. FIG. 40 is a flowchart showing an example of a decoding method by the decoding device. FIG. 41 is a flowchart showing another example of a decoding method by the decoding device. FIG. 42 is a diagram showing an example of the configuration of an encoding device. FIG. 43 is a flowchart showing an example of an encoding method by the encoding device.

[0014] A decoding device according to a first aspect of the present disclosure includes a circuit and a memory connected to the circuit, and in operation, the circuit acquires encoded data including first data representing a three-dimensional object, encoding method information indicating one of encoding methods including second data representing the three-dimensional object, and identification information indicating a three-dimensional space in which the three-dimensional object is included, decodes the first data and the second data corresponding to the three-dimensional space based on the encoded data, renders the first data to generate first presentation data for presentation, renders the second data to generate second presentation data for presentation, and switches from the generated second presentation data to the first presentation data for presentation.

[0015] According to this, the first presentation data and the second presentation data are generated based on the first data and the second data corresponding to the three-dimensional space, and the second presentation data is switched to the first presentation data for presentation, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing a spatial misalignment. Thus, the first presentation data and the second presentation data can be presented appropriately.

[0016] A decoding device according to a second aspect of the present disclosure is the decoding device according to the first aspect, wherein the first data is point cloud data representing the three-dimensional object.

[0017] Therefore, the second presentation data is switched to the first presentation data based on the point cloud data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0018] A decoding device according to a third aspect of the present disclosure is the decoding device according to the first or second aspect, wherein the second data is mesh data representing the three-dimensional object.

[0019] Therefore, the second presentation data based on mesh data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0020] A decoding device according to a fourth aspect of the present disclosure is a decoding device according to the first or second aspect, wherein the second data is three-dimensional model data representing the three-dimensional object, and the three-dimensional model data indicates a machine learning model obtained by machine learning multiple sets of gazes and two-dimensional images.

[0021] Therefore, the second presentation data based on the three-dimensional model data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0022] A decoding device according to a fifth aspect of the present disclosure is a decoding device according to the first or second aspect, wherein the second data is a two-dimensional image of the three-dimensional object when viewed from a predetermined line of sight direction.

[0023] Therefore, the second presentation data based on a two-dimensional image is switched to the first presentation data, so that the two pieces of data representing a three-dimensional object can be switched and presented without causing any spatial misalignment.

[0024] A decoding device according to a sixth aspect of the present disclosure is a decoding device according to any one of the first to fifth aspects, wherein the circuit further receives a request from a user to switch presentation data, and in the presentation, switches from the second presentation data to the first presentation data in response to the switching request.

[0025] Therefore, the switching can be performed at a timing designated by the user.

[0026] A decoding device according to a seventh aspect of the present disclosure is a decoding device according to any one of the first to fifth aspects, wherein the circuit further accepts an operation from a user to change the presentation mode, and during the presentation, changes the presentation mode in accordance with the operation, and switches from the second presentation data to the first presentation data in accordance with the change.

[0027] Therefore, switching can be performed at a timing according to the user's operation.

[0028] A decoding device according to an eighth aspect of the present disclosure is a decoding device according to any one of the first to fifth aspects, wherein the circuit, in the acquisition step, acquires the encoded data from the encoding device via a communication network, and, in the presentation step, switches from the second presentation data to the first presentation data and presents the data depending on the bandwidth of the communication network.

[0029] Therefore, switching can be performed according to the bandwidth of the communication network; for example, when the bandwidth of the communication network changes from less than a specified bandwidth to greater than or equal to the specified bandwidth, the second presentation data can be switched to the first presentation data and presented.

[0030] A decoding device according to a ninth aspect of the present disclosure is a decoding device according to any one of the first to fifth aspects, wherein the circuit switches from the second presentation data to the first presentation data during the presentation depending on the available capabilities of the circuit.

[0031] Therefore, switching can be performed according to the capacity of the available circuit; for example, when the capacity of the available circuit changes from less than a predetermined capacity to greater than or equal to the predetermined capacity, the second presentation data can be switched to the first presentation data and presented.

[0032] A decoding device according to a tenth aspect of the present disclosure is a decoding device according to any one of the first to ninth aspects, wherein the encoded data includes synchronization information for synchronizing a coordinate system of the first data and a coordinate system of the second data, and the circuit, during the presentation, presents the first presentation data and the second presentation data based on the synchronization information.

[0033] Therefore, the first presentation data and the second presentation data can be switched to the first presentation data after the coordinate systems of the first presentation data and the second presentation data are aligned, so that the two pieces of data representing the three-dimensional object can be switched and presented with less spatial misalignment.

[0034] A decoding device according to an eleventh aspect of the present disclosure is a decoding device according to the tenth aspect, wherein the circuit further determines whether to synchronize the coordinate system of the first data with the coordinate system of the second data, and if it determines that the coordinate system of the first data is to be synchronized with the coordinate system of the second data, the circuit presents the first presentation data and the second presentation data based on the synchronization information in the presentation.

[0035] This allows synchronization processing to be performed when necessary and can be skipped when not necessary, potentially reducing the processing load.

[0036] A decoding device according to a twelfth aspect of the present disclosure is a decoding device according to any one of the first to eleventh aspects, wherein the first data and the second data each have a common configuration.

[0037] This reduces the amount of coded data, thereby reducing communication capacity.

[0038] A decoding device according to a thirteenth aspect of the present disclosure is a decoding device according to any one of the first to twelfth aspects, wherein the encoded data includes spatial information for identifying the three-dimensional space in which the three-dimensional object is included, and the circuit further acquires a target area indicating a portion of the three-dimensional space, and identifies, based on the spatial information, first duplicate data that is a portion of the first data and overlaps with the target area, and decodes the identified first duplicate data during the decoding.

[0039] Therefore, for example, by acquiring only the first duplicate data, the volume of data to be acquired can be reduced, thereby reducing communication capacity. Also, for example, it is possible to decrypt only the first duplicate data, thereby reducing processing load.

[0040] A decoding method according to a fourteenth aspect of the present disclosure acquires encoded data including first data representing a three-dimensional object, encoding method information indicating one of encoding methods including second data representing the three-dimensional object, and identification information indicating the three-dimensional space in which the three-dimensional object is included, decodes the first data and the second data based on the encoded data, renders the first data to generate first presentation data for presentation, renders the second data to generate second presentation data for presentation, and switches from the generated second presentation data to the first presentation data for presentation.

[0041] According to this, the first presentation data and the second presentation data are generated based on the first data and the second data corresponding to the three-dimensional space, and the second presentation data is switched to the first presentation data for presentation, so that the first presentation data and the second presentation data can be switched and presented without causing a spatial misalignment, thereby making it possible to appropriately present the first presentation data and the second presentation data.

[0042] A decoding device according to a fifteenth aspect of the present disclosure is a decoding device for decoding first data representing a three-dimensional object, the decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, decodes encoding method information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data, decodes second data of the second encoding method indicated by the encoding method information, and the second data is used to generate second presentation data for presentation.

[0043] This allows the second data of the second encoding method indicated by the encoding method information obtained by decoding to be decoded, thereby obtaining second data for generating appropriate second presentation data for presentation.

[0044] A coding device according to a sixteenth aspect of the present disclosure is a coding device for encoding first data representing a three-dimensional object, comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, generates coding method information representing the three-dimensional object and indicating a second coding method different from the first coding method of the first data, generates second data of the second coding method indicated by the coding method information, generates a bitstream including the coding method information and the second data, and the second data is used to generate second presentation data for presentation.

[0045] This allows a bitstream including encoding method information and second data to be generated, and a decoding device that acquires the bitstream can obtain the second data for generating second presentation data for appropriate presentation.

[0046] These comprehensive or specific aspects may be realized as a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0047] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that represents a superordinate concept will be described as optional components.

[0048] (Embodiment) The configuration of a three-dimensional data encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 1001, a three-dimensional data decoding system 1002, a sensor terminal 1003, and an external connection unit 1004.

[0049] The three-dimensional data encoding system 1001 generates encoded data or multiplexed data by encoding three-dimensional data. Note that the three-dimensional data encoding system 1001 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 1001.

[0050] The three-dimensional data encoding system 1001 includes a three-dimensional data generation system 1011, a presentation unit 1012, an encoding unit 1013, a multiplexing unit 1014, an input / output unit 1015, and a control unit 1016. The three-dimensional data generation system 1011 includes a sensor information acquisition unit 1017 and a three-dimensional data generation unit 1018.

[0051] The sensor information acquisition unit 1017 acquires a sensor signal from the sensor terminal 1003 and outputs the sensor signal to the three-dimensional data generation unit 1018. The three-dimensional data generation unit 1018 generates three-dimensional data from the sensor signal and outputs the three-dimensional data to the encoding unit 1013.

[0052] The presentation unit 1012 presents the sensor signal or the three-dimensional data to the user. For example, the presentation unit 1012 displays information or an image based on the sensor signal or the three-dimensional data.

[0053] The encoding unit 1013 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 1014. The additional information includes, for example, a sensor signal.

[0054] The multiplexing unit 1014 generates multiplexed data by multiplexing the coded data input from the coding unit 1013, the control information, and the additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0055] The input / output unit 1015 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 1016 (or the application execution unit) controls each processing unit. That is, the control unit 1016 controls encoding, multiplexing, etc. The control unit 1016 may also control demultiplexing, decoding, or presentation.

[0056] The sensor signal may be input to the encoding unit 1013 or the multiplexing unit 1014. The input / output unit 1015 may output the three-dimensional data or the encoded data directly to the outside.

[0057] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 1001 is input to the three-dimensional data decoding system 1002 via the external connection unit 1004 .

[0058] The three-dimensional data decoding system 1002 generates three-dimensional data by decoding encoded data or multiplexed data. Note that the three-dimensional data decoding system 1002 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 1002.

[0059] The three-dimensional data decoding system 1002 includes a sensor information acquisition unit 1021 , an input / output unit 1022 , a demultiplexing unit 1023 , a decoding unit 1024 , a presentation unit 1025 , a user interface 1026 , and a control unit 1027 .

[0060] The sensor information acquisition unit 1021 acquires a sensor signal from the sensor terminal 1003 .

[0061] The input / output unit 1022 acquires a transmission signal, decodes multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 1023 .

[0062] The demultiplexing unit 1023 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoding unit 1024 .

[0063] The decoding unit 1024 reconstructs the point cloud data by decoding the encoded data.

[0064] The presentation unit 1025 presents the point cloud data to the user. For example, the presentation unit 1025 displays information or an image based on the point cloud data. The user interface 1026 acquires instructions based on user operations. The control unit 1027 (or the application execution unit) controls each processing unit. That is, the control unit 1027 controls demultiplexing, decoding, presentation, etc.

[0065] The input / output unit 1022 may acquire point cloud data or encoded data directly from the outside. The presentation unit 1025 may acquire additional information such as a sensor signal and present information based on the additional information. The presentation unit 1025 may also present information based on a user instruction acquired by the user interface 1026.

[0066] The sensor terminal 1003 generates a sensor signal, which is information obtained by a sensor. The sensor terminal 1003 is a terminal equipped with a sensor or a camera, and may be, for example, a mobile object such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

[0067] The sensor signals obtainable by the sensor terminal 1003 include, for example, (1) a signal indicating the distance between the sensor terminal 1003 and an object or the reflectance of the object obtained from a LIDAR, millimeter-wave radar, or infrared sensor, and (2) a signal indicating the distance between a camera and an object or the reflectance of the object obtained from multiple monocular camera images or stereo camera images. The sensor signals may also include the attitude, orientation, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor signals may also include temperature, air pressure, humidity, magnetism, etc.

[0068] The external connection unit 1004 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0069] Next, point cloud data will be described. Fig. 2 is a diagram showing the structure of point cloud data. Fig. 3 is a diagram showing an example of the structure of a data file in which information about point cloud data is written.

[0070] Point cloud data includes data on multiple points. Each point includes location information (three-dimensional coordinates) and attribute information for that location. A collection of multiple points is called a point cloud. For example, a point cloud can represent the three-dimensional shape of an object.

[0071] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.

[0072] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0073] The data file configuration example shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.

[0074] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0075] Next, the three-dimensional mesh data will be described. Fig. 4 is a diagram showing the structure of the three-dimensional mesh data. Fig. 5 is a diagram showing an example of the structure of a data file in which information about the three-dimensional mesh data is written.

[0076] Three-dimensional mesh data is a data format used in computer graphics (CG) and represents the three-dimensional shape of an object using a collection of multiple pieces of surface information. Each piece of surface information represents a polygon such as a triangle or a quadrangle. Three-dimensional mesh data is also called a polygon or polygon mesh.

[0077] The components are a set of three-dimensional point clouds, vertices that are multiple three-dimensional points of the three-dimensional point cloud, edges connecting two vertices at multiple three-dimensional points, and faces surrounded by multiple edges. A three-dimensional point cloud is a set of points that include position information in three-dimensional space and attribute information corresponding to the position information. Note that three-dimensional points may simply be referred to as points.

[0078] A vertex may have attribute information such as color information, reflectance, and normal vector for a three-dimensional point. The relationship between the vertices that make up an edge or a face may be indicated by information called connectivity. Note that a vertex may be expressed as a position. The front and back of a face may be expressed by the direction of the normal vector for the three-dimensional point. Also, a vertex may have attribute information for the face.

[0079] An example of a mesh data file format is an object file. In a mesh data file such as that shown in Fig. 5, the position information G(1) to G(N) of the N vertices that make up the mesh and the attribute information A(1) to A(N) of the vertices are indicated as vertex information. In a mesh data file, the vertex information does not need to include attribute information.

[0080] Furthermore, the attribute information does not have to correspond one-to-one to the vertices. The mesh data file in Fig. 5 shows an example in which the three-dimensional mesh data has M pieces of attribute information A2.

[0081] Face information is represented by a combination of vertex indices, where n[1,3,4] indicates a triangular face made up of three vertices: vertex n=1, vertex n=3, and vertex n=4.

[0082] Furthermore, m[2,4,6] indicates that the attribute information of m=2, the attribute information of m=4, and the attribute information of m=6 in the attribute information A2 correspond to three vertices, respectively. Note that although an example in which a face is formed by three vertices has been shown here, the number of vertices of a face is not limited to three, as long as it is an integer equal to or greater than three. For example, if the face is a quadrangle, the number of vertices is four, and if the face is a polygon, the number of vertices is equal to the number of vertices that form the polygon.

[0083] The attribute information A2 may be represented in a file separate from the mesh data file, or may include pointer information thereto. For example, the attribute information may be stored in a two-dimensional attribute map file, and the attribute map file name and two-dimensional coordinates in the attribute map may be represented by the attribute information A2 of the mesh data file. In this way, the attribute information A2 may be included in the mesh data file or represented in a file separate from the mesh data file. Either method can be used to specify attribute information for a three-dimensional point.

[0084] Next, the three-dimensional model will be described with reference to Fig. 6.

[0085] A three-dimensional model is a model generated based on two-dimensional or three-dimensional data.

[0086] The three-dimensional model learning unit 1031 learns, for example, two-dimensional data (two-dimensional images) or three-dimensional data (point clouds or meshes) to generate a three-dimensional model, which is a network model in which three-dimensional shapes and attribute information corresponding to the three-dimensional shapes are learned using a neural network or the like.

[0087] The three-dimensional model learning unit 1031 may generate a three-dimensional model by learning using Neural Radiance Fields (NeRF) based on two-dimensional images. The three-dimensional model learning unit 1031 may generate a three-dimensional model after converting the two-dimensional images into three-dimensional data by performing photogrammetry using the two-dimensional images. The three-dimensional model may be generated using three-dimensional data acquired by a sensor (distance sensor).

[0088] The three-dimensional model data is the elements that make up the three-dimensional model, and includes information indicating the structure of the network model, feature quantities, etc. The three-dimensional model data includes, for example, information about the components of the neural network. The information about the components includes, for example, multiple layers such as an input layer, an intermediate layer, and an output layer, nodes in each layer, weight coefficients for the nodes, and transformation functions for the nodes.

[0089] The three-dimensional model encoding unit 1032 may encode the three-dimensional model data and transmit the encoded three-dimensional model data.

[0090] The three-dimensional model decoding unit 1033 receives the transmitted encoded three-dimensional model data and decodes the three-dimensional model based on the encoded three-dimensional model data.

[0091] The rendering reconstruction unit 1034 reconstructs (generates) two-dimensional data (two-dimensional image) or three-dimensional data (point cloud or mesh) based on the decoded three-dimensional model. For example, when using a three-dimensional model modeled using NeRF, the rendering reconstruction unit 1034 acquires viewpoint position or line-of-sight vector information, generates rendered two-dimensional data (two-dimensional image) based on the three-dimensional model and the viewpoint position or line-of-sight vector, and outputs the two-dimensional data. The generated two-dimensional data represents a three-dimensional object as seen from the viewpoint position, or a two-dimensional image of the three-dimensional object as seen from the line of sight indicated by the line-of-sight vector. The three-dimensional object is the three-dimensional object of the subject that was the source of the two-dimensional data or three-dimensional data input to the three-dimensional model learning unit 1031.

[0092] Next, the types of three-dimensional data will be explained. Fig. 7 shows the types of three-dimensional data. As shown in Fig. 7, three-dimensional data includes static objects and dynamic objects.

[0093] A static object is three-dimensional data at any time (a certain time). A dynamic object is three-dimensional data that changes over time. Hereinafter, point cloud data at a certain time will be referred to as a PCC frame or frame. Mesh data at a certain time will be referred to as a mesh frame or frame.

[0094] The object may be three-dimensional data with a certain area restriction, such as ordinary video data, or may be three-dimensional data with no area restriction, such as map information.

[0095] There may also be sparse point cloud data (sparse mesh data) and dense point cloud data (dense mesh data) with points of various densities.

[0096] Each processing unit will be described in detail below. Sensor information is acquired by various methods, such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The three-dimensional data generation unit 1018 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 1017. The three-dimensional data generation unit 1018 generates position information (geometry information) as the point cloud data, and adds attribute information for the position information to the position information.

[0097] The three-dimensional data generation unit 1018 may process the point cloud data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 1018 may reduce the amount of data by deleting point clouds with overlapping positions. Furthermore, the three-dimensional data generation unit 1018 may convert the position information (such as by shifting, rotating, or normalizing) or process the point cloud data to generate mesh data. Furthermore, the three-dimensional data generation unit 1018 may render the attribute information.

[0098] In FIG. 1, the three-dimensional data generation system 1011 is included in the three-dimensional data encoding system 1001, but it may be provided independently outside the three-dimensional data encoding system 1001.

[0099] The encoding unit 1013 generates encoded data by encoding the three-dimensional data based on a predetermined encoding method. The encoding methods include G-PCC (an encoding method using position information), V-PCC (an encoding method using a video codec), Draco (a mesh encoding method), and V-DMC (a mesh encoding method). The encoding method is not limited to these methods, and may be, for example, a method of encoding a dynamic mesh, or another method that combines these methods.

[0100] The decoding unit 1024 decodes the coded data based on a predetermined coding method to decode the three-dimensional data.

[0101] The multiplexing unit 1014 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the encoded data of the three-dimensional data, the multiplexing unit 1014 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 1014 may also multiplex attribute information related to sensor information or point cloud data.

[0102] Multiplexing methods or file formats include ISOBMFF, and ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0103] The demultiplexing unit 1023 extracts the coded data of the three-dimensional data, other media, time information, and the like from the multiplexed data.

[0104] The input / output unit 1015 transmits the multiplexed data using a method suited to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 1015 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0105] The communication protocol used may be http, ftp, TCP, UDP, etc. A pull-type communication method or a push-type communication method may be used.

[0106] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter waves, etc. are used.

[0107] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0108] Next, a process of dividing three-dimensional data into one or more three-dimensional data will be described. Fig. 8 is a diagram for explaining a process of encoding three-dimensional data. Fig. 9 is a diagram for explaining a process of decoding three-dimensional data.

[0109] 8 , the data dividing unit 1041 divides the three-dimensional data into one or more three-dimensional spaces and generates one or more divided three-dimensional data (i.e., one or more divided three-dimensional data). The encoding unit 1042 may encode one or more divided three-dimensional data to generate encoded data. The data dividing unit 1041 and the encoding unit 1042 may be included in a single encoding device as components of the single encoding device, or may be included in separate devices.

[0110] Each of the one or more three-dimensional spaces may be referred to as a tile or a space. The three-dimensional space may be, for example, a bounding box. Furthermore, the three-dimensional data included in each of the divided three-dimensional spaces may be referred to as a slice. A slice is divided three-dimensional data and includes any of a point cloud, a mesh, and a three-dimensional model having position information (geometry) or attribute information (attribute). Each of the multiple slices is coded by the coding unit 1042 for each component element and output as coded data. The coded data includes the coded multiple slices.

[0111] As shown in FIG. 9 , in the decoding process, the decoding unit 1051 decodes one or more pieces of divided three-dimensional data (one or more slices) based on the encoded data. The data combining unit 1052 combines the one or more pieces of divided three-dimensional data to restore (generate) three-dimensional data. The decoding unit 1051 and the data combining unit 1052 may be included in a single decoding device as components of the single decoding device, or may be included in separate devices. The one or more pieces of divided three-dimensional data decoded by the decoding unit 1051 do not need to be combined. The decoding unit 1051 may decode some of the one or more pieces of divided three-dimensional data based on part of the encoded data and output the decoded some of the divided three-dimensional data. In this case, the decoding device does not need to have the data combining unit 1052.

[0112] FIG. 10 is a two-dimensional schematic diagram showing tiles and slices of three-dimensional data.

[0113] When encoding multiple slices, the encoding device may encode them using a dependency relationship between the multiple slices, or may encode them without using the dependency relationship. When encoding without using the dependency relationship, the encoding device can encode each slice independently, and can reduce processing time by encoding multiple slices using parallel processing. Furthermore, when multiple slices are encoded without using the dependency relationship, the decoding device can decode each slice independently, and can reduce processing time by decoding multiple slices using parallel processing. Furthermore, the decoding device can reduce the amount of processing by performing partial decoding, which decodes some of the multiple slices.

[0114] When encoding using a dependency relationship, the encoding device signals an identifier indicating the dependency relationship and encodes the data in order starting with the dependent data. When multiple slices are encoded using a dependency relationship, the decoding device decodes the data in order starting with the dependent data based on the identifier.

[0115] The division of the three-dimensional data may be performed using any number of divisions and any division method. The division of the three-dimensional data may involve determining the shape of an object and dividing each object into multiple three-dimensional points. The division of the three-dimensional data may also be performed based on the number of three-dimensional points included in a slice. In other words, an upper limit on the number of three-dimensional points included in one slice may be set. The three-dimensional data may also be divided based on whether or not it is included in the three-dimensional space (tile information) using map information or location information. Multiple tile shapes may overlap.

[0116] By dividing the three-dimensional data into a plurality of divided three-dimensional data in this way, adaptive encoding according to the content or object and parallel processing in decoding become possible.

[0117] Next, a method for selecting three-dimensional data to be presented or transmitted from among a plurality of three-dimensional data will be described.

[0118] The server stores multiple pieces of three-dimensional data for the same space. The server stores, for example, point cloud data and mesh data for the same space. The server is an example of an encoding device. The terminal switches the three-dimensional data acquired from the server based on the purpose of the terminal and presents the switched three-dimensional data. The terminal may be, for example, a terminal that analyzes three-dimensional data. In this case, the terminal may switch the three-dimensional data to be presented based on the purpose of analysis or presentation, or user operation. The terminal is an example of a decoding device.

[0119] The switching of three-dimensional data may involve switching between presenting a point cloud or a mesh as the three-dimensional data. Furthermore, the switching of three-dimensional data may involve switching between transmitting a point cloud or a mesh as the three-dimensional data. For example, the terminal may transmit a user's selection result to a server, receive (download) three-dimensional data based on the selection result from the server, and present the received three-dimensional data. The three-dimensional data (point cloud or mesh) may or may not be encoded by the server. If the three-dimensional data is encoded, the terminal may receive the encoded three-dimensional data from the server, decode the three-dimensional data based on the received encoded three-dimensional data, and present the decoded three-dimensional data.

[0120] FIG. 11 is a diagram showing an example of a terminal presentation screen that can be switched in response to a user request.

[0121] As shown in FIG. 11 , the terminal presentation screen 1061 may be switched in response to a user request. The terminal presentation screen includes a point cloud button 1061a and a mesh button 1061b as UIs for accepting user requests. The point cloud button 1061a is a button for accepting a request to present an encoded point cloud (encoded point cloud data). The mesh button 1061b is a button for accepting a request to present an encoded mesh (encoded mesh data). When the user inputs the point cloud button 1061a, the terminal accepts the request to present the encoded point cloud and notifies (transmits) the request as a result of the selection of presentation data to the server. When the user inputs the mesh button 1061b, the terminal accepts the request to present an encoded mesh and notifies (transmits) the request as a result of the selection of presentation data to the server. Note that presentation may also be expressed as display.

[0122] When the server receives the request as the selection result, it transmits three-dimensional data according to the selection result to the terminal. If the received request is a request to present an encoded point cloud, the server transmits the encoded point cloud to the terminal. If the received request is a request to present an encoded mesh, the server transmits the encoded mesh to the terminal.

[0123] Meshes are suitable for presentation because they require a small processing load. On the other hand, point clouds are suitable for measurement because they provide highly accurate positional information of 3D models represented by 3D point clouds. For example, by selecting the type of 3D data appropriate for the required purpose, such as selecting a mesh when observing a 3D model and a point cloud when measuring, a user can reduce the presentation processing load or use the data for accurate measurements.

[0124] FIG. 12 is a diagram showing an example of a terminal presentation screen that is automatically switched in response to a user operation.

[0125] The terminal performs a process of enlarging the three-dimensional data by accepting a user operation. The terminal may present mesh data as shown on a terminal presentation screen 1062 at a magnification smaller than a predetermined magnification, and may present point cloud data as shown on a terminal presentation screen 1063 at a magnification equal to or larger than the predetermined magnification. For example, the terminal may present mesh data at a magnification smaller than the predetermined magnification, and when the mesh data is enlarged to the predetermined magnification in response to a user operation and presented, download point cloud data from a server corresponding to a portion of the three-dimensional data that is highlighted by the enlargement (presented by the enlargement), and decode and present the downloaded point cloud data.

[0126] Although the above example illustrates switching the type of 3D data presented in response to an operation to enlarge or reduce the 3D data, other operations may also be used to switch the type of 3D data presented. For example, when a terminal receives a user's tap on a portion of presented mesh data, the terminal may switch to and present point cloud data corresponding to the portion of the mesh data selected by the tap. The terminal may present the point cloud data superimposed on the mesh data, or may present the point cloud data independently from the mesh data (i.e., without presenting the mesh data). The terminal may predict the enlarged portion before presentation at a predetermined magnification, download the point cloud data corresponding to the enlarged portion, and decode and present the downloaded point cloud data. Furthermore, if the point cloud data and the mesh data have been spatially synchronized (coordinate alignment) in advance, the terminal may present the downloaded point cloud data as is without spatial synchronization. If the point cloud data and the mesh data have not been spatially synchronized, the terminal may align the point cloud data with the mesh data (or the coordinate system of the mesh data) based on synchronization information related to spatial synchronization and present the point cloud data. The terminal may also present the point cloud data without alignment. Although the above describes an example in which point cloud data is spatially synchronized with mesh data, the terminal may also present mesh data in spatial synchronization with point cloud data.

[0127] In this way, the terminal downloads only the necessary point cloud data, decodes the downloaded point cloud data, and presents it. This is expected to reduce the communication volume between the server and the terminal, reduce the processing load on the terminal, and reduce the presentation delay.

[0128] Generally, mesh data has a small data size, while point cloud data has a large data size. In other words, when comparing mesh data and point cloud data of the same 3D object in terms of data size, the mesh data is smaller than the point cloud data. Therefore, by first downloading the mesh data and displaying it initially, the terminal can reduce the delay until initial display.

[0129] In addition, by displaying mesh data, the terminal can reduce the processing load required for observing the 3D model. Also, by displaying mesh data and point cloud data simultaneously, the terminal can improve visibility and make measurements easier.

[0130] The 3D data to be presented may be switched based on the communication network bandwidth. The terminal may present mesh data when the bandwidth is narrow and the network speed is slower than a predetermined speed, and present point cloud data when the network speed is equal to or faster than the predetermined speed.

[0131] Furthermore, the 3D data to be presented may be switched depending on the capabilities of the terminal. For example, the 3D data to be presented may be switched based on the processing performance of the terminal or the usage rate of the CPU of the terminal. While presenting point cloud data, the terminal may switch from presenting point cloud data to presenting mesh data if the terminal's processing performance or CPU resources are insufficient to process the point cloud data. While presenting mesh data, the terminal may switch from presenting mesh data to presenting point cloud data if the terminal's processing performance or CPU resources are determined to be sufficient to process the mesh data.

[0132] Next, another example of switching three-dimensional data will be described.

[0133] When presenting three-dimensional data used at a construction site, the terminal may switch the type of three-dimensional data presented, such as presenting the exterior of a building or map as mesh data, and presenting point cloud data when the data includes areas that require measurement, such as distortions, cracks, and warping of the building.

[0134] The terminal can switch the type of three-dimensional data it presents, such as presenting mesh data when it wants to show the appearance of spaces such as stadiums, halls, and factories, including lighting brightness, color, and atmosphere, and presenting point cloud data when it wants to measure studio shape, equipment placement, aisle clearance, and so on.

[0135] Also, three-dimensional data with different resolutions may be switched and presented.

[0136] When rendering and presenting a 3D model, the terminal may switch between multiple pieces of point cloud data with different resolutions, multiple pieces of mesh data with different resolutions, and the 3D model. For example, the terminal may present low-resolution mesh data when presenting 3D data far from the viewpoint, and switch to high-resolution mesh data when the viewpoint is closer. This can improve the accuracy of the 3D data display.

[0137] When analyzing the movements of athletes in sports, the terminal may present mesh data until a specific movement or location is identified, and then switch to presenting point cloud data of the corresponding part after the specific movement or location is identified, and use the point cloud data to measure the three-dimensional shape.

[0138] The terminal may switch between 3D models with different resolutions in a video game or the like. For example, the terminal may use high-resolution 3D data to present 3D data of important parts and low-resolution 3D data to present 3D data of unimportant parts. This can improve overall processing performance. Note that whether each 3D model is important is determined, for example, by whether it is related to the action of the video game (e.g., in a shooting game, the player, target, gun, etc. are important) or whether it is related to the player's selection (e.g., product information is important in a purchasing scene). Note that whether each 3D model is important may be set in advance for each video game.

[0139] When displaying the terrain data as a three-dimensional model, the terminal may switch between terrain data with different resolutions, thereby enabling more detailed terrain information to be displayed as needed.

[0140] For example, when displaying thumbnails on the web, the terminal may present low-resolution three-dimensional data, and when measuring or presenting in an application using three-dimensional data selected from the thumbnail, high-resolution three-dimensional data may be used for measurement or presentation. For example, when displayed on a two-dimensional display terminal, the three-dimensional data is presented as two-dimensional data of a three-dimensional object (subject) viewed from a specific viewpoint in a specific direction. In other words, the three-dimensional data may be divided into camera information indicating a specific viewpoint and a specific direction and two-dimensional data of a three-dimensional object viewed from the specific viewpoint in the specific direction.

[0141] When display is the priority, the terminal presents mesh data, which requires less processing, and when measuring, replaces it with point cloud data suitable for measurement, making it possible to realize applications with less processing.

[0142] Next, a description will be given of the configuration of the server 1070 and the terminal 1090. Fig. 13 is a block diagram showing an example of the functional configuration of the server and the terminal.

[0143] The server 1070 includes a data generation unit 1071 , a synchronization unit 1075 , a point cloud encoding unit 1076 , a mesh encoding unit 1077 , a model encoding unit 1078 , a multiplexing unit 1079 , and a data extraction unit 1080 .

[0144] The data generation unit 1071 generates three-dimensional data based on at least one of two-dimensional data and three-dimensional data. The generated three-dimensional data includes at least two of point cloud data, mesh data, and three-dimensional model data. The data generation unit 1071 includes a point cloud generation unit 1072, a mesh generation unit 1073, and a model generation unit 1074. The data generation unit 1071 may include at least two of the point cloud generation unit 1072, the mesh generation unit 1073, and the model generation unit 1074. The point cloud generation unit 1072 generates point cloud data based on at least one of the two-dimensional data and the three-dimensional data. The mesh generation unit 1073 generates mesh data based on at least one of the two-dimensional data and the three-dimensional data. The model generation unit 1074 generates three-dimensional model data by performing machine learning based on at least one of the two-dimensional data and the three-dimensional data.

[0145] The two-dimensional data input to the data generation unit 1071 may be a two-dimensional image acquired by a camera. The three-dimensional data input to the data generation unit 1071 may be point cloud data acquired by a sensor such as LiDAR of a space such as a construction site, factory, or office. The data generation unit 1071 may generate color information corresponding to each point included in the point cloud data of the three-dimensional data as attribute information using a two-dimensional image of the two-dimensional data. The three-dimensional data generated by the data generation unit 1071 may be divided into any space. The point cloud data, mesh data, and three-dimensional model data may each be divided into any space.

[0146] The synchronization unit 1075 synchronizes the spatial positions of the point cloud data, mesh data, and 3D model data generated by the data generation unit 1071 or the time of each data (playback time, decoding time, acquisition time, etc.). The time of each data is the playback time, decoding time, acquisition time, etc. The synchronization unit 1075 may generate synchronization information for synchronization without synchronizing the point cloud data, mesh data, and 3D model data. The synchronization unit 1075 may synchronize at least two types of 3D data among the point cloud data, mesh data, and 3D model data generated by the data generation unit 1071, or may perform processing to generate synchronization information (synchronization signal) for synchronization, and may not necessarily perform processing to synchronize three types of 3D data (synchronization processing).

[0147] The point cloud encoding unit 1076 encodes the point cloud data after synchronization processing has been performed by the synchronization unit 1075. Note that the point cloud encoding unit 1076 does not necessarily have to encode the point cloud data. The point cloud data may be encoded in advance, or may be encoded in response to a request from the terminal 1090.

[0148] The mesh encoding unit 1077 encodes the mesh data after the synchronization process has been performed by the synchronization unit 1075 .

[0149] The model encoding unit 1078 encodes the three-dimensional model data after the synchronization process has been performed by the synchronization unit 1075 .

[0150] The multiplexing unit 1079 multiplexes the encoded point cloud data (encoded point cloud), the encoded mesh data (encoded mesh data), the encoded 3D model data, and the synchronization information using a predetermined format or a predetermined multiplexing method. Note that multiplexing by the multiplexing unit 1079 does not have to be performed. In this case, the server 1070 does not have to be equipped with the multiplexing unit 1079.

[0151] The data extraction unit 1080 extracts a portion of the multiplexed three-dimensional data in response to a request from the terminal 1090, and transmits the extracted portion of the three-dimensional data to the terminal 1090. Note that data extraction by the data extraction unit 1080 does not have to be performed. In this case, the server 1070 does not have to be equipped with the data extraction unit 1080. If data extraction by the data extraction unit 1080 is not performed, the server 1070 may transmit to the terminal 1090 the three-dimensional data multiplexed by the multiplexing unit 1079. Furthermore, if multiplexing by the multiplexing unit 1079 is not performed, the server 1070 may transmit to the terminal 1090 the coded point cloud data (coded point cloud), coded mesh data (coded mesh), coded three-dimensional model data (coded three-dimensional model), and synchronization information, or may transmit to the terminal 1090 a bitstream including the coded point cloud data (coded point cloud), coded mesh data (coded mesh), coded three-dimensional model data (coded three-dimensional model), and synchronization information.

[0152] The terminal 1090 includes a control unit 1091 , a decoding unit 1092 , and a presentation unit 1093 .

[0153] The control unit 1091 transmits a request for the part of the three-dimensional data to be presented to the server 1070. The control unit 1091 may accept an operation by the user to specify the part of the three-dimensional data.

[0154] The decoding unit 1092 decodes a portion of the three-dimensional data based on the bit stream (encoded data) acquired from the server 1070 .

[0155] The presentation unit 1093 renders and presents a portion of the decoded three-dimensional data.

[0156] The data generation unit 1071 in Fig. 13 may be realized by a data generation unit 1110 shown in Fig. 14. Fig. 14 is a block diagram showing another example of the data generation unit of the server.

[0157] The data generation unit 1110 includes a point cloud generation unit 1111 , a mesh generation unit 1112 , and a model generation unit 1113 .

[0158] The point cloud generation unit 1111 has the same function as the point cloud generation unit 1072. The point cloud generation unit 1111 acquires point cloud data obtained from the point cloud sensor 1101 and a two-dimensional image obtained from the camera 1102, and generates point cloud data based on the point cloud data and the two-dimensional image. The point cloud data generated by the point cloud generation unit 1111 includes position information of each point and attribute information corresponding to each point indicated by the position information, and includes attribute information (such as color information) extracted from the two-dimensional image.

[0159] The mesh generation unit 1112 generates mesh data based on the point cloud data generated by the point cloud generation unit 1111 .

[0160] The model generation unit 1113 has the same functions as the model generation unit 1074. The model generation unit 1113 acquires point cloud data obtained from the point cloud sensor 1101 and two-dimensional images obtained from the camera 1102, and performs machine learning based on the point cloud data and the two-dimensional images to generate three-dimensional model data.

[0161] The point cloud data, mesh data, and three-dimensional model data may be data generated independently of each other, as described in Fig. 13. The mesh data may be generated from the point cloud data, as described in Fig. 14. Note that the point cloud data may be generated from the mesh data.

[0162] A mesh may be generated from a point cloud, or a point cloud may be generated from a mesh.

[0163] The point cloud data, mesh data, and three-dimensional model data may be generated by the server 1070, or may be generated by a sensor or a terminal 1090 equipped with a sensor. The sensor is, for example, a point cloud sensor 1101 and a camera 1102.

[0164] Next, a process for synchronizing the coordinate systems of a plurality of three-dimensional data will be described. Figures 15 and 16 are diagrams for explaining the process for synchronizing the coordinate systems.

[0165] Even when the three-dimensional data is for the same space, the origin position (origin coordinates) of the local coordinate system handled by the system may differ from the actual coordinates due to differences in the system or coordinate system being handled.

[0166] For example, if mesh data is generated from point cloud data, the origin position of the mesh data and the point cloud data will be the same, but if the mesh data and the point cloud data are generated using different systems, the origin positions of the mesh data and the point cloud data may be different.

[0167] 15 and 16, the origin coordinates of the coordinate system of the mesh data and the origin coordinates of the coordinate system of the point cloud data are both, for example, (x1, y1, z1) in world coordinates, and the coordinate system of the mesh data and the coordinate system of the point cloud data are the same. Note that if these origin coordinates are not the same, they may be corrected to be the same.

[0168] Next, a case where divided three-dimensional space information is synchronized will be described.

[0169] 15 shows a case where the three-dimensional spatial regions for dividing the mesh data and the three-dimensional spatial regions for dividing the point cloud data are the same. Specifically, the number, size, and positions of the three-dimensional spatial regions are the same between the multiple three-dimensional spatial regions in the mesh data coordinate system and the multiple three-dimensional spatial regions in the point cloud coordinate system.

[0170] Focusing on the three-dimensional space area indicated by the thick dotted line, the positions of the origin (black triangle mark) of the bounding box representing the three-dimensional space and the maximum value point (black square mark) of the bounding box are the same in the coordinate system of the mesh data and the coordinate system of the point cloud data.

[0171] In this case, the vertical, horizontal, and depth sizes of this bounding box are the same in the coordinate system of the mesh data and the coordinate system of the point cloud data.

[0172] 16 , the number, size, and positions of the multiple three-dimensional spatial regions in the mesh data coordinate system and the multiple three-dimensional spatial regions in the point cloud coordinate system do not have to be completely identical. The three-dimensional spatial regions in the mesh data coordinate system may be divided into larger regions, and the three-dimensional spatial regions in the point cloud data coordinate system may be further divided into the three-dimensional spatial regions in the mesh data coordinate system. In this example, the combined unit of the multiple three-dimensional spatial regions in the point cloud data coordinate system is identical to the three-dimensional spatial region in the mesh data coordinate system.

[0173] Next, the relationship between the three-dimensional space and the coded data will be described with reference to Fig. 17.

[0174] As described above, the three-dimensional data includes, for example, any one of point cloud data, mesh data, and a three-dimensional model.

[0175] As shown in Fig. 17, when three-dimensional data is divided into three pieces of three-dimensional data in three three-dimensional spaces (tiles or spaces), the encoding device encodes each of the three divided pieces of three-dimensional data, attaches a header to each piece, and creates a data unit. The header signals (assigns) an identifier (Space_ID) of the space to which the encoded data of the data unit belongs, and an identifier (DataUnit_ID) of the data unit.

[0176] The data units are further given a header including an identifier of the data unit or length information of the data unit, and are then unitized to generate coding scheme units.

[0177] Next, the syntax of the coding method unit will be described. Fig. 18 is a diagram showing an example of the syntax of the coding method unit. Fig. 19 is a diagram showing an example of the syntax of the coding point group. Fig. 20 is a diagram showing an example of the syntax of the coding mesh. Fig. 21 is a diagram showing an example of the syntax of the coding 3D model.

[0178] The unit_type indicates the type of data unit stored in the encoding method unit, thereby specifying the type of data unit stored in the encoding method unit.

[0179] The length indicates the length of the data unit.

[0180] data() indicates the body of the data unit.

[0181] In Fig. 19, when unit_type indicates 0, it indicates that the data unit is position information (geometry) of the coded point group. When unit_type indicates 1, it indicates that the data unit is attribute information of the coded point group. When unit_type indicates 2, it indicates that the data unit is metadata of the coded point group.

[0182] In Fig. 20, when unit_type indicates 0, it indicates that the data unit is position information (geometry) of the coded mesh. When unit_type indicates 1, it indicates that the data unit is attribute information of the coded mesh. When unit_type indicates 2, it indicates that the data unit is metadata of the coded mesh.

[0183] In Fig. 21, when unit_type indicates 0, it indicates that the data unit is element 1 of the encoded 3D model. When unit_type indicates 1, it indicates that the data unit is element 2 of the encoded 3D model. When unit_type indicates 2, it indicates that the data unit is metadata of the encoded 3D model.

[0184] 19 to 21 are merely examples and are not limited to the above configurations. These syntaxes may use a partial configuration of the syntax, or may use a type (classification) not described above, or the order of the syntax components may be changed. For example, the syntax of the encoding method unit may have a configuration of the encoding method unit common to multiple encoding methods as shown in FIG. 18, and may include the unit_type, length, and data() shown in FIGS. 19 to 21.

[0185] The coding method unit may further include a header indicating the type of the coding method unit. The coding unit type may include, for example, point_cloud_codec_unit indicating point cloud data, mesh_codec_unit indicating mesh data, and model_codec_unit indicating 3D model data. This allows multiple coding methods to be handled in an integrated manner.

[0186] FIG. 22 is a diagram showing an example of the syntax of three-dimensional data information.

[0187] In the syntax, when multiple encoding methods are stored in one format, the number of 3D data included in the format (number_of_3Dformat) and the type of 3D data (format_type) are indicated, and data of each format may be stored. This makes it possible to handle multiple encoding methods or 3D data in an integrated manner, and also to identify multiple encoding methods or 3D data.

[0188] 3Ddata_info indicates format structure information for storing multiple three-dimensional data.

[0189] number_of_3Dformat indicates the number of 3D formats used.

[0190] format_type indicates the type of format of the three-dimensional data to be stored. For example, the number of format_type and the format corresponding to that number may be defined as follows: When format_type indicates 0, it indicates that the format of the three-dimensional data to be stored is point cloud data. When format_type indicates 1, it indicates that the format of the three-dimensional data to be stored is mesh data. When format_type indicates 2, it indicates that the format of the three-dimensional data to be stored is G-PCC data (g-pcc). When format_type indicates 3, it indicates that the format of the three-dimensional data to be stored is V-DMC data (v-dmc). When format_type indicates 4, it indicates that the format of the three-dimensional data to be stored is three-dimensional model data (3Dmodel).

[0191] Next, the data structure of encoded data of multiple pieces of three-dimensional data will be described for each type of three-dimensional data. Fig. 23 is a diagram for explaining the data structure of an encoded point group. Fig. 24 is a diagram for explaining the data structure of an encoded mesh. Fig. 25 is a diagram for explaining the data structure of an encoded three-dimensional model.

[0192] The encoding device divides each of the multiple types of three-dimensional data into multiple pieces of three-dimensional data for each of multiple spatial regions, encodes each of the multiple pieces of divided three-dimensional data (i.e., multiple divided three-dimensional data), and generates encoded data.

[0193] A header is attached to each piece of encoded data, and at least one of a data_unit_id and a space_id is stored.

[0194] Here, data_unit_id is an identifier that identifies a data unit in the coded data and is unique within the coded data. Furthermore, space_id indicates identification information for a spatial region. If data_unit_id or space_id is common to multiple pieces of three-dimensional data, the same value is indicated in the multiple pieces of three-dimensional data.

[0195] 23 to 25, the data unit with data_unit_id=0 in the encoded point group, the data unit with data_unit_id=3 in the encoded mesh, and the data unit with data_unit_id=0 in the encoded 3D model are all assigned space_id=1, which means that they are 3D data contained in a common 3D space indicated by Space_ID #1.

[0196] The data and headers may be included in a bitstream structure such as a data unit or encoding method, or may be stored in a predetermined file format such as each box of ISOBMFF.

[0197] Next, three-dimensional space information will be described. Fig. 26 is a diagram showing an example of a plurality of three-dimensional spaces in two dimensions. Fig. 27 is a diagram showing an example of a bounding box. Fig. 28 is a diagram showing an example of the syntax of three-dimensional space information.

[0198] In the syntax of the three-dimensional space information, 3Dspace_info is information indicating the divided three-dimensional space. 3Dspace_info can be used for partial decoding.

[0199] The number_of_space indicates the number of divided three-dimensional spaces.

[0200] The space_id indicates the identifier of the divided three-dimensional space.

[0201] The three-dimensional space information includes bounding box information as information for defining the bounding box shown in FIG.

[0202] The bounding box information includes bounding_box_xyz and bounding_box_whd.

[0203] bounding_box_xyz indicates the coordinates of the reference point of the bounding box, and in the example of Fig. 27, is expressed as x, y, and z coordinate values ​​(x0, y0, z0).

[0204] The bounding_box_whd field indicates the size of the bounding box, which in the example of Fig. 27 is expressed as width w, height h, and depth d (w0, h0, d0).

[0205] The three-dimensional spatial information may also include an identifier of a data unit for each piece of coded data. However, the three-dimensional spatial information does not necessarily have to include the identifier. In other words, the identifier does not necessarily have to be signaled.

[0206] The pointcloud_id indicates the identifier of the data unit of the encoded point cloud of the space corresponding to the space_id.

[0207] mesh_id indicates the identifier of the data unit of the encoding mesh of the space corresponding to space_id.

[0208] model_id indicates the identifier of the data unit of the coded three-dimensional model of the space corresponding to space_id.

[0209] When the data unit does not indicate a space_id but indicates a data_unit_id, an identifier of the data unit for each piece of coded data may be stored in the information indicating each space of the three-dimensional space information, thereby allowing the three-dimensional space information to be associated with the divided three-dimensional coded data.

[0210] Furthermore, when a space_id is indicated in a data unit, the three-dimensional spatial information may be associated with an identifier of the data unit for each encoded data by the space_id. In this case, the identifier of the data unit for each encoded data does not need to be stored.

[0211] The division method, the origin of each divided space, and the bounding box size may be the same for mesh data and point cloud data, thereby sharing the three-dimensional spatial information of the point cloud data and the mesh data. Furthermore, the same three-dimensional spatial information may be used for both point cloud data and mesh data. In this way, the three-dimensional spatial information may be shared or the same three-dimensional spatial information may be used between multiple different types of three-dimensional data. Sharing the three-dimensional spatial information facilitates switching between different types of three-dimensional data (e.g., switching between presentations or transmissions). Furthermore, in a format that handles multiple three-dimensional data in an integrated manner, three-dimensional spatial information does not need to be provided for each piece of three-dimensional data; instead, one piece of three-dimensional spatial information can be used for each piece of three-dimensional data, thereby reducing the amount of three-dimensional spatial information.

[0212] In addition to the point cloud data and mesh data, the three-dimensional spatial information of the three-dimensional model may be synchronized with other types of three-dimensional data, or may be made common with the three-dimensional spatial information of other types of three-dimensional data.

[0213] Next, the relationship between the data structure of three-dimensional data and partial decoding will be described. Fig. 29 is a flowchart showing an example of partial decoding. Fig. 30 is a diagram showing an example of a three-dimensional spatial region that is the target of partial decoding. Fig. 31 is a diagram showing an example of the data structure of an encoded point group that is partially decoded. Fig. 32 is a diagram showing an example of the data structure of an encoded mesh that is partially decoded. Fig. 33 is a diagram showing an example of the data structure of an encoded three-dimensional model that is partially decoded.

[0214] In partial decoding, the decoding device first determines a three-dimensional spatial region to be partially decoded (S1001).

[0215] Next, the decoding device uses the three-dimensional spatial information (3Dspace_info) to identify an area that overlaps with the target three-dimensional spatial area from the bounding box information of multiple three-dimensional spatial areas, and obtains the space_id corresponding to the identified area (S1002).

[0216] Next, the decoding device obtains and decodes the data unit having the obtained space_id from the encoded data (S1003). As a result, the decoding device performs partial decoding, which decodes a part of the three-dimensional data. In partial decoding, the decoding device decodes only a part of the three-dimensional data without decoding the entire three-dimensional data.

[0217] For example, as shown in FIG. 30, when the three-dimensional space area to be partially decoded is the area indicated by the thick line, the space_id of the three-dimensional space to be acquired is determined to be #2 from the three-dimensional space information.

[0218] Then, as shown in FIGS. 31 to 33, the data unit associated with Space_id=#2 from the coded data of the multiple types of three-dimensional data is obtained and decoded.

[0219] The decoding device may also obtain a data unit ID instead of space_id from the three-dimensional spatial information, and obtain and partially decode a data unit having the obtained data unit ID.

[0220] Next, a case where spatial synchronization is not achieved will be described. Fig. 34 is a diagram showing an example of coordinate systems of different types of three-dimensional data where spatial synchronization is not achieved. Fig. 35 is a diagram showing an example of the syntax of three-dimensional data information. Fig. 36 is a diagram showing an example of the syntax of three-dimensional space information.

[0221] As shown in Fig. 34, when the origin coordinates of data in the same space are different due to differences in coordinate systems, the encoding device may align these origin coordinates by calculating and correcting the relative coordinate values ​​(x1-x2, y1-y2, z1-z2). The encoding device may notify (transmit) the calculated relative coordinate values ​​to the decoding device (terminal) as synchronization information.

[0222] The synchronization information may be indicated by the relative position of the point cloud data (origin) relative to the position of the mesh data (origin), or may be indicated by the relative position of the mesh data relative to the position of the point cloud data.

[0223] The same applies to three-dimensional model data. In this way, the synchronization information may be indicated by the relative position of the positions (origins) of different types of three-dimensional data. Note that when there are three or more types of three-dimensional data, the relative position is calculated based on any one type of three-dimensional data.

[0224] The three-dimensional data information may include space_sync_information indicating synchronization information, as shown in Fig. 35. The space_sync_information indicates synchronization information of the three-dimensional space, and indicates, for example, the amount of deviation in the three-dimensional space (the difference between the reference coordinates and the current coordinates, i.e., a relative value).

[0225] In the three-dimensional data information, synchronization information (space_sync_information) may be stored for each of a plurality of pieces of format information. In this case, the data format that serves as the synchronization reference may be placed at the beginning of a loop, and synchronization information (relative position information) from the beginning format may be stored in the second loop and thereafter.

[0226] Furthermore, the three-dimensional space information may include space_sync_information indicating synchronization information, as shown in Fig. 36. When the spatial position is shifted for each three-dimensional space, the synchronization information may be stored in a loop for each three-dimensional space.

[0227] FIG. 37 is a diagram illustrating an example of a functional configuration of a terminal.

[0228] The terminal 1120 includes a decoding unit 1121 and a synchronous presentation unit 1122 .

[0229] The decoding unit 1121 decodes the synchronization information based on the three-dimensional data information or the three-dimensional space information. The synchronization presentation unit 1122 aligns the three-dimensional data based on the synchronization information and presents it.

[0230] Next, spatial synchronization will be described with reference to a flowchart shown in FIG.

[0231] First, the system including the encoding device (server) and the decoding device (terminal) determines whether spatial synchronization is required between the point cloud data and the mesh data (S1011).

[0232] If the system determines that spatial synchronization is necessary (Yes in S1011), it executes step S1012, and if the system determines that spatial synchronization is not necessary (No in S1011), it executes step S1013.

[0233] In step S1012, the system determines whether spatial synchronization is achieved between the point cloud data and the mesh data (S1012).

[0234] If the system determines that spatial synchronization has been achieved (Yes in S1012), it executes step S1014, and if the system determines that spatial synchronization has not been achieved (No in S1012), it executes step S1015.

[0235] In step S1013, the system presents the point cloud data and mesh data without spatial synchronization (S1013).

[0236] In step S1014, the system presents the point cloud data and mesh data in spatial synchronization based on the synchronization information (S1014).

[0237] In step S1015, the system presents the point cloud data and mesh data as they are (S1015).

[0238] The processes of steps S1011 to S1012 may be performed by the encoding device or the decoding device, and the processes of steps S1013 to S1015 may be performed by the decoding device.

[0239] Although the above flowchart describes the spatial synchronization process between point cloud data and mesh data, it may also be performed for 3D model data. The spatial synchronization process may be performed between different types of 3D data, and the type of 3D data that is the subject of the spatial synchronization process is not limited.

[0240] Whether spatial synchronization is necessary may be switched depending on the application or use. For example, when three-dimensional data is used for measurement purposes, accurate positioning is required, so it may be determined that spatial synchronization is necessary.

[0241] Also, when displaying and presenting to the user, accurate alignment may not be necessary, and it may be determined that spatial synchronization is not necessary.

[0242] Note that a level of alignment may be defined, and in this case, the system (encoding device or decoding device) may change the accuracy of synchronization based on the level of alignment.

[0243] (Others) In the above embodiment, an example of synchronizing multiple pieces of three-dimensional data has been described using spatial synchronization as an example, but other than that, time synchronization may be performed to synchronize the presentation time, decoding time, or acquisition time. At least one of spatial synchronization and time synchronization may be performed.

[0244] Furthermore, in order to seamlessly switch the color representation presented when switching between three-dimensional data, parameters such as the color matrix, the number of color bits, and HDR may be set to be the same. In this way, attribute information may be synchronized between multiple three-dimensional data.

[0245] In the above embodiment, switching between three-dimensional data has been described using examples of different types of three-dimensional data, such as point cloud data and mesh data. However, the multiple three-dimensional data to be switched are not limited to different types of three-dimensional data. The multiple three-dimensional data may be, for example, three-dimensional data with different resolutions, multiple point cloud data with different numbers of points, or multiple mesh data with different numbers of points or faces. Furthermore, the multiple three-dimensional data to be switched may be three or more three-dimensional data.

[0246] The multiple pieces of 3D data to be switched may be multiple pieces of point cloud data acquired at different times. For example, the multiple pieces of 3D data to be switched may include point cloud data before construction, point cloud data after construction, point cloud data 10 years from now, and modeled mesh data at a construction site.

[0247] In addition to three-dimensional data, three-dimensional model data such as NeRF may also be used. The three-dimensional model data is a model for presenting three-dimensional data and may or may not be encoded. Multiple three-dimensional model data for the same space may be switched, or multiple three-dimensional model data may be switched. In this case, the spatial synchronization method described above can be used. Furthermore, by adding a three-dimensional model data identifier to a data format that handles point cloud data and mesh data in an integrated manner, it becomes possible to handle the three-dimensional data and the three-dimensional model data in an integrated manner. Furthermore, by making the spatial information (origin, bounding box, division method, etc.) of the three-dimensional data and the three-dimensional model data the same, it is possible to share area information of the three-dimensional space, facilitating partial decoding.

[0248] In addition, the three-dimensional data to be processed (three-dimensional data to be presented or transmitted) may be switched in the following order: mesh data lower than the first resolution, mesh data higher than the first resolution, point cloud data lower than the second resolution, and point cloud data higher than the second resolution.

[0249] Regarding the timing of data transmission or encoding, the server may store data in a pre-encoded state of multiple 3D models and extract the requested 3D data based on a request from the terminal. The server may also encode the requested 3D data at the timing of the request from the terminal. The terminal may request the 3D data to be processed in advance and download the requested 3D data from the server. This reduces the time it takes for the terminal to present the data.

[0250] Furthermore, for each of the plurality of divided 3D data, it may be possible to select whether to present point cloud data, mesh data, or 3D model data, or to present two or more of these 3D data in a superimposed manner. For example, the terminal may present divided data corresponding to the avatar's torso as a mesh and divided data corresponding to the face as a point cloud, thereby enabling more accurate presentation.

[0251] In the above-described embodiment, point cloud data, mesh data, and three-dimensional model data are exemplified as three-dimensional data representing a three-dimensional object, but the present invention is not limited to these. For example, a three-dimensional object may be represented by a plurality of sets, each of which includes line-of-sight information indicating a line of sight and a two-dimensional image of the three-dimensional object as viewed from the line of sight. In other words, data including the plurality of sets may be treated as a type of three-dimensional data. Furthermore, the three-dimensional data may be data in another format, such as Gaussian splatting data.

[0252] Fig. 39 is a diagram showing an example of the configuration of a decoding device, and Fig. 40 is a flowchart showing an example of a decoding method by the decoding device.

[0253] The decoding device 1130 includes a circuit 1131 and a memory 1132 connected to the circuit 1131 .

[0254] The circuit 1131 performs the following operations.

[0255] The circuit 1131 acquires encoded data including first data representing a three-dimensional object, encoding format information (format) indicating one of encoding formats including second data representing the three-dimensional object, and identification information indicating a three-dimensional space including the three-dimensional object (S1021). Next, the circuit 1131 decodes the first data and the second data corresponding to the three-dimensional space based on the encoded data (S1022). Next, the circuit 1131 renders the first data to generate first presentation data for presentation (S1023). Next, the circuit 1131 renders the second data to generate second presentation data for presentation (S1024). Next, the circuit 1131 switches from the generated second presentation data to the first presentation data and presents it (S1025). Note that the first presentation data and the second presentation data are, for example, two-dimensional data or three-dimensional data generated by the rendering reconstruction unit 1034.

[0256] According to this, the first presentation data and the second presentation data are generated based on the first data and the second data corresponding to the three-dimensional space, and the first presentation data is switched to be presented, so that the two pieces of data representing the three-dimensional object can be presented without causing a spatial misalignment when they are switched, thereby making it possible to appropriately present the first presentation data and the second presentation data.

[0257] For example, the first data is point cloud data representing the three-dimensional object.

[0258] Therefore, the second presentation data is switched to the first presentation data based on the point cloud data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0259] For example, the second data is mesh data representing the three-dimensional object.

[0260] Therefore, the second presentation data based on mesh data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0261] For example, the second data is three-dimensional model data representing the three-dimensional object, and the three-dimensional model data represents a machine learning model obtained by machine learning a plurality of sets of gazes and two-dimensional images.

[0262] Therefore, the second presentation data based on the three-dimensional model data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0263] For example, the second data is a two-dimensional image of the three-dimensional object when viewed from a predetermined line of sight.

[0264] Therefore, the second presentation data based on a two-dimensional image is switched to the first presentation data, so that the two pieces of data representing a three-dimensional object can be switched and presented without causing any spatial misalignment.

[0265] For example, the circuit may further receive a request to switch presentation data from a user, and in the presentation, the circuit may switch from the second presentation data to the first presentation data in response to the request to switch presentation data.

[0266] Therefore, the switching can be performed at a timing designated by the user.

[0267] For example, the circuit may further receive an operation from a user to change the presentation mode, and the circuit may change the presentation mode in response to the operation, and switch from the second presentation data to the first presentation data in response to the change.

[0268] Therefore, switching can be performed at a timing according to the user's operation.

[0269] For example, the circuit may acquire the encoded data from an encoding device via a communication network during the acquiring step, and may switch from the second presentation data to the first presentation data during the presenting step in accordance with a bandwidth of the communication network.

[0270] Therefore, switching can be performed according to the bandwidth of the communication network; for example, when the bandwidth of the communication network changes from less than a specified bandwidth to greater than or equal to the specified bandwidth, the second presentation data can be switched to the first presentation data and presented.

[0271] For example, the circuit switches from the second presentation data to the first presentation data in accordance with the available capabilities of the circuit during the presentation.

[0272] Therefore, switching can be performed according to the capacity of the available circuit; for example, when the capacity of the available circuit changes from less than a predetermined capacity to greater than or equal to the predetermined capacity, the second presentation data can be switched to the first presentation data and presented.

[0273] For example, the encoded data includes synchronization information for synchronizing a coordinate system of the first data and a coordinate system of the second data, and the circuit, during presentation, presents the first presentation data and the second presentation data based on the synchronization information.

[0274] Therefore, the first presentation data and the second presentation data can be switched to the first presentation data after the coordinate systems of the first presentation data and the second presentation data are aligned, so that the two pieces of data representing the three-dimensional object can be switched and presented with less spatial misalignment.

[0275] For example, the circuit further determines whether to synchronize a coordinate system of the first data with a coordinate system of the second data. If the circuit determines that the coordinate system of the first data is synchronized with the coordinate system of the second data, the circuit presents the first presentation data and the second presentation data based on the synchronization information during the presentation.

[0276] This allows synchronization processing to be performed when necessary and can be skipped when not necessary, potentially reducing the processing load.

[0277] For example, the first data and the second data each have a common structure.

[0278] This reduces the amount of coded data, thereby reducing communication capacity.

[0279] For example, the encoded data includes spatial information for identifying the three-dimensional space in which the three-dimensional object is located. The circuit further acquires a target area indicating a portion of the three-dimensional space. The circuit identifies first duplicate data that is a portion of the first data and overlaps with the target area based on the spatial information. In the decoding process, the circuit decodes the identified first duplicate data.

[0280] Therefore, for example, by acquiring only the first duplicate data, the volume of data to be acquired can be reduced, thereby reducing communication capacity. Also, for example, it is possible to decrypt only the first duplicate data, thereby reducing processing load.

[0281] Alternatively, the circuit 1131 may operate according to the decoding method shown in the flowchart of Fig. 41. Fig. 41 is a flowchart showing another example of the decoding method performed by the decoding device.

[0282] The circuit 1131 decodes encoding information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data (S1031). The circuit 1131 decodes second data in the second encoding method indicated by the encoding information (S1032). The second data is used to generate second presentation data for presentation.

[0283] This allows the second data of the second encoding method indicated by the encoding method information obtained by decoding to be decoded, thereby obtaining second data for generating appropriate second presentation data for presentation.

[0284] Fig. 42 is a diagram showing an example of the configuration of an encoding device, and Fig. 43 is a flowchart showing an example of an encoding method performed by the encoding device.

[0285] The encoding device 1140 includes a circuit 1141 and a memory 1142 connected to the circuit 1141 .

[0286] The circuit 1141 performs the following operations.

[0287] The circuit 1141 generates encoding information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data (S1041). The circuit 1141 generates second data in the second encoding method indicated by the encoding information (S1042). The circuit 1141 generates a bitstream including the encoding information and the second data (S1043). The second data is used to generate second presentation data for presentation.

[0288] This allows a bitstream including encoding method information and second data to be generated, and a decoding device that acquires the bitstream can obtain the second data for generating second presentation data for appropriate presentation.

[0289] Although the three-dimensional data encoding device and the three-dimensional data decoding device according to the embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments.

[0290] Furthermore, each processing unit included in the encoding device, decoding device, server, terminal, etc. according to the above-described embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0291] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.

[0292] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0293] The present disclosure may also be realized as a decoding method or the like executed by a decoding device or the like.

[0294] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0295] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0296] While the encoding device and decoding device according to one or more aspects have been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0297] The present disclosure is applicable to a decoding device and a decoding method.

[0298] 1001 Three-dimensional data encoding system 1002 Three-dimensional data decoding system 1003 Sensor terminal 1004 External connection unit 1011 Three-dimensional data generation system 1012 Presentation unit 1013 Encoding unit 1014 Multiplexing unit 1015 Input / output unit 1016 Control unit 1017 Sensor information acquisition unit 1018 Three-dimensional data generation unit 1021 Sensor information acquisition unit 1022 Input / output unit 1023 Demultiplexing unit 1024 Decoding unit 1025 Presentation unit 1026 User interface 1027 Control unit 1031 Three-dimensional model learning unit 1032 Three-dimensional model encoding unit 1033 Three-dimensional model decoding unit 1034 Rendering reconstruction unit 1041 Data division unit 1042 Encoding unit 1051 Decoding unit 1052 Data combination unit 1061 Terminal presentation screen 1061a Point cloud button 1061b Mesh button 1062 Terminal presentation screen 1063 Terminal presentation screen 1070 Server 1071 Data generation unit 1072 Point cloud generation unit 1073 Mesh generation unit 1074 Model generation unit 1075 Synchronization unit 1076 Point cloud encoding unit 1077 Mesh encoding unit 1078 Model encoding unit 1079 Multiplexing unit 1080 Data extraction unit 1090 Terminal 1101 Point cloud sensor 1102 Camera 1110 Data generation unit 1111 Point cloud generation unit 1112 Mesh generation unit 1113 Model generation unit 1120 Terminal 1121 Decoding unit 1122 Synchronization presentation unit 1130 Decoding device 1131 Circuit 1132 Memory

Claims

1. A decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in an operation, obtains encoded data including first data representing a three-dimensional object and second data representing the three-dimensional object, encoding method information indicating one of encoding methods including the first data representing the three-dimensional object, and identification information indicating a three-dimensional space including the three-dimensional object; decodes the first data and the second data corresponding to the three-dimensional space based on the encoded data; renders the first data to generate first presentation data for presentation; renders the second data to generate second presentation data for presentation; and switches from the generated second presentation data to the first presentation data for presentation.

2. The decoding device according to claim 1, wherein the first data is point cloud data representing the three-dimensional object.

3. The decoding device according to claim 1, wherein the second data is mesh data representing the three-dimensional object.

4. The decoding device of claim 1, wherein the second data is three-dimensional model data representing the three-dimensional object, and the three-dimensional model data indicates a machine learning model obtained by machine learning multiple sets of gazes and two-dimensional images.

5. The decoding device according to claim 1, wherein the second data is a two-dimensional image of the three-dimensional object when viewed from a predetermined line of sight.

6. A decoding device as described in any one of claims 1 to 5, wherein the circuit further receives a presentation data switching request from a user, and in the presentation, switches from the second presentation data to the first presentation data in response to the switching request.

7. A decoding device as described in any one of claims 1 to 5, wherein the circuit further receives an operation from a user to change the presentation mode, changes the presentation mode in response to the operation, and switches from the second presentation data to the first presentation data in response to the change.

8. A decoding device as described in any one of claims 1 to 5, wherein the circuit, in the acquiring step, acquires the encoded data from the encoding device via a communication network, and, in the presenting step, switches from the second presentation data to the first presentation data and presents it depending on the bandwidth of the communication network.

9. A decoding device according to any one of claims 1 to 5, wherein the circuit switches from presenting the second presentation data to presenting the first presentation data depending on the available capabilities of the circuit.

10. A decoding device as described in any one of claims 1 to 5, wherein the encoded data includes synchronization information for synchronizing a coordinate system of the first data and a coordinate system of the second data, and the circuit, during the presentation, presents the first presentation data and the second presentation data based on the synchronization information.

11. The decoding device of claim 10, wherein the circuit further determines whether or not to synchronize the coordinate system of the first data with the coordinate system of the second data, and if it determines that the coordinate system of the first data with the coordinate system of the second data is synchronized, the circuit presents the first presentation data and the second presentation data in the presentation based on the synchronization information.

12. The decoding device according to any one of claims 1 to 5, wherein the first data and the second data each have a common structure.

13. A decoding device as described in any one of claims 1 to 5, wherein the encoded data includes spatial information for identifying the three-dimensional space in which the three-dimensional object is contained, and the circuit further obtains a target area indicating a portion of the three-dimensional space, identifies first duplicate data that is a portion of the first data and overlaps with the target area based on the spatial information, and decodes the identified first duplicate data during the decoding.

14. A decoding method comprising: acquiring encoded data including first data representing a three-dimensional object, encoding method information indicating one of encoding methods including second data representing the three-dimensional object, and identification information indicating a three-dimensional space including the three-dimensional object; decoding the first data and the second data based on the encoded data; rendering the first data to generate first presentation data for presentation; rendering the second data to generate second presentation data for presentation; and switching from the generated second presentation data to the first presentation data for presentation.

15. A decoding device for decoding first data representing a three-dimensional object, comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in an operation, decodes encoding method information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data, and decodes second data of the second encoding method indicated by the encoding method information, the second data being used to generate second presentation data for presentation.

16. A coding device for encoding first data representing a three-dimensional object, comprising: a circuit; and a memory connected to the circuit, the circuit, in operation, generating coding scheme information representing the three-dimensional object and indicating a second coding scheme different from the first coding scheme of the first data, generating second data of the second coding scheme indicated by the coding scheme information, generating a bitstream including the coding scheme information and the second data, the second data being used to generate second presentation data for presentation.