Encoding device, decoding device, encoding method, and decoding method

Abstract

This encoding device acquires a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network (S1601), and generates a bit stream including the acquired network and metadata including the acquired latent code (S1602). The network includes metadata identification information indicating metadata including a latent code corresponding to the network. Accordingly, since the network includes the metadata identification information indicating the metadata including the latent code corresponding to the network, a decoding device that acquires the bit stream can identify the metadata including the latent code corresponding to the network on the basis of the metadata identification information. Thus, the decoding device can output appropriate information on the basis of the network and the latent code.

Description

Encoding device, decoding device, encoding method, and decoding method

[0001] The present disclosure relates to an encoding device, a decoding device, an encoding method, and a decoding method.

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a representation called a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. In a point cloud, the position and color of the points are stored. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just as with two-dimensional moving images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] Furthermore, compression of point clouds is partially supported by a public library (Point Cloud Library) that performs processing related to point clouds.

[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1).

[0006] International Publication No. 2014 / 020663

[0007] ISO / IEC 15938-17:2022 (Information technology - Multimedia content description interface - Part 17: Compression of neural networks for multimedia content description and analysis (https / / www.iso.org / standard / 78480.html))

[0008] The present disclosure aims to provide an encoding device and the like that can reduce the data size of a bitstream generated by encoding multiple three-dimensional data generation models.

[0009] An encoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network, and generates a bitstream including the network and metadata including the latent code, and the network includes metadata identification information indicating the metadata including the latent code corresponding to the network.

[0010] A decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a bitstream including a network constituting a three-dimensional data generation model and metadata including a latent code set for a time corresponding to the network, the network including metadata identification information indicating the metadata including the latent code corresponding to the network, and decodes the network based on the bitstream.

[0011] These comprehensive or specific aspects may be realized as a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0012] The encoding device and the like of the present disclosure generate a bitstream that enables a decoding device to output information appropriately.

[0013] FIG. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to the first embodiment. FIG. 2 is a diagram showing the configuration of point cloud data according to the first embodiment. FIG. 3 is a diagram showing an example of the configuration of a data file in which information about point cloud data is described according to the first embodiment. FIG. 4 is a diagram showing the configuration of three-dimensional mesh data according to the first embodiment. FIG. 5 is a diagram showing an example of the configuration of a data file in which information about three-dimensional mesh data is described according to the first embodiment. FIG. 6 is a diagram for explaining a three-dimensional model according to the first embodiment. FIG. 7 is a diagram showing types of three-dimensional data according to the first embodiment. FIG. 8 is a diagram for explaining the encoding process of three-dimensional data according to the first embodiment. FIG. 9 is a diagram for explaining the decoding process of three-dimensional data according to the first embodiment. FIG. 10 is a diagram showing a two-dimensional schematic diagram of tiles and slices of three-dimensional data according to the first embodiment. FIG. 11 is a block diagram showing an example of the functional configuration of a server and a terminal according to the first embodiment. FIG. 12 is a block diagram showing another example of the data generation unit of the server according to the first embodiment. FIG. 13 is a diagram for explaining the relationship between three-dimensional space and encoded data according to the first embodiment. Fig. 14 is a diagram showing an example of the syntax of a coding scheme unit in Embodiment 1. Fig. 15 is a diagram showing an example of the syntax of a coded point group in Embodiment 1. Fig. 16 is a diagram showing an example of the syntax of a coded mesh in Embodiment 1. Fig. 17 is a diagram showing an example of the syntax of a coded 3D model in Embodiment 1. Fig. 18 is a diagram showing an example of the syntax of 3D data information in Embodiment 1. Fig. 19 is a diagram for explaining the data structure of a coded point group in Embodiment 1. Fig. 20 is a diagram for explaining the data structure of a coded mesh in Embodiment 1. Fig. 21 is a diagram for explaining the data structure of a coded 3D model in Embodiment 1. Fig. 22 is a diagram showing an example of a plurality of 3D spaces in two dimensions in Embodiment 1. Fig. 23 is a diagram showing an example of a bounding box in Embodiment 1.FIG. 24 is a diagram showing an example of the syntax of 3D space information in Embodiment 1. FIG. 25 is a flowchart showing an example of partial decoding in Embodiment 1. FIG. 26 is a diagram showing an example of a 3D space region to be partially decoded in Embodiment 1. FIG. 27 is a diagram showing an example of the data structure of a partially decoded encoded point group in Embodiment 1. FIG. 28 is a diagram showing an example of the data structure of a partially decoded encoded mesh in Embodiment 1. FIG. 29 is a diagram showing an example of the data structure of a partially decoded encoded 3D model in Embodiment 1. FIG. 30 is a diagram showing an example of the configuration of a decoding device in Embodiment 1. FIG. 31 is a flowchart showing an example of a decoding method used by the decoding device in Embodiment 1. FIG. 32 is a flowchart showing another example of a decoding method used by the decoding device. FIG. 33 is a diagram showing an example of the configuration of an encoding device. FIG. 34 is a flowchart showing an example of an encoding method used by the encoding device. FIG. 35 is a diagram for explaining processing during training of a 3D generation model in Embodiment 2. FIG. 36 is a diagram for explaining processing for generating a still image of a subject viewed from an arbitrary viewpoint using a 3D generation model in Embodiment 2. FIG. 37 is a diagram for explaining a moving image generation method using a 3D data generation model of Example 1 in Embodiment 2. FIG. 38 is a diagram for explaining a first example of the configuration of an encoding device of Example 1 in Embodiment 2. FIG. 39 is a diagram for explaining a first example of the configuration of a decoding device of Example 1 in Embodiment 2. FIG. 40 is a diagram for explaining a second example of the configuration of an encoding device of Example 1 in Embodiment 2. FIG. 41 is a diagram for explaining a second example of the configuration of a decoding device of Example 1 in Embodiment 2. FIG. 42 is a diagram for explaining a moving image generation method using an extended 3D data generation model of Example 2 in Embodiment 2. FIG. 43 is a diagram for explaining a first example of the configuration of an encoding device of Example 2 in Embodiment 2. FIG. 44 is a diagram for explaining a first example of the configuration of a decoding device of Example 2 in Embodiment 2. FIG. 45 is a diagram for explaining a second example of the configuration of an encoding device of Example 2 in Embodiment 2. FIG. 46 is a diagram for explaining a second example of the configuration of a decoding device of Example 2 in Embodiment 2.FIG. 47 is a diagram illustrating a moving image generation method using an extended 3D data generation model according to a modification of Embodiment 2. FIG. 48 is a diagram illustrating a moving image generation method using a 3D data generation model according to a modification of Embodiment 2. FIG. 49 is a diagram illustrating an example of the configuration of an encoding device according to Embodiment 2. FIG. 50 is a flowchart illustrating an example of an encoding method by the encoding device according to Embodiment 2. FIG. 51 is a diagram illustrating an example of the configuration of a decoding device according to Embodiment 2. FIG. 52 is a flowchart illustrating an example of a decoding method by the decoding device according to Embodiment 2. FIG. 53 is a diagram illustrating an example of the configuration of an encoding device. FIG. 54 is a diagram illustrating an example of the configuration of a decoding device. FIG. 55 is a diagram illustrating processing during training of a 3D data generation model according to Embodiment 3. FIG. 56 is a diagram illustrating processing for generating a still image of a subject viewed from an arbitrary viewpoint using a 3D data generation model according to Embodiment 3. FIG. 57 is a diagram illustrating a moving image generation method using an extended 3D data generation model of Example 1 according to Embodiment 3. FIG. 58 is a diagram illustrating a first example of the configuration of an encoding device of Example 1 according to Embodiment 3. FIG. 59 is a diagram showing a first example of the configuration of a decoding device according to Example 1 of Embodiment 3. FIG. 60 is a diagram showing a second example of the configuration of a coding device according to Example 1 of Embodiment 3. FIG. 61 is a diagram showing a second example of the configuration of a decoding device according to Example 1 of Embodiment 3. FIG. 62 is a block diagram showing an example of the configuration of a coding device that codes multiple networks according to Example 2 of Embodiment 3. FIG. 63 is a diagram showing an example of coded data of a trained first network according to Embodiment 3. FIG. 64 is a diagram showing an example of coded data of a trained second network according to Embodiment 3. FIG. 65 is a block diagram showing an example of the configuration of a decoding device that decodes multiple networks according to Embodiment 3. FIG. 66 is a diagram showing an example of the configuration of a coding device according to Embodiment 3. FIG. 67 is a flowchart showing a first example of a coding method performed by a coding device according to Embodiment 3. FIG. 68 is a flowchart showing a second example of a coding method performed by a coding device according to Embodiment 3.FIG. 69 is a diagram showing an example of the configuration of a decoding device according to Embodiment 3. FIG. 70 is a flowchart showing a first example of a decoding method performed by the decoding device according to Embodiment 3. FIG. 71 is a flowchart showing a second example of an encoding method performed by the decoding device according to Embodiment 3. FIG. 72 is a diagram showing an example of the syntax of sequence-based metadata according to Embodiment 4. FIG. 73 is a diagram showing an example of frame-based metadata syntax according to Embodiment 4. FIG. 74 is a diagram showing an example of the syntax of a data unit for a high-density network according to Embodiment 4. FIG. 75 is a diagram showing an example of the syntax of a data unit for a low-density network according to Embodiment 4. FIG. 76 is a diagram showing an example of the configuration of a data unit for a first network according to Embodiment 4. FIG. 77 is a diagram showing an example of the configuration of a data unit for a second network according to Embodiment 4. FIG. 78 is a diagram showing an example of the syntax of coded data of an NeRF three-dimensional model according to Embodiment 4. FIG. 79 is a diagram showing an example of an NeRF unit type according to Embodiment 4. FIG. 80A is a diagram showing another example of the syntax of coded data of an NeRF three-dimensional model according to Embodiment 4. FIG. 80B is a diagram showing an example of the syntax of a Latent Data Unit in Embodiment 4. FIG. 81 is a diagram showing an example of the data structure of coded data of an NeRF three-dimensional model in Embodiment 4. FIG. 82 is a diagram showing an example of the syntax of an SPS of an NeRF three-dimensional model in Embodiment 4. FIG. 83 is a diagram showing an example of the syntax of structural information of an NeRF three-dimensional model in Embodiment 4. FIG. 84 is a diagram showing an example of a component_type in Embodiment 4. FIG. 85 is a diagram showing an example of a component coding type in Embodiment 4. FIG. 86 is a diagram for explaining the reference relationship of coded data of an NeRF three-dimensional model in Embodiment 4. FIG. 87 is a diagram showing an example in which data of one frame is divided into three three-dimensional spaces in Embodiment 4.FIG. 88 is a diagram showing an example of IDs assigned to segmented data in the fourth embodiment. FIG. 89 is a diagram for explaining a first example of an encoding method in the fourth embodiment. FIG. 90 is a diagram for explaining a first example of an output of a decoding device in the fourth embodiment. FIG. 91 is a diagram for explaining a second example of an encoding method in the fourth embodiment. FIG. 92 is a diagram for explaining a second example of an output of a decoding device in the fourth embodiment. FIG. 93 is a diagram for explaining a third example of an encoding method in the fourth embodiment. FIG. 94 is a diagram for explaining a third example of an output of a decoding device in the fourth embodiment. FIG. 95 is a diagram for explaining a fourth example of an encoding method in the fourth embodiment. FIG. 96 is a diagram for explaining a fourth example of an output of a decoding device in the fourth embodiment. FIG. 97 is a diagram for explaining a fifth example of an encoding method in the fourth embodiment. FIG. 98 is a diagram for explaining a fifth example of an output of a decoding device in the fourth embodiment. FIG. 99 is a diagram for explaining data exchange between the decoding unit and the control unit in the fourth embodiment. FIG. 100 is a diagram for explaining conformance points in the fourth embodiment. FIG. 101 is a diagram showing an example of a bitstream including multiple networks in the fourth embodiment. FIG. 102 is a diagram showing an example of syntax of a layer structure of multiple networks in the fourth embodiment. FIG. 103 is a block diagram showing an example of the configuration of a modified encoding device in the fourth embodiment. FIG. 104 is a block diagram showing an example of the configuration of a modified decoding device in the fourth embodiment. FIG. 105 is a diagram showing an example of the configuration of an encoding device in the fourth embodiment. FIG. 106 is a flowchart showing a first example of an encoding method by the encoding device in the fourth embodiment. FIG. 107 is a diagram showing an example of the configuration of a decoding device in the fourth embodiment. FIG. 108 is a flowchart showing a first example of a decoding method by the decoding device in the fourth embodiment. FIG. 109 is a flowchart showing a second example of an encoding method by the encoding device in the fourth embodiment. FIG. 110 is a flowchart showing a second example of a decoding method by the decoding device in the fourth embodiment.FIG. 111 is a flowchart showing a third example of the encoding method used by the encoding device in Embodiment 4. FIG. 112 is a flowchart showing a third example of the decoding method used by the decoding device in Embodiment 4. FIG. 113 is a flowchart showing a fourth example of the encoding method used by the encoding device in Embodiment 4. FIG. 114 is a flowchart showing a fourth example of the decoding method used by the decoding device in Embodiment 4. FIG. 115 is a flowchart showing a fifth example of the encoding method used by the encoding device in Embodiment 4. FIG. 116 is a flowchart showing a fifth example of the decoding method used by the decoding device in Embodiment 4.

[0014] An encoding device according to a first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network, and generates a bitstream including the network and metadata including the latent code, and the network includes metadata identification information indicating the metadata including the latent code corresponding to the network.

[0015] According to this, since the network includes metadata identification information indicating the metadata including the latent code corresponding to the network, the decoding device that acquires the bitstream can identify the metadata including the latent code corresponding to the network based on the metadata identification information, and thus can output appropriate information based on the network and the latent code.

[0016] A coding device according to a second aspect of the present disclosure is a coding device according to the first aspect, wherein the bitstream further includes type information indicating the type of data contained in the bitstream, and data contained in the bitstream that is indicated by the type information to be metadata containing a latent code is metadata containing the latent code.

[0017] Therefore, the type information can indicate that the data included in the bitstream is metadata.

[0018] A coding device according to a third aspect of the present disclosure is the coding device according to the first or second aspect, wherein the metadata further includes dimensionality information indicating the number of dimensions of the latent code.

[0019] Therefore, the number of dimensions of the latent code can be appropriately switched for each metadata.

[0020] A coding device according to a fourth aspect of the present disclosure is a coding device according to the third aspect, wherein the bit stream includes a plurality of networks that include the network and correspond to a plurality of times, and a plurality of latent codes that include the latent code and are set for the plurality of times, and the dimension number information indicates the number of dimensions common to the plurality of latent codes.

[0021] Therefore, since the dimension number information is common to a plurality of latent codes, the amount of code for the dimension number information can be reduced.

[0022] A coding device according to a fifth aspect of the present disclosure is a coding device according to any one of the first to third aspects, wherein the bitstream includes a plurality of networks including the network, the plurality of networks including a first network corresponding to a first latent code and a second network corresponding to a second latent code, the metadata includes the first latent code and the second latent code, the first network includes first identification information for referencing the first latent code, and the second network includes second identification information for referencing the second latent code.

[0023] Therefore, a decoding device that acquires a bitstream can identify a first latent code corresponding to the first network and a second latent code corresponding to the second network, and can appropriately output information using the first network and the first latent code, and can also appropriately output information using the second network and the second latent code.

[0024] A coding device according to a sixth aspect of the present disclosure is the coding device according to the fifth aspect, wherein the first latent code and the second latent code correspond to a common time, and the second network has a higher resolution of sampling points used for learning than the first network.

[0025] A coding device according to a seventh aspect of the present disclosure is a coding device according to any one of the first to fourth aspects, wherein the metadata includes network identification information indicating the network to which the latent code corresponds.

[0026] Therefore, the latent code contained in the metadata can be associated with the network corresponding to that latent code, and the decoding device that acquires the bitstream can output appropriate information based on the network and the latent code.

[0027] A decoding device according to an eighth aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a bitstream including a network constituting a three-dimensional data generation model and metadata including a latent code set for a time corresponding to the network, the network including metadata identification information indicating the metadata including the latent code corresponding to the network, and decodes the network based on the bitstream.

[0028] According to this, since the network includes metadata identification information indicating the metadata including the latent code corresponding to the network, the decoding device that acquires the bitstream can identify the metadata including the latent code corresponding to the network based on the metadata identification information, and thus can output appropriate information based on the network and the latent code.

[0029] A decoding device according to a ninth aspect of the present disclosure is a decoding device according to the eighth aspect, wherein the bitstream further includes type information indicating the type of data contained in the bitstream, and data contained in the bitstream that is indicated by the type information to be metadata including a latent code is metadata including the latent code.

[0030] Therefore, the decoding device can identify that the data included in the bitstream is metadata based on the type information.

[0031] A decoding device according to a tenth aspect of the present disclosure is the decoding device according to the eighth or ninth aspect, wherein the metadata further includes dimension number information indicating the number of dimensions of the latent code.

[0032] Therefore, the decoding device can appropriately switch the number of dimensions of the latent code for each piece of metadata, and can output information appropriately.

[0033] A decoding device according to an eleventh aspect of the present disclosure is a decoding device according to the tenth aspect, wherein the bit stream includes a plurality of networks that include the network and correspond to a plurality of times, and a plurality of latent codes that include the latent code and are set for the plurality of times, and the dimension number information indicates the number of dimensions common to the plurality of latent codes.

[0034] Therefore, since the dimension number information is common to a plurality of latent codes, the decoding device can obtain a bitstream in which the amount of code for the dimension number information is reduced.

[0035] A decoding device according to a twelfth aspect of the present disclosure is a decoding device according to any one of the eighth to eleventh aspects, wherein the bitstream includes a plurality of networks including the network, the plurality of networks including a first network corresponding to a first latent code and a second network corresponding to a second latent code, the metadata includes the first latent code and the second latent code, the first network includes first identification information indicating the first latent code, and the second network includes second identification information indicating the second latent code.

[0036] Therefore, a decoding device that acquires a bitstream can identify a first latent code corresponding to the first network and a second latent code corresponding to the second network, and can appropriately output information using the first network and the first latent code, and can also appropriately output information using the second network and the second latent code.

[0037] A decoding device according to a thirteenth aspect of the present disclosure is a decoding device according to the twelfth aspect, wherein the first latent code and the second latent code correspond to a common time, and the second network has a higher resolution of sampling points used for learning than the first network.

[0038] A decoding device according to a fourteenth aspect of the present disclosure is a decoding device according to any one of the eighth to eleventh aspects, wherein the metadata includes network identification information indicating the network to which the latent code corresponds.

[0039] Therefore, the latent code contained in the metadata can be associated with the network corresponding to that latent code, and the decoding device that acquires the bitstream can output appropriate information based on the network and the latent code.

[0040] An encoding method according to a fifteenth aspect of the present disclosure is an encoding method executed by an encoding device, which acquires a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network, and generates a bitstream including the network and metadata including the latent code, wherein the network includes metadata identification information indicating the metadata including the latent code corresponding to the network.

[0041] According to this, since the network includes metadata identification information indicating the metadata including the latent code corresponding to the network, the decoding device that acquires the bitstream can identify the metadata including the latent code corresponding to the network based on the metadata identification information, and thus can output appropriate information based on the network and the latent code.

[0042] A decoding method according to a sixteenth aspect of the present disclosure is a decoding method executed by a decoding device, which obtains a bitstream including a network constituting a three-dimensional data generation model and metadata including a latent code set for a time corresponding to the network, the network including metadata identification information indicating metadata including the latent code corresponding to the network, and decodes the network based on the bitstream.

[0043] According to this, since the network includes metadata identification information indicating the metadata including the latent code corresponding to the network, the decoding device that acquires the bitstream can identify the metadata including the latent code corresponding to the network based on the metadata identification information, and thus can output appropriate information based on the network and the latent code.

[0044] These comprehensive or specific aspects may be realized as a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0045] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that represents a superordinate concept will be described as optional components.

[0046] (Embodiment 1) The configuration of a three-dimensional data encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 1001, a three-dimensional data decoding system 1002, a sensor terminal 1003, and an external connection unit 1004.

[0047] The three-dimensional data encoding system 1001 generates encoded data or multiplexed data by encoding three-dimensional data. Note that the three-dimensional data encoding system 1001 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 1001.

[0048] The three-dimensional data encoding system 1001 includes a three-dimensional data generation system 1011, a presentation unit 1012, an encoding unit 1013, a multiplexing unit 1014, an input / output unit 1015, and a control unit 1016. The three-dimensional data generation system 1011 includes a sensor information acquisition unit 1017 and a three-dimensional data generation unit 1018.

[0049] The sensor information acquisition unit 1017 acquires a sensor signal from the sensor terminal 1003 and outputs the sensor signal to the three-dimensional data generation unit 1018. The three-dimensional data generation unit 1018 generates three-dimensional data from the sensor signal and outputs the three-dimensional data to the encoding unit 1013.

[0050] The presentation unit 1012 presents the sensor signal or the three-dimensional data to the user. For example, the presentation unit 1012 displays information or an image based on the sensor signal or the three-dimensional data.

[0051] The encoding unit 1013 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 1014. The additional information includes, for example, a sensor signal.

[0052] The multiplexing unit 1014 generates multiplexed data by multiplexing the coded data input from the coding unit 1013, the control information, and the additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0053] The input / output unit 1015 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 1016 (or the application execution unit) controls each processing unit. That is, the control unit 1016 controls encoding, multiplexing, etc. The control unit 1016 may also control demultiplexing, decoding, or presentation.

[0054] The sensor signal may be input to the encoding unit 1013 or the multiplexing unit 1014. The input / output unit 1015 may output the three-dimensional data or the encoded data directly to the outside.

[0055] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 1001 is input to the three-dimensional data decoding system 1002 via the external connection unit 1004 .

[0056] The three-dimensional data decoding system 1002 generates three-dimensional data by decoding encoded data or multiplexed data. Note that the three-dimensional data decoding system 1002 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 1002.

[0057] The three-dimensional data decoding system 1002 includes a sensor information acquisition unit 1021 , an input / output unit 1022 , a demultiplexing unit 1023 , a decoding unit 1024 , a presentation unit 1025 , a user interface 1026 , and a control unit 1027 .

[0058] The sensor information acquisition unit 1021 acquires a sensor signal from the sensor terminal 1003 .

[0059] The input / output unit 1022 acquires a transmission signal, decodes multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 1023 .

[0060] The demultiplexing unit 1023 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoding unit 1024 .

[0061] The decoding unit 1024 reconstructs the point cloud data by decoding the encoded data.

[0062] The presentation unit 1025 presents the point cloud data to the user. For example, the presentation unit 1025 displays information or an image based on the point cloud data. The user interface 1026 acquires instructions based on user operations. The control unit 1027 (or the application execution unit) controls each processing unit. That is, the control unit 1027 controls demultiplexing, decoding, presentation, etc.

[0063] The input / output unit 1022 may acquire point cloud data or encoded data directly from the outside. The presentation unit 1025 may acquire additional information such as a sensor signal and present information based on the additional information. The presentation unit 1025 may also present information based on a user instruction acquired by the user interface 1026.

[0064] The sensor terminal 1003 generates a sensor signal, which is information obtained by a sensor. The sensor terminal 1003 is a terminal equipped with a sensor or a camera, and may be, for example, a mobile object such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

[0065] The sensor signals obtainable by the sensor terminal 1003 include, for example, (1) a signal indicating the distance between the sensor terminal 1003 and an object or the reflectance of the object obtained from a LIDAR, millimeter-wave radar, or infrared sensor, and (2) a signal indicating the distance between a camera and an object or the reflectance of the object obtained from multiple monocular camera images or stereo camera images. The sensor signals may also include the attitude, orientation, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor signals may also include temperature, air pressure, humidity, magnetism, etc.

[0066] The external connection unit 1004 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0067] Next, point cloud data will be described. Fig. 2 is a diagram showing the structure of point cloud data. Fig. 3 is a diagram showing an example of the structure of a data file in which information about point cloud data is written.

[0068] Point cloud data includes data on multiple points. Each point includes location information (three-dimensional coordinates) and attribute information for that location. A collection of multiple points is called a point cloud. For example, a point cloud can represent the three-dimensional shape of an object.

[0069] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.

[0070] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0071] The data file configuration example shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.

[0072] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0073] Next, the three-dimensional mesh data will be described. Fig. 4 shows the structure of the three-dimensional mesh data. Fig. 5 shows an example of the structure of a data file in which information about the three-dimensional mesh data is written.

[0074] Three-dimensional mesh data is a data format used in computer graphics (CG) and represents the three-dimensional shape of an object using a collection of multiple pieces of surface information. Each piece of surface information represents a polygon such as a triangle or a quadrangle. Three-dimensional mesh data is also called a polygon or polygon mesh.

[0075] The components are a set of three-dimensional point clouds, vertices that are multiple three-dimensional points of the three-dimensional point cloud, edges connecting two vertices at multiple three-dimensional points, and faces surrounded by multiple edges. A three-dimensional point cloud is a set of points that include position information in three-dimensional space and attribute information corresponding to the position information. Note that three-dimensional points may simply be referred to as points.

[0076] A vertex may have attribute information such as color information, reflectance, and normal vector for a three-dimensional point. The relationship between the vertices that make up an edge or a face may be indicated by information called connectivity. Note that a vertex may be expressed as a position. The front and back of a face may be expressed by the direction of the normal vector for the three-dimensional point. Also, a vertex may have attribute information for the face.

[0077] An example of a mesh data file format is an object file. In a mesh data file such as that shown in Fig. 5, the position information G(1) to G(N) of the N vertices that make up the mesh and the attribute information A(1) to A(N) of the vertices are indicated as vertex information. In a mesh data file, the vertex information does not need to include attribute information.

[0078] Furthermore, the attribute information does not have to correspond one-to-one to the vertices. The mesh data file in Fig. 5 shows an example in which the three-dimensional mesh data has M pieces of attribute information A2.

[0079] Face information is represented by a combination of vertex indices, where n[1,3,4] indicates a triangular face made up of three vertices: vertex n=1, vertex n=3, and vertex n=4.

[0080] Furthermore, m[2,4,6] indicates that the attribute information of m=2, the attribute information of m=4, and the attribute information of m=6 in the attribute information A2 correspond to three vertices, respectively. Note that although an example in which a face is formed by three vertices has been shown here, the number of vertices of a face is not limited to three, as long as it is an integer equal to or greater than three. For example, if the face is a quadrangle, the number of vertices is four, and if the face is a polygon, the number of vertices is equal to the number of vertices that form the polygon.

[0081] The attribute information A2 may be represented in a file separate from the mesh data file, or may include pointer information thereto. For example, the attribute information may be stored in a two-dimensional attribute map file, and the attribute map file name and two-dimensional coordinates in the attribute map may be represented by the attribute information A2 of the mesh data file. In this way, the attribute information A2 may be included in the mesh data file or represented in a file separate from the mesh data file. Either method can be used to specify attribute information for a three-dimensional point.

[0082] Next, the three-dimensional model will be described with reference to Fig. 6.

[0083] A three-dimensional model is a model generated based on two-dimensional or three-dimensional data.

[0084] The three-dimensional model learning unit 1031 learns, for example, two-dimensional data (two-dimensional images) or three-dimensional data (point clouds or meshes) to generate a three-dimensional model, which is a network model in which three-dimensional shapes and attribute information corresponding to the three-dimensional shapes are learned using a neural network or the like.

[0085] The three-dimensional model learning unit 1031 may generate a three-dimensional model by learning using Neural Radiance Fields (NeRF) based on two-dimensional images. The three-dimensional model learning unit 1031 may generate a three-dimensional model after converting the two-dimensional images into three-dimensional data by performing photogrammetry using the two-dimensional images. The three-dimensional model may be generated using three-dimensional data acquired by a sensor (distance sensor).

[0086] The three-dimensional model data is the elements that make up the three-dimensional model, and includes information indicating the structure of the network model, feature quantities, etc. The three-dimensional model data includes, for example, information about the components of the neural network. The information about the components includes, for example, multiple layers such as an input layer, an intermediate layer, and an output layer, nodes in each layer, weight coefficients for the nodes, and transformation functions for the nodes.

[0087] The three-dimensional model encoding unit 1032 may encode the three-dimensional model data and transmit the encoded three-dimensional model data.

[0088] The three-dimensional model decoding unit 1033 receives the transmitted encoded three-dimensional model data and decodes the three-dimensional model based on the encoded three-dimensional model data.

[0089] The rendering reconstruction unit 1034 reconstructs (generates) two-dimensional data (two-dimensional image) or three-dimensional data (point cloud or mesh) based on the decoded three-dimensional model. For example, when using a three-dimensional model modeled using NeRF, the rendering reconstruction unit 1034 acquires viewpoint position or line-of-sight vector information, generates rendered two-dimensional data (two-dimensional image) based on the three-dimensional model and the viewpoint position or line-of-sight vector, and outputs the two-dimensional data. The generated two-dimensional data represents a three-dimensional object as seen from the viewpoint position, or a two-dimensional image of the three-dimensional object as seen from the line of sight indicated by the line-of-sight vector. The three-dimensional object is the three-dimensional object of the subject that was the source of the two-dimensional data or three-dimensional data input to the three-dimensional model learning unit 1031.

[0090] Next, the types of three-dimensional data will be explained. Fig. 7 shows the types of three-dimensional data. As shown in Fig. 7, three-dimensional data includes static objects and dynamic objects.

[0091] A static object is three-dimensional data at any time (a certain time). A dynamic object is three-dimensional data that changes over time. Hereinafter, point cloud data at a certain time will be referred to as a PCC frame or frame. Mesh data at a certain time will be referred to as a mesh frame or frame.

[0092] The object may be three-dimensional data with a certain area restriction, such as ordinary video data, or may be three-dimensional data with no area restriction, such as map information.

[0093] There may also be sparse point cloud data (sparse mesh data) and dense point cloud data (dense mesh data) with points of various densities.

[0094] Each processing unit will be described in detail below. Sensor information is acquired by various methods, such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The three-dimensional data generation unit 1018 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 1017. The three-dimensional data generation unit 1018 generates position information (geometry information) as the point cloud data, and adds attribute information for the position information to the position information.

[0095] The three-dimensional data generation unit 1018 may process the point cloud data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 1018 may reduce the amount of data by deleting point clouds with overlapping positions. Furthermore, the three-dimensional data generation unit 1018 may convert the position information (such as by shifting, rotating, or normalizing) or process the point cloud data to generate mesh data. Furthermore, the three-dimensional data generation unit 1018 may render the attribute information.

[0096] In FIG. 1, the three-dimensional data generation system 1011 is included in the three-dimensional data encoding system 1001, but it may be provided independently outside the three-dimensional data encoding system 1001.

[0097] The encoding unit 1013 generates encoded data by encoding the three-dimensional data based on a predetermined encoding method. The encoding methods include G-PCC (an encoding method using position information), V-PCC (an encoding method using a video codec), Draco (a mesh encoding method), and V-DMC (a mesh encoding method). The encoding method is not limited to these methods, and may be, for example, a method of encoding a dynamic mesh, or another method that combines these methods.

[0098] The decoding unit 1024 decodes the coded data based on a predetermined coding method to decode the three-dimensional data.

[0099] The multiplexing unit 1014 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the encoded data of the three-dimensional data, the multiplexing unit 1014 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 1014 may also multiplex attribute information related to sensor information or point cloud data.

[0100] Multiplexing methods or file formats include ISOBMFF, and ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0101] The demultiplexing unit 1023 extracts the coded data of the three-dimensional data, other media, time information, and the like from the multiplexed data.

[0102] The input / output unit 1015 transmits the multiplexed data using a method suited to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 1015 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0103] The communication protocol used may be http, ftp, TCP, UDP, etc. A pull-type communication method or a push-type communication method may be used.

[0104] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter waves, etc. are used.

[0105] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0106] Next, a process of dividing three-dimensional data into one or more three-dimensional data will be described. Fig. 8 is a diagram for explaining a process of encoding three-dimensional data. Fig. 9 is a diagram for explaining a process of decoding three-dimensional data.

[0107] 8 , the data dividing unit 1041 divides the three-dimensional data into one or more three-dimensional spaces and generates one or more divided three-dimensional data (i.e., one or more divided three-dimensional data). The encoding unit 1042 may encode one or more divided three-dimensional data to generate encoded data. The data dividing unit 1041 and the encoding unit 1042 may be included in a single encoding device as components of the single encoding device, or may be included in separate devices.

[0108] Each of the one or more three-dimensional spaces may be referred to as a tile or a space. The three-dimensional space may be, for example, a bounding box. Furthermore, the three-dimensional data included in each of the divided three-dimensional spaces may be referred to as a slice. A slice is divided three-dimensional data and includes any of a point cloud, a mesh, and a three-dimensional model having position information (geometry) or attribute information (attribute). Each of the multiple slices is coded by the coding unit 1042 for each component element and output as coded data. The coded data includes the coded multiple slices.

[0109] As shown in FIG. 9 , in the decoding process, the decoding unit 1051 decodes one or more pieces of divided three-dimensional data (one or more slices) based on the encoded data. The data combining unit 1052 combines the one or more pieces of divided three-dimensional data to restore (generate) three-dimensional data. The decoding unit 1051 and the data combining unit 1052 may be included in a single decoding device as components of the single decoding device, or may be included in separate devices. The one or more pieces of divided three-dimensional data decoded by the decoding unit 1051 do not need to be combined. The decoding unit 1051 may decode some of the one or more pieces of divided three-dimensional data based on part of the encoded data and output the decoded some of the divided three-dimensional data. In this case, the decoding device does not need to have the data combining unit 1052.

[0110] FIG. 10 is a two-dimensional schematic diagram showing tiles and slices of three-dimensional data.

[0111] When encoding multiple slices, the encoding device may encode them using a dependency relationship between the multiple slices, or may encode them without using the dependency relationship. When encoding without using the dependency relationship, the encoding device can encode each slice independently, and can reduce processing time by encoding multiple slices using parallel processing. Furthermore, when multiple slices are encoded without using the dependency relationship, the decoding device can decode each slice independently, and can reduce processing time by decoding multiple slices using parallel processing. Furthermore, the decoding device can reduce the amount of processing by performing partial decoding, which decodes some of the multiple slices.

[0112] When encoding using a dependency relationship, the encoding device signals an identifier indicating the dependency relationship and encodes the data in order starting with the dependent data. When multiple slices are encoded using a dependency relationship, the decoding device decodes the data in order starting with the dependent data based on the identifier.

[0113] The division of the three-dimensional data may be performed using any number of divisions and any division method. The division of the three-dimensional data may involve determining the shape of an object and dividing each object into multiple three-dimensional points. The division of the three-dimensional data may also be performed based on the number of three-dimensional points included in a slice. In other words, an upper limit on the number of three-dimensional points included in one slice may be set. The three-dimensional data may also be divided based on whether or not it is included in the three-dimensional space (tile information) using map information or location information. Multiple tile shapes may overlap.

[0114] By dividing the three-dimensional data into a plurality of divided three-dimensional data in this way, adaptive encoding according to the content or object and parallel processing in decoding become possible.

[0115] Next, a method for selecting three-dimensional data to be presented or transmitted from among a plurality of three-dimensional data will be described.

[0116] The server stores multiple pieces of three-dimensional data for the same space. The server stores, for example, point cloud data and mesh data for the same space. The server is an example of an encoding device. The terminal switches the three-dimensional data acquired from the server based on the purpose of the terminal and presents the switched three-dimensional data. The terminal may be, for example, a terminal that analyzes three-dimensional data. In this case, the terminal may switch the three-dimensional data to be presented based on the purpose of analysis or presentation, or user operation. The terminal is an example of a decoding device.

[0117] The switching of three-dimensional data may involve switching between presenting a point cloud or a mesh as the three-dimensional data. Furthermore, the switching of three-dimensional data may involve switching between transmitting a point cloud or a mesh as the three-dimensional data. For example, the terminal may transmit a user's selection result to a server, receive (download) three-dimensional data based on the selection result from the server, and present the received three-dimensional data. The three-dimensional data (point cloud or mesh) may or may not be encoded by the server. If the three-dimensional data is encoded, the terminal may receive the encoded three-dimensional data from the server, decode the three-dimensional data based on the received encoded three-dimensional data, and present the decoded three-dimensional data.

[0118] Next, a description will be given of the configuration of the server 1070 and the terminal 1090. Fig. 11 is a block diagram showing an example of the functional configuration of the server and the terminal.

[0119] The server 1070 includes a data generation unit 1071 , a synchronization unit 1075 , a point cloud encoding unit 1076 , a mesh encoding unit 1077 , a model encoding unit 1078 , a multiplexing unit 1079 , and a data extraction unit 1080 .

[0120] The data generation unit 1071 generates three-dimensional data based on at least one of two-dimensional data and three-dimensional data. The generated three-dimensional data includes at least two of point cloud data, mesh data, and three-dimensional model data. The data generation unit 1071 includes a point cloud generation unit 1072, a mesh generation unit 1073, and a model generation unit 1074. The data generation unit 1071 may include at least two of the point cloud generation unit 1072, the mesh generation unit 1073, and the model generation unit 1074. The point cloud generation unit 1072 generates point cloud data based on at least one of the two-dimensional data and the three-dimensional data. The mesh generation unit 1073 generates mesh data based on at least one of the two-dimensional data and the three-dimensional data. The model generation unit 1074 generates three-dimensional model data by performing machine learning based on at least one of the two-dimensional data and the three-dimensional data.

[0121] The two-dimensional data input to the data generation unit 1071 may be a two-dimensional image acquired by a camera. The three-dimensional data input to the data generation unit 1071 may be point cloud data acquired by a sensor such as LiDAR of a space such as a construction site, factory, or office. The data generation unit 1071 may generate color information corresponding to each point included in the point cloud data of the three-dimensional data as attribute information using a two-dimensional image of the two-dimensional data. The three-dimensional data generated by the data generation unit 1071 may be divided into any space. The point cloud data, mesh data, and three-dimensional model data may each be divided into any space.

[0122] The synchronization unit 1075 synchronizes the spatial positions of the point cloud data, mesh data, and 3D model data generated by the data generation unit 1071 or the time of each data (playback time, decoding time, acquisition time, etc.). The time of each data is the playback time, decoding time, acquisition time, etc. The synchronization unit 1075 may generate synchronization information for synchronization without synchronizing the point cloud data, mesh data, and 3D model data. The synchronization unit 1075 may synchronize at least two types of 3D data among the point cloud data, mesh data, and 3D model data generated by the data generation unit 1071, or may perform processing to generate synchronization information (synchronization signal) for synchronization, and may not necessarily perform processing to synchronize three types of 3D data (synchronization processing).

[0123] The point cloud encoding unit 1076 encodes the point cloud data after synchronization processing has been performed by the synchronization unit 1075. Note that the point cloud encoding unit 1076 does not necessarily have to encode the point cloud data. The point cloud data may be encoded in advance, or may be encoded in response to a request from the terminal 1090.

[0124] The mesh encoding unit 1077 encodes the mesh data after the synchronization process has been performed by the synchronization unit 1075 .

[0125] The model encoding unit 1078 encodes the three-dimensional model data after the synchronization process has been performed by the synchronization unit 1075 .

[0126] The multiplexing unit 1079 multiplexes the encoded point cloud data (encoded point cloud), the encoded mesh data (encoded mesh data), the encoded 3D model data, and the synchronization information using a predetermined format or a predetermined multiplexing method. Note that multiplexing by the multiplexing unit 1079 does not have to be performed. In this case, the server 1070 does not have to be equipped with the multiplexing unit 1079.

[0127] The data extraction unit 1080 extracts a portion of the multiplexed three-dimensional data in response to a request from the terminal 1090, and transmits the extracted portion of the three-dimensional data to the terminal 1090. Note that data extraction by the data extraction unit 1080 does not have to be performed. In this case, the server 1070 does not have to be equipped with the data extraction unit 1080. If data extraction by the data extraction unit 1080 is not performed, the server 1070 may transmit to the terminal 1090 the three-dimensional data multiplexed by the multiplexing unit 1079. Furthermore, if multiplexing by the multiplexing unit 1079 is not performed, the server 1070 may transmit to the terminal 1090 the coded point cloud data (coded point cloud), coded mesh data (coded mesh), coded three-dimensional model data (coded three-dimensional model), and synchronization information, or may transmit to the terminal 1090 a bitstream including the coded point cloud data (coded point cloud), coded mesh data (coded mesh), coded three-dimensional model data (coded three-dimensional model), and synchronization information.

[0128] The terminal 1090 includes a control unit 1091 , a decoding unit 1092 , and a presentation unit 1093 .

[0129] The control unit 1091 transmits a request for the part of the three-dimensional data to be presented to the server 1070. The control unit 1091 may accept an operation by the user to specify the part of the three-dimensional data.

[0130] The decoding unit 1092 decodes a portion of the three-dimensional data based on the bit stream (encoded data) acquired from the server 1070 .

[0131] The presentation unit 1093 renders and presents a portion of the decoded three-dimensional data.

[0132] The data generation unit 1071 in Fig. 11 may be realized by a data generation unit 1110 shown in Fig. 12. Fig. 12 is a block diagram showing another example of the data generation unit of the server.

[0133] The data generation unit 1110 includes a point cloud generation unit 1111 , a mesh generation unit 1112 , and a model generation unit 1113 .

[0134] The point cloud generation unit 1111 has the same function as the point cloud generation unit 1072. The point cloud generation unit 1111 acquires point cloud data obtained from the point cloud sensor 1101 and a two-dimensional image obtained from the camera 1102, and generates point cloud data based on the point cloud data and the two-dimensional image. The point cloud data generated by the point cloud generation unit 1111 includes position information of each point and attribute information corresponding to each point indicated by the position information, and includes attribute information (such as color information) extracted from the two-dimensional image.

[0135] The mesh generation unit 1112 generates mesh data based on the point cloud data generated by the point cloud generation unit 1111 .

[0136] The model generation unit 1113 has the same functions as the model generation unit 1074. The model generation unit 1113 acquires point cloud data obtained from the point cloud sensor 1101 and two-dimensional images obtained from the camera 1102, and performs machine learning based on the point cloud data and the two-dimensional images to generate three-dimensional model data.

[0137] The point cloud data, mesh data, and three-dimensional model data may be data generated independently of each other, as described in Fig. 11. The mesh data may be generated from the point cloud data, as described in Fig. 12. Note that the point cloud data may be generated from the mesh data.

[0138] A mesh may be generated from a point cloud, or a point cloud may be generated from a mesh.

[0139] The point cloud data, mesh data, and three-dimensional model data may be generated by the server 1070, or may be generated by a sensor or a terminal 1090 equipped with a sensor. The sensor is, for example, a point cloud sensor 1101 and a camera 1102.

[0140] Next, the relationship between the three-dimensional space and the coded data will be described with reference to Fig. 13.

[0141] As described above, the three-dimensional data includes, for example, any one of point cloud data, mesh data, and a three-dimensional model.

[0142] As shown in Fig. 13, when three-dimensional data is divided into three pieces of three-dimensional data in three three-dimensional spaces (tiles or spaces), the encoding device encodes each of the three divided pieces of three-dimensional data, attaches a header to each piece, and creates a data unit. The header signals (assigns) an identifier (Space_ID) of the space to which the encoded data of the data unit belongs, and an identifier (DataUnit_ID) of the data unit.

[0143] The data units are further given a header including an identifier of the data unit or length information of the data unit, and are then unitized to generate coding scheme units.

[0144] Next, the syntax of the coding method unit will be described. Fig. 14 is a diagram showing an example of the syntax of the coding method unit. Fig. 15 is a diagram showing an example of the syntax of the coding point group. Fig. 16 is a diagram showing an example of the syntax of the coding mesh. Fig. 17 is a diagram showing an example of the syntax of the coding 3D model.

[0145] The unit_type indicates the type of data unit stored in the encoding method unit, thereby specifying the type of data unit stored in the encoding method unit.

[0146] The length indicates the length of the data unit.

[0147] data() indicates the body of the data unit.

[0148] In Fig. 15, when unit_type indicates 0, it indicates that the data unit is position information (geometry) of the coded point group. When unit_type indicates 1, it indicates that the data unit is attribute information of the coded point group. When unit_type indicates 2, it indicates that the data unit is metadata of the coded point group.

[0149] In Fig. 16, when unit_type indicates 0, it indicates that the data unit is position information (geometry) of the coded mesh. When unit_type indicates 1, it indicates that the data unit is attribute information of the coded mesh. When unit_type indicates 2, it indicates that the data unit is metadata of the coded mesh.

[0150] In Fig. 17, when unit_type indicates 0, it indicates that the data unit is element 1 of the encoded 3D model. When unit_type indicates 1, it indicates that the data unit is element 2 of the encoded 3D model. When unit_type indicates 2, it indicates that the data unit is metadata of the encoded 3D model.

[0151] Note that the syntaxes shown in Figures 15 to 17 are merely examples and are not limited to the above configurations. These syntaxes may use a partial configuration of the syntax, or a type (classification) not described above may be used, or the order of the syntax components may be changed. For example, the syntax of the encoding method unit may have a configuration of the encoding method unit common to multiple encoding methods as shown in Figure 14, and may include the unit_type, length, and data() shown in Figures 15 to 17.

[0152] The coding method unit may further include a header indicating the type of the coding method unit. The coding unit type may include, for example, point_cloud_codec_unit indicating point cloud data, mesh_codec_unit indicating mesh data, and model_codec_unit indicating 3D model data. This allows multiple coding methods to be handled in an integrated manner.

[0153] FIG. 18 is a diagram showing an example of the syntax of three-dimensional data information.

[0154] In the syntax, when multiple encoding methods are stored in one format, the number of 3D data included in the format (number_of_3Dformat) and the type of 3D data (format_type) are indicated, and data of each format may be stored. This makes it possible to handle multiple encoding methods or 3D data in an integrated manner, and also to identify multiple encoding methods or 3D data.

[0155] 3Ddata_info indicates format structure information for storing multiple three-dimensional data.

[0156] number_of_3Dformat indicates the number of 3D formats used.

[0157] format_type indicates the type of format of the three-dimensional data to be stored. For example, the number of format_type and the format corresponding to that number may be defined as follows: When format_type indicates 0, it indicates that the format of the three-dimensional data to be stored is point cloud data. When format_type indicates 1, it indicates that the format of the three-dimensional data to be stored is mesh data. When format_type indicates 2, it indicates that the format of the three-dimensional data to be stored is G-PCC data (g-pcc). When format_type indicates 3, it indicates that the format of the three-dimensional data to be stored is V-DMC data (v-dmc). When format_type indicates 4, it indicates that the format of the three-dimensional data to be stored is three-dimensional model data (3Dmodel).

[0158] Next, the data structure of encoded data of multiple pieces of three-dimensional data will be described for each type of three-dimensional data. Fig. 19 is a diagram for explaining the data structure of an encoded point group. Fig. 20 is a diagram for explaining the data structure of an encoded mesh. Fig. 21 is a diagram for explaining the data structure of an encoded three-dimensional model.

[0159] The encoding device divides each of the multiple types of three-dimensional data into multiple pieces of three-dimensional data for each of multiple spatial regions, encodes each of the multiple pieces of divided three-dimensional data (i.e., multiple divided three-dimensional data), and generates encoded data.

[0160] A header is attached to each piece of encoded data, and at least one of a data_unit_id and a space_id is stored.

[0161] Here, data_unit_id is an identifier that identifies a data unit in the coded data and is unique within the coded data. Furthermore, space_id indicates identification information for a spatial region. If data_unit_id or space_id is common to multiple pieces of three-dimensional data, the same value is indicated in the multiple pieces of three-dimensional data.

[0162] 19 to 21, the data unit with data_unit_id=0 in the encoded point cloud, the data unit with data_unit_id=3 in the encoded mesh, and the data unit with data_unit_id=0 in the encoded 3D model are all assigned space_id=1, which means that they are 3D data contained in a common 3D space indicated by Space_ID #1.

[0163] The data and headers may be included in a bitstream structure such as a data unit or encoding method, or may be stored in a predetermined file format such as each box of ISOBMFF.

[0164] Next, three-dimensional space information will be described. Fig. 22 is a diagram showing an example of a plurality of three-dimensional spaces in two dimensions. Fig. 23 is a diagram showing an example of a bounding box. Fig. 24 is a diagram showing an example of the syntax of three-dimensional space information.

[0165] In the syntax of the three-dimensional space information, 3Dspace_info is information indicating the divided three-dimensional space. 3Dspace_info can be used for partial decoding.

[0166] The number_of_space indicates the number of divided three-dimensional spaces.

[0167] The space_id indicates the identifier of the divided three-dimensional space.

[0168] The three-dimensional space information includes bounding box information as information for defining the bounding box shown in FIG.

[0169] The bounding box information includes bounding_box_xyz and bounding_box_whd.

[0170] bounding_box_xyz indicates the coordinates of the reference point of the bounding box, and in the example of Fig. 23, is expressed as x, y, and z coordinate values ​​(x0, y0, z0).

[0171] The bounding_box_whd field indicates the size of the bounding box, which in the example of Fig. 23 is expressed as width w, height h, and depth d (w0, h0, d0).

[0172] The three-dimensional spatial information may also include an identifier of a data unit for each piece of coded data. However, the three-dimensional spatial information does not necessarily have to include the identifier. In other words, the identifier does not necessarily have to be signaled.

[0173] The pointcloud_id indicates the identifier of the data unit of the encoded point cloud of the space corresponding to the space_id.

[0174] mesh_id indicates the identifier of the data unit of the encoding mesh of the space corresponding to space_id.

[0175] model_id indicates the identifier of the data unit of the coded three-dimensional model of the space corresponding to space_id.

[0176] When the data unit does not indicate a space_id but indicates a data_unit_id, an identifier of the data unit for each piece of coded data may be stored in information indicating each space of the three-dimensional space information, thereby allowing the three-dimensional space information to be associated with the divided three-dimensional coded data.

[0177] Furthermore, when a space_id is indicated in a data unit, the three-dimensional spatial information may be associated with an identifier of the data unit for each encoded data by the space_id. In this case, the identifier of the data unit for each encoded data does not need to be stored.

[0178] The division method, the origin of each divided space, and the bounding box size may be the same for mesh data and point cloud data, thereby sharing the three-dimensional spatial information of the point cloud data and the mesh data. Furthermore, the same three-dimensional spatial information may be used for both point cloud data and mesh data. In this way, the three-dimensional spatial information may be shared or the same three-dimensional spatial information may be used between multiple different types of three-dimensional data. Sharing the three-dimensional spatial information facilitates switching between different types of three-dimensional data (e.g., switching between presentations or transmissions). Furthermore, in a format that handles multiple three-dimensional data in an integrated manner, three-dimensional spatial information does not need to be provided for each piece of three-dimensional data; instead, one piece of three-dimensional spatial information can be used for each piece of three-dimensional data, thereby reducing the amount of three-dimensional spatial information.

[0179] In addition to the point cloud data and mesh data, the three-dimensional spatial information of the three-dimensional model may be synchronized with other types of three-dimensional data, or may be made common with the three-dimensional spatial information of other types of three-dimensional data.

[0180] Next, the relationship between the data structure of three-dimensional data and partial decoding will be described. Fig. 25 is a flowchart showing an example of partial decoding. Fig. 26 is a diagram showing an example of a three-dimensional spatial region that is the target of partial decoding. Fig. 27 is a diagram showing an example of the data structure of an encoded point group that is partially decoded. Fig. 28 is a diagram showing an example of the data structure of an encoded mesh that is partially decoded. Fig. 29 is a diagram showing an example of the data structure of an encoded three-dimensional model that is partially decoded.

[0181] In partial decoding, the decoding device first determines a three-dimensional spatial region to be partially decoded (S1001).

[0182] Next, the decoding device uses the three-dimensional spatial information (3Dspace_info) to identify an area that overlaps with the target three-dimensional spatial area from the bounding box information of multiple three-dimensional spatial areas, and obtains the space_id corresponding to the identified area (S1002).

[0183] Next, the decoding device obtains and decodes the data unit having the obtained space_id from the encoded data (S1003). As a result, the decoding device performs partial decoding, which decodes a part of the three-dimensional data. In partial decoding, the decoding device decodes only a part of the three-dimensional data without decoding the entire three-dimensional data.

[0184] For example, as shown in FIG. 26, when the three-dimensional space area to be partially decoded is the area indicated by the thick line, the space_id of the three-dimensional space to be acquired is determined to be #2 from the three-dimensional space information.

[0185] Then, as shown in FIGS. 27 to 29, the data unit associated with Space_id=#2 from the coded data of the multiple types of three-dimensional data is obtained and decoded.

[0186] The decoding device may also obtain a data unit ID instead of space_id from the three-dimensional spatial information, and obtain and partially decode a data unit having the obtained data unit ID.

[0187] In the above-described embodiment, point cloud data, mesh data, and three-dimensional model data are exemplified as three-dimensional data representing a three-dimensional object, but the present invention is not limited to these. For example, a three-dimensional object may be represented by a plurality of sets, each of which includes line-of-sight information indicating a line of sight and a two-dimensional image of the three-dimensional object as viewed from the line of sight. In other words, data including the plurality of sets may be treated as a type of three-dimensional data. Furthermore, the three-dimensional data may be data in another format, such as Gaussian splatting data.

[0188] Fig. 30 is a diagram showing an example of the configuration of a decoding device, and Fig. 31 is a flowchart showing an example of a decoding method performed by the decoding device.

[0189] The decoding device 1130 includes a circuit 1131 and a memory 1132 connected to the circuit 1131 .

[0190] The circuit 1131 performs the following operations.

[0191] The circuit 1131 acquires encoded data including first data representing a three-dimensional object, encoding format information (format) indicating one of encoding formats including second data representing the three-dimensional object, and identification information indicating a three-dimensional space including the three-dimensional object (S1021). Next, the circuit 1131 decodes the first data and the second data corresponding to the three-dimensional space based on the encoded data (S1022). Next, the circuit 1131 renders the first data to generate first presentation data for presentation (S1023). Next, the circuit 1131 renders the second data to generate second presentation data for presentation (S1024). Next, the circuit 1131 switches from the generated second presentation data to the first presentation data and presents it (S1025). Note that the first presentation data and the second presentation data are, for example, two-dimensional data or three-dimensional data generated by the rendering reconstruction unit 1034.

[0192] According to this, the first presentation data and the second presentation data are generated based on the first data and the second data corresponding to the three-dimensional space, and the first presentation data is switched to be presented, so that the two pieces of data representing the three-dimensional object can be presented without causing a spatial misalignment when they are switched, thereby making it possible to appropriately present the first presentation data and the second presentation data.

[0193] For example, the first data is point cloud data representing the three-dimensional object.

[0194] Therefore, the second presentation data is switched to the first presentation data based on the point cloud data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0195] For example, the second data is mesh data representing the three-dimensional object.

[0196] Therefore, the second presentation data based on mesh data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0197] For example, the second data is three-dimensional model data representing the three-dimensional object, and the three-dimensional model data represents a machine learning model obtained by machine learning a plurality of sets of gazes and two-dimensional images.

[0198] Therefore, the second presentation data based on the three-dimensional model data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0199] For example, the second data is a two-dimensional image of the three-dimensional object when viewed from a predetermined line of sight.

[0200] Therefore, the second presentation data based on a two-dimensional image is switched to the first presentation data, so that the two pieces of data representing a three-dimensional object can be switched and presented without causing any spatial misalignment.

[0201] For example, the circuit may further receive a request to switch presentation data from a user, and in the presentation, the circuit may switch from the second presentation data to the first presentation data in response to the request to switch presentation data.

[0202] Therefore, the switching can be performed at a timing designated by the user.

[0203] For example, the circuit may further receive an operation from a user to change the presentation mode, and the circuit may change the presentation mode in response to the operation, and switch from the second presentation data to the first presentation data in response to the change.

[0204] Therefore, switching can be performed at a timing according to the user's operation.

[0205] For example, the circuit may acquire the encoded data from an encoding device via a communication network during the acquiring step, and may switch from the second presentation data to the first presentation data during the presenting step in accordance with a bandwidth of the communication network.

[0206] Therefore, switching can be performed according to the bandwidth of the communication network; for example, when the bandwidth of the communication network changes from less than a specified bandwidth to greater than or equal to the specified bandwidth, the second presentation data can be switched to the first presentation data and presented.

[0207] For example, the circuit switches from the second presentation data to the first presentation data in accordance with the available capabilities of the circuit during the presentation.

[0208] Therefore, switching can be performed according to the capacity of the available circuit; for example, when the capacity of the available circuit changes from less than a predetermined capacity to greater than or equal to the predetermined capacity, the second presentation data can be switched to the first presentation data and presented.

[0209] For example, the encoded data includes synchronization information for synchronizing a coordinate system of the first data and a coordinate system of the second data, and the circuit, during presentation, presents the first presentation data and the second presentation data based on the synchronization information.

[0210] Therefore, the first presentation data and the second presentation data can be switched to the first presentation data after the coordinate systems of the first presentation data and the second presentation data are aligned, so that the two pieces of data representing the three-dimensional object can be switched and presented with less spatial misalignment.

[0211] For example, the circuit further determines whether to synchronize a coordinate system of the first data with a coordinate system of the second data. If the circuit determines that the coordinate system of the first data is synchronized with the coordinate system of the second data, the circuit presents the first presentation data and the second presentation data based on the synchronization information during the presentation.

[0212] This allows synchronization processing to be performed when necessary and can be skipped when not necessary, potentially reducing the processing load.

[0213] For example, the first data and the second data each have a common structure.

[0214] This reduces the amount of coded data, thereby reducing communication capacity.

[0215] For example, the encoded data includes spatial information for identifying the three-dimensional space in which the three-dimensional object is located. The circuit further acquires a target area indicating a portion of the three-dimensional space. The circuit identifies first duplicate data that is a portion of the first data and overlaps with the target area based on the spatial information. In the decoding process, the circuit decodes the identified first duplicate data.

[0216] Therefore, for example, by acquiring only the first duplicate data, the volume of data to be acquired can be reduced, thereby reducing communication capacity. Also, for example, it is possible to decrypt only the first duplicate data, thereby reducing processing load.

[0217] Alternatively, the circuit 1131 may operate according to the decoding method shown in the flowchart of Fig. 32. Fig. 32 is a flowchart showing another example of the decoding method performed by the decoding device.

[0218] The circuit 1131 decodes encoding information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data (S1031). The circuit 1131 decodes second data in the second encoding method indicated by the encoding information (S1032). The second data is used to generate second presentation data for presentation.

[0219] This allows the second data of the second encoding method indicated by the encoding method information obtained by decoding to be decoded, thereby obtaining second data for generating appropriate second presentation data for presentation.

[0220] Fig. 33 is a diagram showing an example of the configuration of an encoding device, and Fig. 34 is a flowchart showing an example of an encoding method performed by the encoding device.

[0221] The encoding device 1140 includes a circuit 1141 and a memory 1142 connected to the circuit 1141 .

[0222] The circuit 1141 performs the following operations.

[0223] The circuit 1141 generates encoding information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data (S1041). The circuit 1141 generates second data in the second encoding method indicated by the encoding information (S1042). The circuit 1141 generates a bitstream including the encoding information and the second data (S1043). The second data is used to generate second presentation data for presentation.

[0224] This allows a bitstream including encoding method information and second data to be generated, and a decoding device that acquires the bitstream can obtain the second data for generating appropriate second presentation data for presentation.

[0225] Second Embodiment A method for generating a still image of a subject (three-dimensional object) viewed from an arbitrary viewpoint in a still space using a three-dimensional data generation model, which is a learning model obtained based on learning, will be described.

[0226] Fig. 35 is a diagram for explaining the processing during learning of a 3D generation model in embodiment 2. Fig. 36 is a diagram for explaining the processing for generating a still image of a subject viewed from an arbitrary viewpoint using a 3D generation model in embodiment 2.

[0227] The information processing device can generate a still image viewed from an arbitrary viewpoint in a still space by acquiring a three-dimensional data generation model through learning. For example, there is a three-dimensional data generation model generated by a method such as NeRF (Neural Radiance Fields).

[0228] During learning, the information processing device acquires learning data including, for example, a viewpoint A image (ground truth value) acquired from an arbitrary viewpoint A and viewpoint information (such as the camera posture) of viewpoint A when the image was acquired. The viewpoint information may include viewpoint A and the line of sight direction from viewpoint A. The information processing device, for example, uses an evaluation function 1402 to input the viewpoint information from the learning data into a three-dimensional data generation model 1401, thereby optimizing parameters of a network included in the three-dimensional data generation model so as to minimize the difference between the generated image of viewpoint A output from the three-dimensional data generation model 1401 and the viewpoint A image, which is an input image corresponding to viewpoint A. The information processing device can acquire a more accurate three-dimensional data generation model by performing this learning process using multiple learning data corresponding to multiple different viewpoints. The learning process is performed for each of the multiple viewpoints. In other words, the same process as the learning process for viewpoint A is performed for each viewpoint.

[0229] During generation, when viewpoint information of viewpoint B, for example, is input to the trained three-dimensional data generation model 1403, the information processing device outputs a generated image of viewpoint B, and when viewpoint information of viewpoint Z different from viewpoint B is input, the information processing device outputs a generated image of viewpoint Z. The viewpoint information of viewpoint B may include viewpoint B and the line of sight direction from viewpoint B. The viewpoint information of viewpoint Z may include viewpoint Z and the line of sight direction from viewpoint Z.

[0230] In this way, a still image viewed from an arbitrary viewpoint in a still space can be generated by acquiring the three-dimensional data generation model 1403 through learning. However, a moving image cannot be generated in this state.

[0231] While Figure 36 illustrates an example of a 3D data generation model that generates an image from a viewpoint when viewpoint information is input, the present invention is not limited to this, and the data output from the 3D data generation model may be in any format. For example, the 3D data generation model may be a network model that outputs 3D data of a target space obtained by learning in the form of point cloud data or mesh data. This allows a user to view the target space in a stereoscopic manner using 3D data such as point cloud data or mesh data, and also allows the user to measure the dimensions of an object in the target space output as 3D data using the point cloud data or mesh data.

[0232] 37 is a diagram for explaining a moving image generation method using a three-dimensional data generation model of Example 1 in Embodiment 2. Note that in this Example, an example configuration of a device and a method for encoding or decoding three-dimensional data generation models NNt0 to NNt5 generated corresponding to times t0 to t5 will be described, but the present invention is not necessarily limited to this, and may also be applied to a device and a method for encoding or decoding a three-dimensional data generation model at each time during an arbitrary period.

[0233] This example illustrates a method for generating a moving image of a target object (subject) viewed from an arbitrary viewpoint using a three-dimensional data generation model. With this method, as shown in FIG. 37 , for example, by acquiring a three-dimensional data generation model corresponding to each time, still images of the target object viewed from an arbitrary viewpoint at each time can be generated, and a moving image can be generated by arranging the generated still images in chronological order. More specifically, when generating a moving image from times t0 to t5, multiple three-dimensional data generation models NNt0 to NNt5 corresponding to times t0 to t5 are generated by learning, and viewpoint information (such as camera posture) of viewpoint A from which a moving image is to be generated is input to the generated three-dimensional data generation models NNt0 to NNt5 corresponding to times t0 to t5. As a result, generated images of viewpoint A from times t0 to t5 are output by the three-dimensional data generation models NNt0 to NNt5, and by temporally connecting these images, a moving image of the target object viewed from viewpoint A from times t0 to t5 can be generated.

[0234] However, in this case, it is necessary to store multiple 3D data generation models corresponding to multiple times, which requires a huge storage capacity for storing the data of the multiple 3D data generation models in storage or a huge network bandwidth for transmitting the data of the multiple 3D data generation models over a network. Therefore, the data size may be reduced by data encoding the multiple 3D data generation models corresponding to multiple times using, for example, Neural Network Coding (NNC) of the Moving Picture Experts Group (MPEG) standard. This disclosure describes a method for more efficiently compressing this data.

[0235] NNC is described in Non-Patent Document 1.

[0236] FIG. 38 is a diagram illustrating a first example of the configuration of an encoding device according to a first example of the second embodiment.

[0237] The encoding device 1420 includes a three-dimensional data generation model acquisition unit 1421 , a buffer unit 1422 , and a network model encoding unit 1423 .

[0238] The 3D data generation model acquisition unit 1421 acquires learning data from times t0 to t5 and uses the acquired learning data from times t0 to t5 to generate 3D data generation models NNt0 to NNt5 for times t0 to t5 through learning. The learning data includes multiple viewpoint images obtained by photographing the target object from one or more viewpoint positions and one or more line-of-sight directions for each of the times t0 to t5, and one or more pieces of viewpoint information indicating one or more viewpoint positions and one or more line-of-sight directions corresponding to the multiple viewpoint images. The one or more pieces of viewpoint information may be the position and orientation of the camera when each of the multiple viewpoint images was captured. Note that the learning data is not limited to this and may further include information obtained from other sensors. For example, the learning data may include point cloud data, depth images, etc. acquired at each time using a LiDAR or TOF sensor. This improves the accuracy of the 3D data generation model obtained through learning.

[0239] The buffer unit 1422 stores the three-dimensional data generation model at time t generated by the three-dimensional data generation model acquisition unit 1421. The buffer unit 1422 is realized by a storage device such as a memory. The three-dimensional data generation model at time t stored in the buffer unit 1422 may be used, for example, as an initial model when the three-dimensional data generation model acquisition unit 1421 acquires (generates) a three-dimensional data generation model after time t by learning. This makes it possible to shorten the learning time while improving the accuracy of the three-dimensional data generation model after time t.

[0240] The buffer unit 1422 may store multiple 3D data generation models corresponding to multiple times. This allows, for example, a single initial model to be generated by processing such as averaging based on the multiple 3D data generation models stored in the buffer unit 1422. The 3D data generation model acquisition unit 1421 can acquire a highly accurate 3D data generation model by learning a 3D data generation model after time t using this initial model. If the 3D data generation model acquisition unit 1421 does not refer to past 3D data generation models during learning, the encoding device 1420 may not need to include the buffer unit 1422. This allows the amount of memory used by the buffer unit 1422 to be reduced.

[0241] The network model encoding unit 1423 encodes the three-dimensional data generation models NNt0 to NNt5 acquired by the three-dimensional data generation model acquisition unit 1421 and outputs a bit stream.

[0242] As a network model encoding method, for example, data size may be reduced by data encoding using the NNC of the MPEG standard. That is, the network model encoding unit 1423 encodes the three-dimensional data generation models NNt0 to NNt5 using the NNC and adds the encoding results to a bitstream. In other words, the network model encoding unit 1423 generates encoded data as the encoding result and generates a bitstream including the encoded data.

[0243] Specifically, the network model encoding unit 1423 first encodes the three-dimensional data generation model NNt0 at time t0 using an NNC and adds the encoding result to the bit stream. Next, the network model encoding unit 1423 encodes the three-dimensional data generation model NNt1 at time t1 using an NNC and adds the encoding result to the bit stream. In this way, the network model encoding unit 1423 may reduce the amount of code by encoding the three-dimensional data generation model at each time in sequence using an NNC and adding each encoding result to the bit stream.

[0244] At this time, the network model encoding unit 1423 may add time information indicating the time to which the encoded three-dimensional data generation model corresponds to the bitstream as metadata. This allows the decoding device to know the time to which the decoded three-dimensional data generation model corresponds by decoding and referring to the metadata included in the bitstream, and to appropriately generate a moving image from an arbitrary viewpoint of the target object.

[0245] The metadata is not limited to time information, and may include information related to the acquisition (generation) of learning data, or information required for the decoding device to generate a moving image.

[0246] For example, the network model encoding unit 1423 may add, as metadata, information about the frame rate of the camera when acquiring (generating) the training data, which allows the decoding device to decode the frame rate of the generated video from the bitstream and appropriately set the frame rate.

[0247] Alternatively, the network model encoding unit 1423 may add a frame number corresponding to each time to the bitstream as metadata instead of the time information, and link each frame number to the time information using another parameter. For example, the network model encoding unit 1423 may add the time information and frame rate of the first frame as metadata, and the decoding device may calculate the time information of each frame from the metadata, thereby reducing the amount of code by the amount of time information for each frame.

[0248] Furthermore, the network model encoding unit 1423 may add viewpoint information of the viewpoint image used in training to the bitstream. This allows the decoding device to generate high-quality video by, for example, preferentially selecting a viewpoint that is close to the viewpoint position corresponding to the image used in training. This is because the closer the viewpoint position or time is to the time of training, the more likely it is that the 3D data generation model will be able to generate viewpoint images with higher quality.

[0249] FIG. 39 is a diagram illustrating a first example of the configuration of a decoding device according to a first example of the second embodiment.

[0250] The decoding device 1425 includes a network model decoding unit 1426 and a rendering unit 1427 .

[0251] The network model decoding unit 1426 acquires the bit stream and decodes the three-dimensional data generation models NNt0 to NNt5 for times t0 to t5 and metadata such as time information based on the acquired bit stream.

[0252] The rendering unit 1427 generates a moving image of viewpoint A based on viewpoint information of viewpoint A specified by a user or a system, using the three-dimensional data generation models NNt0 to NNt5 decoded by the network model decoding unit 1426 and metadata such as time information. Specifically, the rendering unit 1427 inputs the viewpoint information of viewpoint A into the three-dimensional data generation model NNt0 at time t0 to generate an image IMGt0 of viewpoint A at time t0, and then inputs the viewpoint information of viewpoint A into the three-dimensional data generation model NNt1 at time t1 to generate an image IMGt1 of viewpoint A at time t1. The rendering unit 1427 applies the image generation process for each of these times to each of times t2 to t5, thereby generating images IMGt2 to IMGt5 of viewpoint A at times t2 to t5. Then, the rendering unit 1427 uses the images IMGt0 to IMGt5 and metadata such as time information to generate a moving image of the target object from time t0 to t5 as viewed from viewpoint A. The moving image may include, for example, the images IMGt0 to IMGt5 and presentation time information for calculating the presentation times of the images IMGt0 to IMGt5 based on the times t0 to t5.

[0253] Note that the viewpoint information may change depending on the time. For example, viewpoint information of viewpoint A may be input to the three-dimensional data generation models NNt0 to NNt3 from time t0 to t3, and viewpoint information of viewpoint B may be input to the three-dimensional data generation models NNt4 to NNt5 from time t4 to t5. This allows the rendering unit 1427 to generate multiple images of the target object viewed from viewpoint A from time t0 to t3, and multiple images of the target object viewed from viewpoint B from time t4 to t5. In other words, the rendering unit 1427 can generate a moving image of the target object in which the viewpoint switches from viewpoint A to viewpoint B at time t4.

[0254] The rendering unit 1427 does not necessarily need to generate a moving image, but may generate a still image of a specified viewpoint at a specified time, thereby allowing the user to switch between generating a moving image or a still image depending on the application.

[0255] Note that the rendering unit 1427 is not limited to generating moving images or still images from the three-dimensional data generation model. For example, the rendering unit 1427 may generate point cloud data or mesh data from the three-dimensional data generation model and output the generated point cloud data or mesh data as dynamic point cloud data or dynamic mesh data. This allows the user to view dynamic three-dimensional data of a dynamic target object on a head-mounted display (HMD) or the like, and also allows the user to measure the amount of movement of the target object using the dynamic three-dimensional data.

[0256] FIG. 40 is a diagram illustrating a second example of the configuration of the encoding device according to the first example of the second embodiment.

[0257] The encoding device 1430 includes a three-dimensional data generation model acquisition unit 1431 , a buffer unit 1432 , a difference calculation unit 1433 , and a network model encoding unit 1434 .

[0258] The three-dimensional data generation model acquisition unit 1431 is the same as the three-dimensional data generation model acquisition unit 1421 of the encoding device 1420 .

[0259] The buffer unit 1432 is similar to the buffer unit 1422 of the encoding device 1420, but differs from the buffer unit 1422 in that it inputs a three-dimensional data generation model stored in a memory or the like to the difference calculation unit 1433 as a reference three-dimensional data generation model.

[0260] The difference calculation unit 1433 calculates difference information indicating the difference between the three-dimensional data generation models NNt0 to NNt5 for times t0 to t5 generated by the three-dimensional data generation model acquisition unit 1431 and each of the three-dimensional data generation models (hereinafter referred to as reference three-dimensional data generation models) generated by the three-dimensional data generation model acquisition unit 1431 before each time. Here, the difference information may include, for example, differences in weight parameters at the nodes of each network model. For example, the difference calculation unit 1433 acquires the three-dimensional data generation model NNt5 for time t5 from the three-dimensional data generation model acquisition unit 1431, and acquires the three-dimensional data generation model NNt4 for time t4 from the buffer unit 1432 as the reference three-dimensional data generation model.

[0261] The difference calculation unit 1433 may use the three-dimensional data generation model NNt5 and the three-dimensional data generation model NNt4 to calculate, for example, the difference (amount of change) between the weight parameters of the nodes of the network model in the three-dimensional data generation model NNt5 and the weight parameters of the nodes of the network model in the three-dimensional data generation model NNt4, and input difference information indicating this difference to the network model encoding unit 1434. The difference information is then encoded by the network model encoding unit 1434. In other words, the encoding device 1430 may reduce the amount of data by predicting information related to the network model in the three-dimensional data generation model NNt5 from the three-dimensional data generation model NNt4 and encoding the difference from the predicted value. Such predictive encoding reduces the value of the difference to be encoded when there is little change in the three-dimensional data generation model over time, such as when the target object is almost stationary, thereby improving encoding efficiency. For example, the encoding device 1430 may set RNNt0 = 0 and RNNtn = NNt(n-1) (n is an integer value from 1 to 5) and reduce the amount of bits by predictive encoding using the three-dimensional data generation model from the previous time as a reference three-dimensional data generation model.

[0262] In the second example, the encoding device 1430 predictively encodes information related to the network model in the 3D data generation model NNt5 from information related to the network model in the 3D data generation model NNt4, but this is not necessarily limited to this. For example, the encoding device 1430 may select a reference 3D data generation model to be used for prediction from one or more 3D data generation models stored in the buffer unit 1432 and perform predictive encoding using the selected 3D data generation model. In this case, the encoding device 1430 may add information indicating the selected 3D data generation model (reference 3D data generation model information) to the bitstream to communicate the selected 3D data generation model to the decoding device. This allows the encoding device 1430 to select the optimal reference 3D data generation model in terms of encoding efficiency, thereby improving encoding efficiency. Furthermore, the decoding device can appropriately decode a bitstream with improved encoding efficiency by decoding the reference 3D data generation model information.

[0263] When the encoding device 1430 performs predictive encoding by referring to two or more 3D data generation models stored in the buffer unit 1432, the encoding device 1430 may add information indicating the two or more reference 3D data generation models to the bitstream. This allows the encoding device 1430 to improve the encoding efficiency of predictive encoding using two or more reference 3D data generation models. Furthermore, the decoding device can appropriately decode the bitstream with improved encoding efficiency.

[0264] Note that in cases where a reference 3D data generation model is not stored in the buffer unit 1432, for example, when encoding the first 3D data generation model in data order (first frame), the encoding device 1430 may encode the 3D data generation model to be processed without prediction, without calculating the difference from a predicted value (hereinafter referred to as intra prediction), or may calculate and encode the difference from a predicted value set to 0. Furthermore, when setting a certain time t as a random access point, the encoding device 1430 may encode the 3D data generation model corresponding to time t using intra prediction, or may calculate and encode the difference from a predicted value set to 0. This allows the decoding device to start decoding the 3D data generation model from the first 3D data generation model in data order (first frame) or the random access point, thereby improving functionality during playback.

[0265] Furthermore, a set of multiple 3D data generation models (multiple frames) (hereinafter referred to as a GOF (Group of Frame)) may be defined, and the first frame of the GOF may be coded using intra-prediction. This allows the decoding device to randomly access the first frame of the GOF, and by decoding the first frame of the GOF, functionality such as fast-forward playback can be enhanced.

[0266] The encoding device 1430 may also add permission information to the bitstream, which indicates whether or not inter-GOF predictive reference is permitted. For example, if the bitstream includes permission information indicating that inter-GOF predictive reference is prohibited, the decoding device can determine that multiple GOFs can be decoded in parallel. Furthermore, for example, allowing inter-GOF predictive reference can improve encoding efficiency.

[0267] The network model encoding unit 1434 is similar to the network model encoding unit 1423 of the encoding device 1420, but differs in that it encodes the difference information d0 to d5 of the three-dimensional data generation models NNt0 to NNt5 input from the difference calculation unit 1433 and outputs a bit stream.

[0268] Note that although the encoding device 1430 has been described with the difference calculation unit 1433 and the network model encoding unit 1434 separated, this is not necessarily limited to this, and for example, the difference calculation unit 1433 may be included in the network model encoding unit 1434. In other words, the network model encoding unit 1434 may perform the processing of the difference calculation unit 1433.

[0269] The encoding device 1430 may add predictive coding information to the bitstream, indicating whether the 3D data generation model was encoded using intra prediction or predictive coding using a reference 3D data generation model (hereinafter referred to as inter prediction). This allows the decoding device to appropriately determine whether to use intra prediction or inter prediction to decode the 3D data generation model by decoding the predictive coding information.

[0270] FIG. 41 is a diagram illustrating a second example of the configuration of the decoding device according to the first embodiment of the present invention.

[0271] The decoding device 1435 includes a network model decoding unit 1436 , an adding unit 1437 , a buffer unit 1438 , and a rendering unit 1439 .

[0272] The network model decoding unit 1436 acquires the bit stream and, based on the acquired bit stream, decodes the difference information d0 to d5 of the three-dimensional data generation models NNt0 to NNt5 for times t0 to t5, as well as metadata such as time information.

[0273] The adder 1437 adds, for the corresponding times, the difference information d0 to d5 of the three-dimensional data generation model decoded by the network model decoding unit 1436 and corresponding to times t0 to t5, and the reference three-dimensional data generation models RNNt0 to RNNt5 acquired from the buffer unit 1438, to calculate three-dimensional data generation models NNt0 to NNt5. In this way, the decoding device 1435 may set RNNt0 = 0 and RNNtn = NNt(n-1) (n is a value from 1 to 5), and perform predictive decoding using the three-dimensional data generation model of the previous time as the reference three-dimensional data generation model.

[0274] Note that, in the decoding device 1435 in the second example, the addition unit 1437 and the network model decoding unit 1436 are described separately, but this is not necessarily limited to this, and for example, the addition unit 1437 may be included in the network model decoding unit 1436. In other words, the network model decoding unit 1436 may perform the processing of the addition unit 1437.

[0275] Note that in cases where a reference 3D data generation model is not stored in the buffer unit 1438, for example, when decoding the first 3D data generation model (first frame) in data order, the decoding device 1435 may perform decoding without prediction (hereinafter referred to as intra prediction) without the addition unit 1437 adding the difference information and the reference 3D data generation model, or may perform decoding by adding a prediction value set to 0 and the difference information. Furthermore, when setting a certain time t as a random access point, the decoding device 1435 may decode the 3D data generation model corresponding to time t using intra prediction, or may decode by adding a prediction value set to 0 and the difference information. Furthermore, when predictive coding information indicating that the 3D data generation model to be decoded has been coded using intra prediction is included in the bitstream, the 3D data generation model may be decoded using intra prediction, or may be decoded by adding a prediction value set to 0 and the difference information. This allows the decoding device 1435 to start decoding the three-dimensional data generation model from the first three-dimensional data generation model (first frame) in data order, a random access point, or a three-dimensional data generation model coded using intra-prediction, thereby improving functionality during playback.

[0276] In the second example, the decoding device 1435 predictively decodes information related to the network model in the 3D data generation model NNt5 from information related to the network model in the 3D data generation model NNt4, but this is not necessarily limited to this. For example, the decoding device 1435 may select a reference 3D data generation model to be used for prediction from one or more 3D data generation models stored in the buffer unit 1438 and perform predictive decoding using the selected 3D data generation model. In this case, the decoding device 1435 may decode information indicating the selected 3D data generation model (reference 3D data generation model information) from the bitstream. This allows the decoding device 1435 to appropriately decode a bitstream with improved coding efficiency by decoding the reference 3D data generation model information from the bitstream generated by the encoding device 1430, selecting the reference 3D data generation model that is optimal in terms of coding efficiency.

[0277] Note that when performing predictive decoding with reference to two or more 3D data generation models stored in the buffer unit 1438, the decoding device 1435 may decode information indicating the two or more reference 3D data generation models from the bitstream. This allows the decoding device 1435 to appropriately decode a bitstream in which the coding efficiency of predictive coding has been improved using two or more reference 3D data generation models.

[0278] The rendering unit 1439 is similar to the rendering unit 1427 of the decoding device 1425. The rendering unit 1439 does not necessarily need to generate a moving image, but may generate a still image of specified viewpoint information at a specified time.

[0279] [Example 2] Fig. 42 is a diagram for explaining a moving image generation method using an extended three-dimensional data generation model of Example 2 in Embodiment 2. Note that in this example, an example configuration of a device and a method for encoding or decoding extended three-dimensional data generation models NNt0-2 and NNt3-5 generated corresponding to periods t0 to t2 and periods t3 to t5, respectively, from time t0 to t5, will be described, but the present invention is not necessarily limited to this, and may also be applied to a device and a method for encoding or decoding an extended three-dimensional data generation model for any period.

[0280] This embodiment describes a method for generating moving images of a target object (subject) viewed from an arbitrary viewpoint using a three-dimensional data generation model. With this method, for example, as shown in FIG. 42 , by acquiring a three-dimensional data generation model (hereinafter referred to as an extended three-dimensional data generation model) capable of generating images from an arbitrary viewpoint within a certain time range (period), still images of the target object viewed from an arbitrary viewpoint at an arbitrary time within each period can be generated, and moving images can be generated by arranging the generated still images in chronological order. The extended three-dimensional data generation model is a three-dimensional data generation model generated, for example, by a method such as NeRF, similar to the three-dimensional data generation model of the first embodiment.

[0281] More specifically, when generating a moving image from time t0 to t5, an extended three-dimensional data generation model NNt0-2 capable of representing the period from time t0 to t2 and an extended three-dimensional data generation model NNt3-5 capable of representing the period from time t3 to t5 are generated by learning, and viewpoint information (camera posture, etc.) of viewpoint A from which a moving image is to be generated is input to the generated extended three-dimensional data generation model NNt0-2 and extended three-dimensional data generation model NNt3-5. As a result, generated images from viewpoint A from time t0 to t5 are output by the extended three-dimensional data generation model NNt0-2 and extended three-dimensional data generation model NNt3-5, and by connecting these images in time, a moving image from time t0 to t5 in which the target object is viewed from viewpoint A can be generated.

[0282] However, in this case, it is necessary to store an augmented 3D data generation model corresponding to each period (each time zone), which requires a huge storage capacity for storing the data of the augmented 3D data generation model in storage, or a huge network bandwidth for transmitting the data of multiple 3D data generation models over a network. Therefore, the data size may be reduced by data encoding the augmented 3D data generation model corresponding to each period using, for example, Neural Network Coding (NNC) of the Moving Picture Experts Group (MPEG) standard. This disclosure describes a method for more efficiently compressing this data.

[0283] With the above configuration, the information processing device may generate any viewpoint image at any time within the period from time t0 to time t5. For example, when acquiring the extended 3D data generation model NNt0-2, the information processing device generates the extended 3D data generation model NNt0-2 by machine learning based on multiple viewpoint images captured at times t0, t1, and t2 as learning data and camera postures corresponding to the multiple viewpoints. When generating a moving image of viewpoint A, the information processing device may generate not only viewpoint image A at times t0, t1, and t2, but also images of any viewpoint at times t0.5 and t1.5 between times t0, t1, and t2, for example. Time t0.5 is a time between times t0 and t1, and time t1.5 is a time between times t1 and t2.

[0284] This allows the information processing device to generate images from any viewpoint corresponding not only to the time corresponding to the image during learning, but also to a time shifted from the time corresponding to the image during learning, thereby enabling the generation of moving images from viewpoint A at a high frame rate.

[0285] The information processing device may perform learning using not only the learning data at times t0, t1, and t2 but also, for example, learning data at time t3 as learning data for the extended 3D data generation model NNt0-2. This allows for highly accurate generation of viewpoint images from any viewpoint after time t2, for example, an image from an arbitrary viewpoint at time t2.5.

[0286] Furthermore, the information processing device may use not only the learning data corresponding to times t3, t4, and t5 as the learning data for the extended 3D data generation model NNt3-5, but also, for example, learning data corresponding to times t2 and t6 for learning. This allows the information processing device to generate, with high accuracy, images from any viewpoint before time t3 or images from any viewpoint after time t5. Note that, for example, in the above example, when generating a viewpoint image at time 2.5, which is between times t2 and t3 when the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5 switch, the information processing device may generate a viewpoint image at time t2.5 for each of the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5, and then generate an average image of the two generated viewpoint images at time t2.5 as the viewpoint image at time t2.5. This allows for the generation of a highly accurate viewpoint image at time t2.5.

[0287] In this way, the information processing device can generate an image of the target object viewed from the specified viewpoint at the specified time by specifying the time within the period corresponding to the extended three-dimensional data generation model and viewpoint information for the extended three-dimensional data generation model.

[0288] FIG. 43 is a diagram illustrating a first example of a configuration of an encoding device according to a second example of the second embodiment.

[0289] The encoding device 1450 includes an extended three-dimensional data generation model acquisition unit 1451 , a buffer unit 1452 , and a network model encoding unit 1453 .

[0290] The extended 3D data generation model acquisition unit 1451 acquires learning data for each of the periods t0 to t2 and t3 to t5, and generates an extended 3D data generation model NNt0-2 for the period t0 to t2 and an extended 3D data generation model NNt3-5 for the period t3 to t5 by learning using the acquired learning data for each period. The learning data includes multiple viewpoint images obtained by photographing the target object from one or more viewpoint positions and one or more line-of-sight directions for each of the times t0 to t5, and one or more pieces of viewpoint information indicating one or more viewpoint positions and one or more line-of-sight directions corresponding to the multiple viewpoint images. The one or more pieces of viewpoint information may be the position and orientation of the camera when each of the multiple viewpoint images was captured. Note that the learning data is not limited to this and may further include information obtained from other sensors. For example, the learning data may include point cloud data, depth images, etc. acquired at each time using a LiDAR or TOF sensor. This improves the accuracy of the extended 3D data generation model obtained by learning.

[0291] The buffer unit 1452 stores the extended 3D data generation model for the period tm-n from time tm (m is an integer) to time tn (n is an integer greater than m) generated by the extended 3D data generation model acquisition unit 1451. The buffer unit 1452 is realized by a storage device such as a memory. The extended 3D data generation model for the period tm-n stored in the buffer unit 1452 may be used, for example, as an initial model when the extended 3D data generation model acquisition unit 1451 acquires (generates) an extended 3D data generation model for a period after the period tm-n by learning. This makes it possible to shorten the learning time while improving the accuracy of the extended 3D data generation model for a period after the period tm-n.

[0292] The buffer unit 1452 may store multiple extended 3D data generation models corresponding to multiple time periods. As a result, for example, a single initial model may be generated based on the multiple extended 3D data generation models stored in the buffer unit 1452, for example, by processing such as averaging. The extended 3D data generation model acquisition unit 1451 can acquire a highly accurate extended 3D data generation model by learning an extended 3D data generation model for a period after the period tm-n using this initial model. If the extended 3D data generation model acquisition unit 1451 does not refer to extended 3D data generation models for past periods during learning, the encoding device 1450 may not need to include the buffer unit 1452. This reduces the amount of memory used by the buffer unit 1452.

[0293] The network model encoding unit 1453 encodes the extended three-dimensional data generation models NNt0-2 and NNt3-5 acquired by the extended three-dimensional data generation model acquisition unit 1451, and outputs a bit stream.

[0294] As a network model encoding method, for example, the data size may be reduced by data encoding using the NNC of the MPEG standard. That is, the network model encoding unit 1453 encodes the extended three-dimensional data generation models NNt0-2 and NNt3-5 using NNC and adds the encoding results to a bitstream. In other words, the network model encoding unit 1453 generates encoded data as the encoding result and generates a bitstream including the encoded data.

[0295] Specifically, the network model encoding unit 1453 first encodes the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 using an NNC and adds the encoding result to the bit stream. Next, the network model encoding unit 1453 encodes the extended three-dimensional data generation model NNt3-5 for the period t3 to t5 using an NNC and adds the encoding result to the bit stream. In this way, the network model encoding unit 1453 may reduce the amount of code by encoding the extended three-dimensional data generation models for each period in sequence using an NNC and adding each encoding result to the bit stream.

[0296] At this time, the network model encoding unit 1453 may add time information indicating to which period the encoded extended 3D data generation model corresponds to as metadata to the bitstream. This allows the decoding device to know to which period the decoded extended 3D data generation model corresponds by decoding and referring to the metadata included in the bitstream, and to appropriately generate moving images from any viewpoint of the target object.

[0297] The network model encoding unit 1453 may generate information indicating the period for which the extended 3D data generation model can generate viewpoint images as time information, and add the generated time information to the bitstream as metadata. This allows the decoding device to know the period for which the extended 3D data generation model can generate viewpoint images by decoding this metadata, and thereby generate a moving image appropriately.

[0298] The metadata is not limited to time information, and may include information related to the acquisition (generation) of learning data, or information required for the decoding device to generate a moving image.

[0299] For example, the network model encoding unit 1453 may add, as metadata, information about the frame rate of the camera when acquiring (generating) the training data, which allows the decoding device to decode the frame rate of the generated video from the bitstream and appropriately set the frame rate.

[0300] Alternatively, the network model encoding unit 1453 may add frame numbers corresponding to each period to the bit stream as metadata instead of time information, and link each frame number to the time information using another parameter. For example, the network model encoding unit 1453 may add the time information and frame rate of the first frame as metadata, and the decoding device may calculate the time information of each frame from the metadata, thereby reducing the amount of code by the amount of time information for each frame.

[0301] Furthermore, the network model encoding unit 1453 may add viewpoint information of the viewpoint image used in training or time information indicating the time when the viewpoint image was captured to the bitstream. This allows the decoding device to generate high-quality video by, for example, preferentially selecting a viewpoint close to the viewpoint position corresponding to the image used in training or a time close to the time corresponding to the image used in training. This is because the closer the viewpoint position or time is to the time during training, the more likely it is that the extended 3D data generation model will be able to generate viewpoint images with higher quality.

[0302] FIG. 44 is a diagram illustrating a first example of a configuration of a decoding device according to a second example of the second embodiment.

[0303] The decoding device 1455 includes a network model decoding unit 1456 and a rendering unit 1457 .

[0304] The network model decoding unit 1456 acquires the bit stream and, based on the acquired bit stream, decodes the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 and the extended three-dimensional data generation model NNt3-5 for the period t3 to t5, as well as metadata such as time information corresponding to these extended three-dimensional data generation models NNt0-2 and NNt3-5.

[0305] The rendering unit 1457 generates a moving image of viewpoint A based on viewpoint information of viewpoint A specified by a user or a system, using the extended three-dimensional data generation models NNt0-2 and NNt3-5 decoded by the network model decoding unit 1456 and metadata such as time information. Specifically, the rendering unit 1457 inputs the viewpoint information of viewpoint A and the times within the period t0 to t2 into the extended three-dimensional data generation model NNt0-2 for the period t0 to t2, and generates an image IMGt0 of viewpoint A at time t0, an image IMGt1 of viewpoint A at time t1, and an image IMGt2 of viewpoint A at time t2. The rendering unit 1457 applies the image generation process for the period t0 to t2 to the extended three-dimensional data generation model NNt3-5 for the period t3 to t5, thereby generating images IMGt3 to IMGt5 of viewpoint A at times t3 to t5. Then, the rendering unit 1457 uses the images IMGt0 to IMGt5 and metadata such as time information to generate a moving image of the target object from time t0 to t5 as viewed from viewpoint A. The moving image may include, for example, the images IMGt0 to IMGt5 and presentation time information for calculating the presentation times of the images IMGt0 to IMGt5 based on the times t0 to t5.

[0306] Note that the viewpoint information may be changed according to time. For example, viewpoint information of viewpoint A may be input to the extended three-dimensional data generation model NNt0-2 for the period t0 to t2, and viewpoint information of viewpoint B may be input to the extended three-dimensional data generation model NNt3-5 for the period t3 to t5. This allows the rendering unit 1457 to generate multiple images of the target object viewed from viewpoint A from time t0 to t2, and multiple images of the target object viewed from viewpoint B from time t3 to t5. In other words, the rendering unit 1457 can generate a moving image of the target object in which the viewpoint switches from viewpoint A to viewpoint B at time t3.

[0307] The rendering unit 1457 does not necessarily need to generate a moving image, but may generate a still image of a specified viewpoint at a specified time, thereby allowing the user to switch between generating a moving image or a still image depending on the application.

[0308] Note that the rendering unit 1457 is not limited to generating moving images or still images from the extended 3D data generation model. For example, the rendering unit 1457 may generate point cloud data or mesh data from the extended 3D data generation model for a period that the extended 3D data generation model can represent, and output the generated point cloud data or mesh data as dynamic point cloud data or dynamic mesh data. This allows the user to view dynamic 3D data of a dynamic target object on a head-mounted display (HMD) or the like, and also allows the user to measure the amount of movement of the target object using the dynamic 3D data.

[0309] FIG. 45 is a diagram illustrating a second example of the configuration of the encoding device according to the second embodiment.

[0310] The encoding device 1460 includes an extended three-dimensional data generation model acquisition unit 1461 , a buffer unit 1462 , a difference calculation unit 1463 , and a network model encoding unit 1464 .

[0311] The extended three-dimensional data generation model acquisition unit 1461 is the same as the extended three-dimensional data generation model acquisition unit 1451 of the encoding device 1450 .

[0312] The buffer unit 1462 is similar to the buffer unit 1452 of the encoding device 1450, but differs from the buffer unit 1452 in that it inputs an extended 3D data generation model stored in a memory or the like to the difference calculation unit 1463 as a reference extended 3D data generation model.

[0313] The difference calculation unit 1463 calculates difference information indicating the difference between the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 and the extended three-dimensional data generation model NNt3-5 for the period t3 to t5 generated by the extended three-dimensional data generation model acquisition unit 1461, and the extended three-dimensional data generation model (hereinafter referred to as the reference extended three-dimensional data generation model) generated by the extended three-dimensional data generation model acquisition unit 1461 before each period. Here, the difference information may include, for example, differences in weight parameters at the nodes of each network model. The difference calculation unit 1463 acquires, for example, the extended three-dimensional data generation model NNt3-5 for the period t3 to t5 from the extended three-dimensional data generation model acquisition unit 1461, and acquires the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 from the buffer unit 1462 as the reference extended three-dimensional data generation model.

[0314] The difference calculation unit 1463 may use the extended three-dimensional data generation model NNt3-5 and the extended three-dimensional data generation model NNt0-2 to calculate, for example, the difference (amount of change) between the weight parameters of the nodes of the network model in the extended three-dimensional data generation model NNt3-5 and the weight parameters of the nodes of the network model in the extended three-dimensional data generation model NNt0-2, and input difference information indicating this difference to the network model encoding unit 1464. The difference information is then encoded by the network model encoding unit 1464. In other words, the encoding device 1460 may reduce the amount of data by predicting information related to the network model in the extended three-dimensional data generation model NNt3-5 from the extended three-dimensional data generation model NNt0-2 and encoding the difference from the predicted value. Such predictive encoding reduces the value of the difference to be encoded when there is little change in the extended three-dimensional data generation model over time, such as when the target object is almost stationary, thereby improving encoding efficiency. For example, the encoding device 1460 may set RNNt0-2=0 and RNNt3-5=NNt0-2, and reduce the amount of bits by predictive encoding using the extended 3D data generation model of the previous time period as the reference extended 3D data generation model.

[0315] Note that, in the second example, the encoding device 1460 predictively encodes information related to the network model in the extended 3D data generation model NNt3-5 from information related to the network model in the extended 3D data generation model NNt0-2, but this is not necessarily limited to this. For example, the encoding device 1460 may select a reference extended 3D data generation model to be used for prediction from one or more extended 3D data generation models stored in the buffer unit 1462, and perform predictive encoding using the selected extended 3D data generation model. In this case, the encoding device 1460 may add information indicating the selected extended 3D data generation model (reference extended 3D data generation model information) to the bitstream in order to communicate the selected extended 3D data generation model to the decoding device. This allows the encoding device 1460 to select the optimal reference extended 3D data generation model in terms of encoding efficiency, thereby improving encoding efficiency. Furthermore, the decoding device can appropriately decode a bitstream with improved encoding efficiency by decoding the reference extended 3D data generation model information.

[0316] When the encoding device 1460 performs predictive encoding by referring to two or more extended 3D data generation models stored in the buffer unit 1462, the encoding device 1460 may add information indicating the two or more reference extended 3D data generation models to the bitstream. This allows the encoding device 1460 to improve the encoding efficiency of predictive encoding using two or more reference extended 3D data generation models. Furthermore, the decoding device can appropriately decode the bitstream with improved encoding efficiency.

[0317] In addition, in cases where a reference extended 3D data generation model is not stored in the buffer unit 1462, for example, when encoding the first extended 3D data generation model in data order (first frame), the encoding device 1460 may encode the extended 3D data generation model to be processed without prediction, without calculating the difference from a predicted value (hereinafter referred to as intra prediction), or may calculate and encode the difference from a predicted value set to 0. Furthermore, when setting a certain period tm-n as a random access point, the encoding device 1460 may encode the extended 3D data generation model corresponding to the period tm-n using intra prediction, or may calculate and encode the difference from a predicted value set to 0. This allows the decoding device to start decoding the extended 3D data generation model from the first extended 3D data generation model in data order (first frame) or the random access point, thereby improving functionality during playback.

[0318] Furthermore, a set of multiple extended 3D data generation models (multiple frames) (hereinafter referred to as a GOF (Group of Frame)) may be defined, and the first frame of the GOF may be coded using intra prediction. This allows a decoding device to randomly access the first frame of the GOF, and by decoding the first frame of the GOF, functionality such as fast-forward playback can be enhanced.

[0319] The encoding device 1460 may also add permission information to the bitstream indicating whether or not inter-GOF predictive reference is permitted. For example, if the bitstream includes permission information indicating that inter-GOF predictive reference is prohibited, the decoding device can determine that multiple GOFs can be decoded in parallel. Furthermore, for example, allowing inter-GOF predictive reference can improve encoding efficiency.

[0320] The network model encoding unit 1464 is similar to the network model encoding unit 1453 of the encoding device 1450, but differs in that it encodes the difference information d0-2 and d3-5 of the extended three-dimensional data generation models NNt0-2 and NNt3-5 input from the difference calculation unit 1463 and outputs a bit stream.

[0321] Note that although the encoding device 1460 has been described with the difference calculation unit 1463 and the network model encoding unit 1464 separated, this is not necessarily limited to this, and for example, the difference calculation unit 1463 may be included in the network model encoding unit 1464. In other words, the network model encoding unit 1464 may perform the processing of the difference calculation unit 1463.

[0322] The encoding device 1460 may add predictive coding information to the bitstream, indicating whether the extended 3D data generation model was encoded using intra prediction or predictive coding using a reference extended 3D data generation model (hereinafter referred to as inter prediction). This allows the decoding device to appropriately determine whether to use intra prediction or inter prediction to decode the extended 3D data generation model by decoding the predictive coding information.

[0323] FIG. 46 is a diagram illustrating a second example of the configuration of a decoding device according to Example 2 in the second embodiment.

[0324] The decoding device 1465 includes a network model decoding unit 1466 , an adding unit 1467 , a buffer unit 1468 , and a rendering unit 1469 .

[0325] The network model decoding unit 1466 acquires the bit stream and, based on the acquired bit stream, decodes the differential information d0-2, d3-5 of the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5 for the period t0 to t2, as well as metadata such as time information.

[0326] The adder 1467 adds, for the corresponding periods, difference information d0-2, d3-5 of the extended three-dimensional data generation models NNt0-2, NNt3-5 decoded by the network model decoding unit 1466 and corresponding to the periods t0 to t2, t3 to t5, to the reference extended three-dimensional data generation models RNNt0-2, RNNt3-5 acquired from the buffer unit 1468, to calculate the extended three-dimensional data generation models NNt0-2, NNt3-5. In this way, the decoding device 1465 may set RNNt0-2 = 0, RNNt3-5 = NNt0-2, and perform predictive decoding on the extended three-dimensional data generation model of the previous time period as the reference extended three-dimensional data generation model.

[0327] Note that, although the decoding device 1465 in the second example is described with the addition unit 1467 and the network model decoding unit 1466 separated, this is not necessarily limited thereto, and for example, the addition unit 1467 may be included in the network model decoding unit 1466. In other words, the network model decoding unit 1466 may perform the processing of the addition unit 1467.

[0328] In addition, in cases where a reference extended 3D data generation model is not stored in the buffer unit 1468, for example, when decoding the first extended 3D data generation model (first frame) in data order, the decoding device 1465 may perform decoding without prediction (hereinafter referred to as intra prediction) without the addition unit 1467 adding the difference information and the reference extended 3D data generation model, or may perform decoding by adding a prediction value set to 0 and the difference information. Furthermore, when a certain period tm-n is set as a random access point, the decoding device 1465 may decode the extended 3D data generation model corresponding to the period tm-n using intra prediction, or may decode by adding a prediction value set to 0 and the difference information. Furthermore, when predictive encoding information indicating that the extended 3D data generation model to be decoded has been encoded using intra prediction is included in the bitstream, the extended 3D data generation model may be decoded using intra prediction, or may be decoded by adding a prediction value set to 0 and the difference information. This allows the decoding device 1465 to start decoding the extended 3D data generation model from the first extended 3D data generation model (first frame) in data order, a random access point, or an extended 3D data generation model coded using intra prediction, thereby improving functionality during playback.

[0329] In the second example, the decoding device 1465 predictively decodes information related to the network model in the extended 3D data generation model NNt3-5 from information related to the network model in the extended 3D data generation model NNt0-2, but this is not necessarily limited to this. The decoding device 1465 may, for example, select a reference extended 3D data generation model to be used for prediction from one or more extended 3D data generation models stored in the buffer unit 1468, and perform predictive decoding using the selected extended 3D data generation model. In this case, the decoding device 1465 may decode information indicating the selected extended 3D data generation model (reference extended 3D data generation model information) from the bitstream. As a result, the decoding device 1465 can appropriately decode a bitstream with improved coding efficiency by decoding the reference extended 3D data generation model information from the bitstream generated by the encoding device 1460, in which the optimal reference extended 3D data generation model in terms of coding efficiency is selected.

[0330] Note that the decoding device 1465 may decode information indicating the two or more reference extended 3D data generation models from the bitstream when performing predictive decoding with reference to two or more extended 3D data generation models stored in the buffer unit 1468. This allows the decoding device 1465 to appropriately decode a bitstream in which the coding efficiency of predictive coding has been improved using two or more reference extended 3D data generation models.

[0331] The rendering unit 1469 is similar to the rendering unit 1427 of the decoding device 1425. The rendering unit 1469 does not necessarily need to generate a moving image, but may generate a still image of specified viewpoint information at a specified time.

[0332] [Modification] The encoding device 1460 may include information regarding the number of images that the extended 3D data generation model can generate (i.e., the upper limit of the number of images) in the metadata that is added to the bit stream of the extended 3D data generation model for the period tm-n. This allows the decoding device 1465 to know the number of images that the decoded extended 3D data generation model can generate, and, for example, to appropriately set the frame rate of the video to be generated or to calculate the number of delays before the video is displayed.

[0333] The encoding device 1460 may add, as time information to be added to the bitstream, information indicating up to what time unit (i.e., the smallest time unit) the extended 3D data generation model can generate viewpoint images. The encoding device 1460 may add, as time information to the bitstream, for example, whether viewpoint images can be generated up to a time unit of 1 msec, or whether viewpoint images can be generated up to a time unit of 1 μmsec. This allows the decoding device 1465 to know up to what time unit viewpoint information is to be generated, and can generate high-frame-rate video or 3D data accordingly.

[0334] Furthermore, the encoding device 1460 may add information about when the extended 3D data generation model was trained to the metadata to be added to the bitstream of the extended 3D data generation model for the period tm-n. For example, the encoding device 1460 adds time information or viewpoint information of the image used for training to the bitstream as metadata, and the decoding device 1465 can decode the metadata to know the time or viewpoint information at which the extended 3D data generation model can generate viewpoint images with high image quality, thereby creating high-quality moving images.

[0335] The width of the period of viewpoint images for which the extended 3D data generation model can be generated may be dynamically switched as shown in Fig. 47. Specifically, the width of the period may be switched depending on the subject. Fig. 47 is a diagram for explaining a moving image generation method using an extended 3D data generation model according to a modification of the second embodiment.

[0336] For example, in a scene with many still objects among the subjects (a scene in which the number of still objects among the multiple subjects is equal to or greater than a first number, or a scene in which the volume (area) occupied by the still objects among the multiple subjects is equal to or greater than a first amount), the encoding device 1460 widens the width of the period of the training data used during training (i.e., lengthens the period), thereby generating an extended 3D data generation model over a long period during which high-quality viewpoint images can be generated. For example, in a scene with many dynamic objects among the subjects (a scene in which the number of dynamic objects among the multiple subjects is equal to or greater than a first number, or a scene in which the volume (area) occupied by the dynamic objects among the multiple subjects is equal to or greater than a first amount), the encoding device 1460 narrows the width of the training data used during training, thereby generating an extended 3D data generation model over a short period during which high-quality viewpoint images can be generated, even for dynamic objects.

[0337] The encoding device 1460 may use training data for a certain period tm-n (e.g., a group of frames (GOF) representing a collection of frames within the period tm-n among the training images) to generate an extended 3D data generation model NNtm-n for that period tm-n. In this case, the encoding device 1460 buffers the training image frames for the period tm-n to generate an extended 3D data generation model, and compresses and transmits the data, resulting in a transmission delay equivalent to the GOF size. The encoding device 1460 may add information regarding the transmission delay, such as the number of GOF frames or the number of delayed frames, to the bitstream. This allows the decoding device 1465 to obtain delay information by decoding the bitstream and appropriately reproduce video or 3D data taking the delay into account.

[0338] In the above embodiment, an example has been described in which a still image from an arbitrary viewpoint at a certain time or period is generated using a 3D data generation model or an extended 3D data generation model, but this is not necessarily limited to this. For example, the 3D data generation model or the extended 3D data generation model may generate (output) 3D data such as point cloud data or mesh data at a certain time within a certain period, as shown in FIG. 48 . This allows a user to measure the dimensions of a target object or view higher-resolution 3D data. FIG. 48 is a diagram for explaining a moving image generation method using a 3D data generation model according to a modification of the second embodiment.

[0339] Furthermore, the encoding devices 1420 and 1460 may include, in the metadata of the bitstream, information indicating a recommended output format according to the use case, among output formats such as images, point cloud data, mesh data, etc. This allows the user to select a recommended output format according to the use case.

[0340] The encoding devices 1420 and 1460 may add one or more pieces of viewpoint information to the metadata added to the bitstream of the 3D data generation model or the extended 3D data generation model. For example, the encoding devices 1420 and 1460 may include, in the metadata, recommended viewpoint information for viewing the target object, or the user's viewpoint information when acquiring the training data. This allows the decoding devices 1425 and 1465 to generate moving images or 3D data using viewpoint information selected from the one or more pieces of viewpoint information added to the bitstream in accordance with the user's intentions, etc.

[0341] Note that a default viewpoint may be predetermined from one or more pieces of viewpoint information, and the decoding devices 1425 and 1465 may generate moving images or three-dimensional data using the predetermined default viewpoint unless specified by the user. This allows the decoding devices 1425 and 1465 to automatically generate moving images or three-dimensional data without user specification.

[0342] The present embodiment can be used in the following manner, for example.

[0343] First, the encoding devices 1420 and 1460 acquire data of a dynamic object to be transmitted to a remote location using a camera or sensor, and use the data of the dynamic object as learning data to generate a three-dimensional data generation model or an extended three-dimensional data generation model of the dynamic object.

[0344] Next, the encoding devices 1420 and 1460 encode the three-dimensional data generation model or the extended three-dimensional data generation model using the encoding method described in this embodiment, and transmit a bitstream containing the encoding results to a remote location.

[0345] The decoding devices 1425 and 1465 then decode the bitstream received at the remote location, generate moving images or 3D data from any viewpoint using the decoded 3D data generation model of the dynamic object or the extended 3D data generation model, and utilize the generated 3D data for viewing, measurement, etc. In this way, the present embodiment may be applied to general use cases in which information about a certain space is shared at remote locations.

[0346] If there are one or more objects in a space that you want to transmit to a remote location, the three-dimensional data generation modeling, encoding / transmission, decoding, and rendering processes described in this embodiment may be applied separately to each object. For example, a foreground dynamic object and a background static object in a space may be generated as three-dimensional data generation models and encoded / transmitted separately. This allows the application of three-dimensional data generation modeling or encoding methods that are optimal for each object, thereby improving encoding efficiency.

[0347] Furthermore, the present invention is not limited to this, and the three-dimensional data generation and modeling, encoding and transmission, decoding, and rendering processes shown in this embodiment may be applied separately to one or more multiple objects as a single object. This makes it possible to transmit one or more multiple objects to a remote location while reducing the amount of processing.

[0348] Fig. 49 is a diagram showing an example of a configuration of a coding device in Embodiment 2. Fig. 50 is a flowchart showing an example of a coding method performed by the coding device in Embodiment 2.

[0349] The encoding device 1470 includes a circuit 1471 and a memory 1472. The encoding device 1470 is a device that realizes the encoding devices 1420 and 1460.

[0350] The circuit 1471 performs the following operations.

[0351] The circuit 1471 acquires a first three-dimensional data generation model (e.g., three-dimensional data generation model NNt0) corresponding to a first time (e.g., time t0) and a second three-dimensional data generation model (e.g., three-dimensional data generation model NNt1) corresponding to a second time (e.g., time t1) (S1401). The circuit 1471 generates a bitstream by encoding the acquired first three-dimensional data generation model and second three-dimensional data generation model (S1402). When viewpoint information including a viewpoint and a line of sight is input, each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as seen from the viewpoint and the line of sight.

[0352] This makes it possible to generate a bitstream that includes a first three-dimensional data generation model that can obtain a two-dimensional image corresponding to a first time according to arbitrary viewpoint information, and a second three-dimensional data generation model that can obtain a two-dimensional image corresponding to a second time, thereby generating a bitstream that compresses data that can obtain moving images from arbitrary viewpoints. This makes it possible to reduce the memory capacity required to store data that can obtain moving images from arbitrary viewpoints, or the network bandwidth required to transmit that data.

[0353] For example, each of the first three-dimensional data generation model and the second three-dimensional data generation model is a learning model using a neural network.

[0354] For example, the bitstream includes first time information indicating the first time and second time information indicating the second time.

[0355] For example, the bitstream includes a first frame number corresponding to the first time and a second frame number corresponding to the second time.

[0356] For example, the bitstream includes frame rate information regarding frame rates of a plurality of training images used to generate the first 3D data generation model and the second 3D data generation model, the plurality of training images being two-dimensional images captured at a plurality of different times.

[0357] For example, the bitstream includes viewpoint information including viewpoints and line-of-sight directions of a plurality of learning images used to generate the first 3D data generation model and the second 3D data generation model.

[0358] For example, the plurality of learning images are two-dimensional images obtained by photographing the subject from different viewpoints and line-of-sight directions, and the viewpoint information includes the different viewpoints and line-of-sight directions.

[0359] For example, in encoding the second 3D data generation model, the circuit 1471 calculates difference information indicating a difference between the first 3D data generation model and the second 3D data generation model. The bit stream includes the difference information.

[0360] For example, the difference includes a difference between weight parameters associated with nodes included in the first three-dimensional data generation model and the second three-dimensional data generation model.

[0361] For example, the bitstream includes reference information indicating that the difference information has been calculated with reference to the first 3D data generation model.

[0362] For example, the first time corresponds to a random access point. The first 3D data generation model is coded using intra prediction, or is coded using inter prediction with a prediction value of 0.

[0363] For example, the first three-dimensional data generation model and the second three-dimensional data generation model are included in one set of a plurality of sets, and the first three-dimensional data generation model is the first in data order among the plurality of three-dimensional data generation models included in the one set.

[0364] For example, the bitstream includes permission information indicating whether or not, in encoding each of the plurality of three-dimensional data generation models, the three-dimensional data generation model is permitted to reference three-dimensional data generation models included in other sets.

[0365] For example, the first three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt0-2) corresponds to a first period (e.g., period t0 to t2) including the first time (e.g., time t0), and the second three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt3-5) corresponds to a second period (e.g., period t3 to t5) including the second time (e.g., time t3).

[0366] For example, the plurality of first learning images used to generate the first three-dimensional data generation model are two-dimensional images obtained by photographing at a plurality of different times during the first period.

[0367] For example, when a time included in the first period is input, the first three-dimensional data generation model outputs a two-dimensional image of the subject at the input time.

[0368] For example, the bitstream includes number information indicating the upper limit of the number of images that can be generated by the first 3D data generation model.

[0369] For example, the bitstream includes first information about the first training images, the first information including a plurality of viewpoints, a plurality of line-of-sight directions, and a plurality of different timings corresponding to the first training images.

[0370] For example, the first period or the second period is dynamically determined depending on the subject.

[0371] For example, the circuit 1471 stores the generated first three-dimensional data generation model in the memory 1472. The circuit 1471 generates the second three-dimensional data generation model based on the first three-dimensional data generation model stored in the memory 1472.

[0372] For example, the circuit 1471 stores the generated first and second three-dimensional data generation models in the memory 1472. The circuit 1471 generates an initial model based on the first and second three-dimensional data generation models stored in the memory 1472. The circuit 1471 generates a third three-dimensional data generation model (e.g., three-dimensional data generation model NNt2) corresponding to a third time (e.g., time t2) based on the initial model.

[0373] Fig. 51 is a diagram showing an example of the configuration of a decoding device in Embodiment 2. Fig. 52 is a flowchart showing an example of a decoding method performed by the decoding device in Embodiment 2.

[0374] The decoding device 1480 includes a circuit 1481 and a memory 1482. The decoding device 1480 is a device that realizes the decoding devices 1425 and 1465.

[0375] The circuit 1481 performs the following operations.

[0376] The circuit 1481 acquires a bitstream (S1411). The circuit 1481 decodes from the bitstream a first 3D data generation model (e.g., 3D data generation model NNt0) corresponding to a first time (e.g., time t0) and a second 3D data generation model (e.g., 3D data generation model NNt1) corresponding to a second time (e.g., time t1) (S1412). When viewpoint information including a viewpoint and a line of sight is input, each of the first 3D data generation model and the second 3D data generation model outputs a 2D image of a subject as seen from the viewpoint and the line of sight.

[0377] According to this, based on a bitstream that compresses data that allows for video images from any viewpoint, it is possible to decode a first three-dimensional data generation model that allows for a two-dimensional image corresponding to a first time point in accordance with any viewpoint information, and a second three-dimensional data generation model that allows for a two-dimensional image corresponding to a second time point. Therefore, it is possible to appropriately decode a bitstream that can reduce the memory capacity for storing data that allows for video images from any viewpoint to be obtained, or the network bandwidth for transmitting that data.

[0378] For example, each of the first three-dimensional data generation model and the second three-dimensional data generation model is a learning model using a neural network.

[0379] For example, the bitstream includes first time information indicating the first time and second time information indicating the second time.

[0380] For example, the bitstream includes a first frame number corresponding to the first time and a second frame number corresponding to the second time.

[0381] For example, the bitstream includes frame rate information regarding frame rates of a plurality of training images used to generate the first 3D data generation model and the second 3D data generation model, the plurality of training images being two-dimensional images captured at a plurality of different times.

[0382] For example, the bitstream includes viewpoint information including viewpoints and line-of-sight directions of a plurality of learning images used to generate the first 3D data generation model and the second 3D data generation model.

[0383] For example, the plurality of learning images are two-dimensional images obtained by photographing the subject from different viewpoints and line-of-sight directions, and the viewpoint information includes the different viewpoints and line-of-sight directions.

[0384] For example, the bitstream includes difference information indicating a difference between the first 3D data generation model and the second 3D data generation model.

[0385] For example, the difference includes a difference between weight parameters associated with nodes included in the first three-dimensional data generation model and the second three-dimensional data generation model.

[0386] For example, the bitstream includes reference information indicating that the difference information has been calculated with reference to the first 3D data generation model.

[0387] For example, the first time corresponds to a random access point. The first 3D data generation model is coded using intra prediction, or is coded using inter prediction with a prediction value of 0.

[0388] For example, the first three-dimensional data generation model and the second three-dimensional data generation model are included in one set of a plurality of sets, and the first three-dimensional data generation model is the first in data order among the plurality of three-dimensional data generation models included in the one set.

[0389] For example, the bitstream includes permission information indicating whether or not, in encoding each of the plurality of three-dimensional data generation models, the three-dimensional data generation model is permitted to reference three-dimensional data generation models included in other sets.

[0390] For example, the first three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt0-2) corresponds to a first period (e.g., period t0 to t2) including the first time (e.g., time t0), and the second three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt3-5) corresponds to a second period (e.g., period t3 to t5) including the second time (e.g., time t3).

[0391] For example, the plurality of first learning images used to generate the first three-dimensional data generation model are two-dimensional images obtained by photographing at a plurality of different times during the first period.

[0392] For example, when a time included in the first period is input, the first three-dimensional data generation model outputs a two-dimensional image of the subject at the input time.

[0393] For example, the bitstream includes number information indicating the upper limit of the number of images that can be generated by the first 3D data generation model.

[0394] For example, the bitstream includes first information about the first training images, the first information including a plurality of viewpoints, a plurality of line-of-sight directions, and a plurality of different timings corresponding to the first training images.

[0395] For example, the first period or the second period is dynamically determined depending on the subject.

[0396] For example, the circuit 1471 stores the generated first three-dimensional data generation model in the memory 1472. The circuit 1471 generates the second three-dimensional data generation model based on the first three-dimensional data generation model stored in the memory 1472.

[0397] For example, the circuit 1471 stores the generated first and second three-dimensional data generation models in the memory 1472. The circuit 1471 generates an initial model based on the first and second three-dimensional data generation models stored in the memory 1472. The circuit 1471 generates a third three-dimensional data generation model (e.g., three-dimensional data generation model NNt2) corresponding to a third time (e.g., time t2) based on the initial model.

[0398] (Other) In one embodiment, a method for generating a moving image viewed from a predetermined viewpoint is disclosed. The generation of the moving image is realized, for example, by a device including a memory and a circuit connected to the memory. This example device stores a 3D data generation model (Neural Network) generated by learning in memory, and the circuit acquires the 3D data generation model (Neural Network) stored in the memory and generates a moving image based on the 3D data generation model. Note that the 3D data generation model or the extended 3D data generation model does not necessarily have to be stored in memory. For example, the encoding devices 1420 and 1460 may acquire specification information specifying a URL on a network and acquire the 3D data generation model based on the specification information.

[0399] FIG. 53 is a diagram illustrating an example of the configuration of an encoding device.

[0400] The encoding device 1490 includes a processor 1491 and a memory 1492 .

[0401] The processor 1491 is a circuit that performs information processing and is a circuit that can access the memory 1492. For example, the processor 1491 is a dedicated or general-purpose electronic circuit that encodes a three-dimensional data generation model. The processor 1491 may be a processor such as a CPU. The processor 1491 may also be a collection of multiple electronic circuits. For example, the processor 1491 may also fulfill the roles of multiple components of the aforementioned encoding device, excluding the component for storing information.

[0402] The memory 1492 is a dedicated or general-purpose memory that stores information used by the processor 1491 to encode the three-dimensional data generation model. The memory 1492 may be an electronic circuit and may be connected to the processor 1491. The memory 1492 may also be included in the processor 1491. The memory 1492 may also be a collection of multiple electronic circuits. The memory 1492 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as storage, a recording medium, or the like. The memory 1492 may also be a non-volatile memory or a volatile memory.

[0403] For example, the three-dimensional data generation model to be encoded may be stored in the memory 1492, or a stream corresponding to the encoded three-dimensional data generation model may be stored in the memory 1492. Furthermore, the memory 1492 may store a program that allows the processor 1491 to encode the three-dimensional data generation model.

[0404] It should be noted that not all of the components of the encoding device described above may be implemented, and not all of the processes described above may be performed, in the encoding device 1490. Some of the components may be included in other devices, and some of the processes described above may be performed by other devices.

[0405] FIG. 54 is a diagram illustrating an example of the configuration of a decoding device.

[0406] The decoding device 1495 includes a processor 1496 and a memory 1497 .

[0407] The processor 1496 is a circuit that processes information and is a circuit that can access the memory 1497. For example, the processor 1496 is a dedicated or general-purpose electronic circuit that decodes a stream. The processor 1496 may be a processor such as a CPU. The processor 1496 may also be a collection of multiple electronic circuits. For example, the processor 1496 may fulfill the roles of multiple components of the above-mentioned decoding device, excluding the component for storing information.

[0408] The memory 1497 is a dedicated or general-purpose memory that stores information for the processor 1496 to decode the stream. The memory 1497 may be an electronic circuit and may be connected to the processor 1496. The memory 1497 may also be included in the processor 1496. The memory 1497 may also be a collection of multiple electronic circuits. The memory 1497 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 1497 may also be a non-volatile memory or a volatile memory.

[0409] For example, a three-dimensional data generation model or a stream may be stored in memory 1497. Furthermore, memory 1497 may also store a program for processor 1496 to decode the stream.

[0410] Note that not all of the components of the above-described decoding device may be implemented, and not all of the above-described processes may be performed, in the decoding device 1495. Some of the components may be included in another device, and some of the above-described processes may be executed by another device.

[0411] Third Embodiment Another method for generating a still image of a subject (three-dimensional object) viewed from an arbitrary viewpoint in a still space using a three-dimensional data generation model, which is a learning model obtained based on learning, will be described.

[0412] In the above embodiment, an example of a method for generating a still image seen from an arbitrary viewpoint at an arbitrary time by generating an extended three-dimensional data generation model capable of generating an image from an arbitrary viewpoint within a certain time range (period), and generating a moving image by arranging the still images in chronological order, and a method for encoding or decoding the extended three-dimensional data generation model, is shown. In this embodiment, when an extended three-dimensional data generation model is acquired by learning, a method for generating a more accurate extended three-dimensional data generation model by also learning the latent code Zt at time t, and an example of encoding the extended three-dimensional data generation model and the latent code is shown.

[0413] Fig. 55 is a diagram for explaining the processing during learning of a three-dimensional data generation model in embodiment 3. Fig. 56 is a diagram for explaining the processing for generating a still image of a subject viewed from an arbitrary viewpoint using a three-dimensional data generation model in embodiment 3.

[0414] As in the above embodiment, the information processing device can generate a still image viewed from an arbitrary viewpoint in a still space by acquiring a three-dimensional data generation model through learning. For example, there is a three-dimensional data generation model generated by a method such as NeRF (Neural Radiance Fields).

[0415] During training, the information processing device acquires training data including, for example, a viewpoint A image (ground truth value) acquired from an arbitrary viewpoint A at an arbitrary time t, viewpoint information (such as the camera posture) of the viewpoint A when the image was acquired, and a latent code Zt corresponding to the time t. The viewpoint information may include the viewpoint A and the line of sight direction from the viewpoint A. The information processing device, for example, uses an evaluation function 1502 to input the viewpoint information from the training data into the extended three-dimensional data generation model 1501, thereby optimizing the network parameters and latent code included in the extended three-dimensional data generation model 1501 so as to minimize the difference between the generated image of viewpoint A output from the extended three-dimensional data generation model 1501 and the viewpoint A image that is the input image of viewpoint A at time t. In other words, the information processing device trains the three-dimensional data generation model at time t using the latent code Zt set for the arbitrary time t.

[0416] The information processing device can acquire a more accurate 3D data generation model by performing this learning process using one or more learning data corresponding to one or more multiple time points and one or more multiple viewpoints. The learning process is performed for the learning data corresponding to each of the multiple viewpoints. In other words, the same process as the learning process for viewpoint A is performed for each viewpoint.

[0417] The latent code may be, for example, a one-dimensional value or a multidimensional vector having n-dimensional values ​​(n is an integer equal to or greater than 2). For example, if the latent code has 256-dimensional values, it may be expressed as vector information represented by a total of 256 values, value0t to value255t, such as the latent code Zt at time t = (value0t, value1t, ..., value255t). In this way, by expressing the latent code as a multidimensional vector, the characteristics of each time can be more accurately expressed.

[0418] The initial value of the latent code used for learning may be set to 0 or a random value. If the latent code is a multidimensional vector, all components may be set to 0 or all components may be set to random values. This reduces the processing required to calculate the initial value.

[0419] Furthermore, the initial value of the latent code used for learning is not limited to the above, and may be set to the value of an already generated extended three-dimensional data generation model and its associated latent code. That is, the initial value of the latent code may be set to the value of a latent code corresponding to a time different from the time corresponding to the latent code, or may be set to the value of a learned latent code. Specifically, when learning the extended three-dimensional data generation model and latent code for times t5 to t9, if the extended three-dimensional data generation model and latent code for times t0 to t4 have already been learned, the model parameters of the learned extended three-dimensional data generation model for times t0 to t4 may be used as the initial values ​​for learning the extended three-dimensional data generation model for times t5 to t9, or the learned latent code for times t0 to t4 may be used as the initial values ​​for learning the latent code for times t5 to t9. This allows for highly accurate generation of the extended three-dimensional data generation model and latent code.

[0420] Note that, when there is little change in the object, such as little movement, between times t0-t4 and t5-t9, the values ​​of the already generated extended 3D data generation model and the associated latent code may be set as initial values. This allows for highly accurate generation of the extended 3D data generation model and latent code in scenes with little movement.

[0421] Next, at the time of generation, for example, when viewpoint information of viewpoint B and a latent code Zt0 corresponding to time t0 are input to the trained extended 3D data generation model 1503, a generated image of viewpoint B at time t0 is output. When viewpoint information of viewpoint Z different from viewpoint B and a latent code Zt4 corresponding to time t4 are input, a generated image of viewpoint Z at time t4 is output. The viewpoint information of viewpoint B may include viewpoint B and the line of sight direction from viewpoint B. The viewpoint information of viewpoint Z may include viewpoint Z and the line of sight direction from viewpoint Z.

[0422] In this way, the information processing device can generate still images viewed from any viewpoint within a certain time range by acquiring the extended 3D data generation model 1503 and the latent code through learning. The information processing device can then generate multiple still images corresponding to multiple times and arrange them in chronological order to generate a moving image.

[0423] While FIG. 56 illustrates an example of an extended 3D data generation model that generates an image of a viewpoint at a certain time when viewpoint information and a latent code at a certain time are input, the present invention is not limited to this, and any data format may be output from the extended 3D data generation model. For example, the extended 3D data generation model may be a network model (network) that outputs a 3D model of a target space at a certain time obtained through learning in the form of point cloud data or mesh data. This allows a user to stereoscopically view the target space at a certain time using 3D data such as point cloud data or mesh data, or to measure the dimensions of an object in the target space at a certain time output as 3D data using point cloud data or mesh data. Note that the viewpoint image, generated image, and still image in this embodiment may represent 2D images.

[0424] [Example 1] Fig. 57 is a diagram for explaining a moving image generation method using an extended three-dimensional data generation model of Example 1 in Embodiment 3. Note that in this example, an example configuration of a device and a method for encoding or decoding extended three-dimensional data generation models NNt0-2 and NNt3-5 generated corresponding to periods t0 to t2 and periods t3 to t5, respectively, and latent codes Zt0 to Zt2 and Zt3 to Zt5 will be described, but the present invention is not necessarily limited to this, and may be applied to a device and a method for encoding or decoding extended three-dimensional data generation models in any period.

[0425] This example shows a method for generating moving images of a target object (subject) viewed from an arbitrary viewpoint using a three-dimensional data generation model. With this method, for example, as shown in FIG. 57 , by acquiring a three-dimensional data generation model (hereinafter referred to as an extended three-dimensional data generation model) capable of generating images from an arbitrary viewpoint within a certain time range (period), still images of the target object viewed from an arbitrary viewpoint at an arbitrary time within each period can be generated, and moving images can be generated by arranging the generated still images in chronological order. The extended three-dimensional data generation model is a three-dimensional data generation model generated using a method such as NeRF.

[0426] More specifically, when generating a moving image from time t0 to t5, an extended three-dimensional data generation model NNt0-2 and latent codes Zt0 to Zt2 capable of representing the period from time t0 to t2, and an extended three-dimensional data generation model NNt3-5 and latent codes Zt3 to Zt5 capable of representing the period from time t3 to t5 are generated by learning, and viewpoint information (such as camera posture) of viewpoint A from which a moving image is to be generated and the latent code Zt0-5 corresponding to times t0 to t5 are input to the generated extended three-dimensional data generation model NNt0-2 and extended three-dimensional data generation model NNt3-5. As a result, generated images from viewpoint A from time t0 to t5 are output by the extended three-dimensional data generation model NNt0-2 and extended three-dimensional data generation model NNt3-5, and by connecting these images in time, a moving image from time t0 to t5 in which the target object is viewed from viewpoint A can be generated.

[0427] In this embodiment, an example of encoding and decoding an extended three-dimensional data generation model NNt0-2 and latent codes Zt0 to Zt2 generated for the periods t0 to t2 and t3 to t5, respectively, and an extended three-dimensional data generation model NNt3-5 and latent codes Zt3 to Zt5 will be described, but this is not necessarily limited to this, and the present invention may also be applied to encoding and decoding an extended three-dimensional data generation model for any period.

[0428] However, in this case, it is necessary to store an extended 3D data generation model and a latent code corresponding to each period (each time zone), which requires a huge storage capacity for storing the data of the extended 3D data generation model in storage, or a huge network bandwidth for transmitting the data of multiple 3D data generation models over a network. Therefore, the data size may be reduced by data encoding the extended 3D data generation model corresponding to each period using, for example, Neural Network Coding (NNC) of the Moving Picture Experts Group (MPEG) standard. Furthermore, the data size may be reduced by arithmetic coding the latent code. This disclosure describes a method for more efficiently compressing this data.

[0429] With the above configuration, the information processing device may generate any viewpoint image at any time within the period from time t0 to time t5. For example, when acquiring the extended three-dimensional data generation model NNt0-2 and the latent codes Zt0 to Zt2, the information processing device generates the extended three-dimensional data generation model NNt0-2 and the latent codes Zt0 to Zt2 by machine learning based on multiple viewpoint images captured at times t0, t1, and t2 as learning data, and the camera postures and latent codes Zt0 to Zt2 corresponding to the multiple viewpoints. When generating a moving image of viewpoint A, the information processing device may generate not only viewpoint image A at times t0, t1, and t2, but also images of any viewpoint at times t0.5 and t1.5 between times t0, t1, and t2, for example. Time t0.5 is a time between times t0 and t1, and time t1.5 is a time between times t1 and t2.

[0430] This allows the information processing device to generate images from any viewpoint corresponding not only to the time corresponding to the image during learning, but also to a time shifted from the time corresponding to the image during learning, thereby enabling the generation of moving images from viewpoint A at a high frame rate.

[0431] In this case, the information processing device may calculate latent codes Zt0.5, Zt1.5, and Zt2.5 corresponding to times t0.5, t1.5, and t2.5 using already-learned latent codes. For example, the information processing device may calculate latent code Zt0.5 using latent codes Zt0 and Zt1. Specifically, the information processing device may calculate Zt0.5 based on the average value of Zt0 and Zt1. In this way, the information processing device calculates a latent code corresponding to a certain time from already-calculated latent codes, thereby eliminating the need to store latent codes for all times and reducing data size. The information processing device may also learn, for example, learning data for time t3 as learning data for the extended 3D data generation model NNt0-2 and the latent codes Zt0 to Zt2, in addition to learning data for times t0, t1, and t2. This allows the information processing device to generate viewpoint images after time t2, for example, a viewpoint image for time 2.5, with high accuracy.

[0432] Furthermore, the information processing device may perform learning using not only the learning data corresponding to times t3, t4, and t5 but also, for example, learning data corresponding to times t2 and t6 as learning data for the extended 3D data generation model NNt3-5 and the latent codes Zt3 to Zt5. This allows the information processing device to generate, with high accuracy, an image from an arbitrary viewpoint before time t3 or an image from an arbitrary viewpoint after time t5.

[0433] In the above example, when generating a viewpoint image at time 2.5, which is between times t2 and t3 when the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5 switch as the switching point of the extended three-dimensional data generation model, the information processing device may generate a viewpoint image at time t2.5 for each of the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5, and generate an average image of the two generated viewpoint images at time t2.5 as the viewpoint image at time t2.5. This allows for the generation of a highly accurate viewpoint image at time t2.5.

[0434] The information processing device may calculate the latent code Zt2.5 using the latent code Zt2 and the latent code Zt3. Specifically, the information processing device may calculate the latent code Zt2.5 based on the average value of the latent code Zt2 and the latent code Zt3. In this way, the information processing device calculates the latent code corresponding to a certain time from the latent code that has already been calculated, thereby eliminating the need to store the latent codes for all times and reducing the data size.

[0435] In this way, by specifying a latent code corresponding to a time within a period corresponding to the extended three-dimensional data generation model and viewpoint information for the extended three-dimensional data generation model, the information processing device can generate an image of the target object viewed from the specified viewpoint at the time corresponding to the specified latent code.

[0436] FIG. 58 is a diagram illustrating a first example of the configuration of an encoding device according to a first example of the third embodiment.

[0437] The encoding device 1510 includes an extended three-dimensional data generation model acquisition unit 1511 , a buffer unit 1512 , a network model encoding unit 1513 , and a latent code encoding unit 1514 .

[0438] The extended 3D data generation model acquisition unit 1511 acquires learning data for each of the periods t0 to t2 and t3 to t5, and uses the acquired learning data for each period to generate, by learning, an extended 3D data generation model NNt0-2 and latent codes Zt0 to Zt2 for the period t0 to t2, and an extended 3D data generation model NNt3-5 and latent codes Zt3 to Zt5 for the period t3 to t5. The learning data includes multiple viewpoint images obtained by photographing a target object from one or more viewpoint positions and one or more line-of-sight directions for each of the times t0 to t5, one or more pieces of viewpoint information indicating one or more viewpoint positions and one or more line-of-sight directions corresponding to the multiple viewpoint images, and latent codes Zt0 to Zt5 corresponding to each of the times t0 to t5. The one or more pieces of viewpoint information may be the position and orientation of the camera when each of the multiple viewpoint images was captured. Note that the learning data is not limited to this and may further include information obtained from other sensors. For example, the training data may include point cloud data, depth images, etc. acquired at each time using a LiDAR or TOF sensor, thereby improving the accuracy of the augmented 3D data generation model obtained by training.

[0439] The buffer unit 1512 stores the extended three-dimensional data generation model and latent codes Ztm to Ztn for the period tm-n from time tm (m is an integer) to time tn (n is an integer greater than m) generated by the extended three-dimensional data generation model acquisition unit 1511. The buffer unit 1512 is realized by a storage device such as a memory. The extended three-dimensional data generation model and latent codes Ztm to Ztn for the period tm-n stored in the buffer unit 1512 may be used, for example, as an initial model or initial latent code when the extended three-dimensional data generation model acquisition unit 1511 acquires (generates) an extended three-dimensional data generation model for a period after the period tm-n by learning. This makes it possible to shorten the learning time while improving the accuracy of the extended three-dimensional data generation model for a period after the period tm-n.

[0440] The buffer unit 1512 may store a plurality of extended three-dimensional data generation models and a plurality of latent codes corresponding to a plurality of time periods. As a result, for example, one initial model and one initial latent code may be generated by processing such as averaging based on the plurality of extended three-dimensional data generation models and the plurality of latent codes stored in the buffer unit 1512. The extended three-dimensional data generation model acquisition unit 1511 can acquire a highly accurate extended three-dimensional data generation model by learning an extended three-dimensional data generation model for a period after the period tm-n using this initial model and initial latent code.

[0441] Note that if the extended three-dimensional data generation model acquisition unit 1511 does not refer to the extended three-dimensional data generation model and latent code from the past period during learning, the encoding device 1510 does not need to include the buffer unit 1512. This allows the amount of memory used as the buffer unit 1512 to be reduced.

[0442] The network model encoding unit 1513 uses the method described in the above embodiment to encode the extended three-dimensional data generation models NNt0-2 and NNt3-5 acquired by the extended three-dimensional data generation model acquisition unit 1511 and outputs a bit stream.

[0443] Note that the data size may be reduced by encoding the input network model using, for example, the NNC of the MPEG standard as a network model encoding method. That is, the network model encoding unit 1513 encodes the extended three-dimensional data generation models NNt0-2 and NNt3-5 using NNC and adds the encoding results to a bitstream. In other words, the network model encoding unit 1513 generates encoded data as the encoding result and generates a bitstream including the encoded data.

[0444] As a network model encoding method, for example, the MPEG standard NNC (Neural Network Coding) may be used to encode the input network model, thereby reducing the data size.

[0445] The network model coding method is not limited to the extended 3D data generation model, and may be compatible with coding of other network models. This eliminates the need to provide a network model coding unit 1513 for each network model to be coded, making it possible to reduce memory requirements and circuit scale. The network model may also be simply referred to as a network.

[0446] The latent code encoding unit 1514 encodes the latent codes Zt0 to Zt5 output from the extended 3D data generation model acquisition unit 1511 and adds them to the bitstream. Note that, as a latent code encoding method, for example, when the latent code is Zt=(value0t, value1t, ..., value255t), the latent code encoding unit 1514 may binarize each value of value0t to value255t, assign a context to each bit after binarization, and perform arithmetic encoding. This can improve encoding efficiency when the values ​​of value0t to value255t are close to each other.

[0447] Furthermore, if there is no correlation between the values ​​of value0t to value255t, the latent code encoding unit 1514 may perform arithmetic encoding using a bypass mode without assigning a context to each bit after binarization. This allows the number of contexts to be reduced.

[0448] Alternatively, the latent code encoding unit 1514 may assign a context to a portion of each bit after binarization and perform arithmetic encoding on the remaining portion using a bypass mode. This can reduce the number of contexts while improving encoding efficiency.

[0449] Note that the encoding method of the latent code is not limited to this, and any encoding method may be used. For example, the latent code encoding unit 1514 may map the latent code to pixels of a two-dimensional image and encode the image using image encoding. As a more specific example, if the image codec is a codec compatible with YUV420, the latent code encoding unit 1514 may assign value0t to value255t of the latent code Zt = (value0t, value1t, ..., value255t) to 256 pixels of the Y component, and set a certain value, such as 0 or half the maximum possible value, to the other YUV pixel values, and then image-encode the YUV420 image. This allows for improved encoding efficiency by applying predictive encoding or arithmetic encoding provided in image encoding.

[0450] At this time, the latent code encoding unit 1514 may add the number of dimensions of the latent code to the bitstream. This allows the decoding device to identify which part of the Y component is the latent code value, and correctly decode the latent code. Similarly, when using an image codec compatible with YUV444 or YUV400, the latent code encoding unit 1514 may map the latent code to a part of the Y component, U component, or V component and perform image encoding. This allows the latent code to be encoded as an image regardless of the image codec used.

[0451] The latent code encoding unit 1514 may encode the latent code using lossless encoding, which allows the decoding device to restore the latent code having the same value as the encoding device.

[0452] The latent code encoding unit 1514 may add information indicating whether the latent code has been encoded by arithmetic coding or by using an image codec to the bitstream. This allows the decoding device to switch between arithmetic decoding and image decoding of the latent code according to the value added to the bitstream, thereby enabling the bitstream to be decoded appropriately.

[0453] The latent code encoding unit 1514 may apply predictive coding to the encoding of the latent code. For example, when the latent code is a multidimensional vector such as Zt = (value0t, value1t, ..., value255t), predictive coding may be applied between the components of the vector. More specifically, instead of directly encoding the value of value1t, the latent code encoding unit 1514 may encode the difference value obtained by subtracting value0t from value1t. Similarly, instead of directly encoding the value of value2t, the latent code encoding unit 1514 may encode the difference value obtained by subtracting value1t from value2t. In other words, when encoding the value of a certain component A, the latent code encoding unit 1514 may calculate a predicted value P of component A using the values ​​of one or more components encoded or decoded before component A, and then encode the value obtained by subtracting the predicted value P from the value of component A. This improves encoding efficiency when the correlation between components in the latent code is high.

[0454] The predicted value P may be calculated from the average value, minimum value, or maximum value of one or more components encoded or decoded before the component A. This can improve the accuracy of the predicted value P. In this way, the amount of coding may be reduced by sequentially encoding the latent codes for each time period using the latent code encoding unit 1514 and adding each encoding result to the bitstream in which the network model is encoded.

[0455] In this case, the encoding device 1510 may add time information indicating to which time period the encoded extended three-dimensional data generation model or latent code corresponds as metadata to the bitstream. This allows the decoding device to know to which time period the decoded extended three-dimensional data generation model or latent code corresponds by decoding and referring to the metadata, and to appropriately generate a moving image from an arbitrary viewpoint of the target object.

[0456] FIG. 59 is a diagram illustrating a first example of the configuration of a decoding device according to a first example of the third embodiment.

[0457] The decoding device 1515 includes a network model decoding unit 1516 , a latent code decoding unit 1517 , and a rendering unit 1518 .

[0458] The network model decoding unit 1516 acquires the bit stream and, based on the acquired bit stream, decodes the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 and the extended three-dimensional data generation model NNt3-5 for the period t3 to t5, as well as metadata such as time information corresponding to these extended three-dimensional data generation models NNt0-2 and NNt3-5.

[0459] The latent code decoding unit 1517 decodes information related to the latent code from the input bit stream and decodes latent codes Zt0 to Zt5 for periods t0 to t2 and t3 to t5. The decoding method for the latent code may be switched to match the encoding method of the encoding device 1510. For example, if the encoding device 1510 arithmetically encodes the latent code, the decoding device 1515 may decode the latent code using arithmetic decoding, and if the encoding device 1510 image-encodes the latent code, the decoding device 1515 may decode the latent code using image decoding.

[0460] The latent code decoding unit 1517 may switch the decoding method of the latent code using information added to the bitstream, thereby allowing the decoding device 1515 to appropriately decode the bitstream.

[0461] Note that, when predictive coding is applied to the coding of the latent code in the coding device 1510, the decoding device 1515 may apply predictive decoding. For example, when the latent code is a multidimensional vector such as Zt = (value0t, value1t, ..., value255t), the decoding device 1515 may apply predictive decoding between vector components. More specifically, for example, when the coding device 1510 arithmetically codes a difference value obtained by subtracting value0t from value1t, the decoding device 1515 may arithmetically decode the difference value and add the decoded value0t to decode value1t.

[0462] Similarly, if the encoding device 1510 arithmetically encodes the difference value obtained by subtracting value1t from value2t, the decoding device 1515 may arithmetically decode the difference value and add the decoded value1t to decode value2t. That is, when encoding a value of a certain component A, the encoding device 1510 may calculate a predicted value P of component A using the values ​​of one or more components encoded or decoded before component A, and arithmetically encode the difference value obtained by subtracting the predicted value P from the value of component A. The decoding device 1515 may arithmetically decode the difference value, calculate the predicted value P in the same manner as the encoding device 1510, and decode the value of component A by adding the decoded difference value and the predicted value P. This allows the decoding device 1515 to appropriately decode a bitstream with improved coding efficiency when the correlation between components in the latent code is high. The predicted value P may also be calculated from the average value, minimum value, or maximum value of one or more components decoded before component A. This improves the accuracy of the predicted value P.

[0463] The rendering unit 1518 generates a moving image of viewpoint A at time t based on the latent code Zt at time t specified by the user or the system, etc., and viewpoint information of viewpoint A, using the extended three-dimensional data generation models NNt0-2, NNt3-5 decoded by the network model decoding unit 1516 and metadata such as time information. Specifically, the rendering unit 1518 inputs the viewpoint information of viewpoint A and the latent codes Zt0, Zt1, and Zt2 corresponding to the times within the period t0 to t2 into the extended three-dimensional data generation model NNt0-2 for the period t0 to t2, and generates an image IMGt0 of viewpoint A at time t0, an image IMGt1 of viewpoint A at time t1, and an image IMGt2 of viewpoint A at time t2. The rendering unit 1518 applies the image generation process for the period t0 to t2 to the extended 3D data generation model NNt3-5 for the period t3 to t5 to generate images IMGt3 to IMGt5 from the viewpoint A at times t3 to t5. Then, the rendering unit 1518 generates a video of the target object from times t0 to t5 as viewed from the viewpoint A, using the images IMGt0 to IMGt5 and metadata such as time information. The video may include, for example, the images IMGt0 to IMGt5 and presentation time information for calculating the presentation times of the images IMGt0 to IMGt5 based on the times t0 to t5.

[0464] Note that the viewpoint information may be changed depending on the time. For example, viewpoint information of viewpoint A and latent codes Zt0 to Zt2 may be input to the extended three-dimensional data generation model NNt0-2 for the period t0 to t2, and viewpoint information of viewpoint B and latent codes Zt3 to Zt5 may be input to the extended three-dimensional data generation model NNt3-5 for the period t3 to t5. As a result, the rendering unit 1518 generates multiple images of the target object viewed from viewpoint A from time t0 to t2, and multiple images of the target object viewed from viewpoint B from time t3 to t5. In other words, the rendering unit 1518 can generate a moving image of the target object in which the viewpoint switches from viewpoint A to viewpoint B at time t3.

[0465] The rendering unit 1518 does not necessarily have to generate a moving image, but may generate a still image of a specified viewpoint at a specified time, thereby allowing the user to switch between generating a moving image or a still image depending on the application.

[0466] Note that the rendering unit 1518 is not limited to generating moving images or still images from the extended 3D data generation model. For example, the rendering unit 1518 may generate point cloud data or mesh data from the extended 3D data generation model for a period that the extended 3D data generation model can represent, and output the generated point cloud data or mesh data as dynamic point cloud data or dynamic mesh data. This allows the user to view dynamic 3D data of a dynamic target object on a head-mounted display (HMD) or the like, and also allows the user to measure the amount of movement of the target object using the dynamic 3D data.

[0467] FIG. 60 is a diagram illustrating a second example of the configuration of the encoding device according to the first example of the third embodiment.

[0468] The encoding device 1520 includes an extended three-dimensional data generation model acquisition unit 1521 , a buffer unit 1522 , a difference calculation unit 1523 , a network model encoding unit 1524 , and a latent code encoding unit 1525 .

[0469] The extended three-dimensional data generation model acquisition unit 1521 is the same as the extended three-dimensional data generation model acquisition unit 1511 of the encoding device 1510 .

[0470] The buffer unit 1522 is similar to the buffer unit 1512 of the encoding device 1510, but differs from the buffer unit 1512 in that it inputs an extended three-dimensional data generation model stored in a memory or the like to the difference calculation unit 1523 as a reference extended three-dimensional data generation model, and inputs a latent code stored in a memory or the like to the difference calculation unit 1523 as a reference latent code.

[0471] The difference calculation unit 1523 calculates difference information indicating the difference between the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 and the extended three-dimensional data generation model NNt3-5 for the period t3 to t5 generated by the extended three-dimensional data generation model acquisition unit 1521, and the extended three-dimensional data generation model (hereinafter referred to as the reference extended three-dimensional data generation model) generated by the extended three-dimensional data generation model acquisition unit 1521 before each period. Here, the difference information may include, for example, differences in weight parameters at the nodes of each network model. For example, the difference calculation unit 1523 acquires the extended three-dimensional data generation model NNt3-5 for the period t3 to t5 from the extended three-dimensional data generation model acquisition unit 1521, and acquires the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 from the buffer unit 1522 as the reference extended three-dimensional data generation model.

[0472] Furthermore, the difference calculation unit 1523 may use the extended three-dimensional data generation model NNt3-5 and the extended three-dimensional data generation model NNt0-2 to calculate, for example, the difference (amount of change) between the weight parameters of the nodes of the network model in the extended three-dimensional data generation model NNt3-5 and the weight parameters of the nodes of the network model in the extended three-dimensional data generation model NNt0-2, and input difference information indicating this difference to the network model encoding unit 1524. The difference information is then encoded by the network model encoding unit 1524. In other words, the encoding device 1520 may reduce the amount of data by predicting information related to the network model in the extended three-dimensional data generation model NNt3-5 from the extended three-dimensional data generation model NNt0-2 and encoding the difference from the predicted value. Such predictive encoding reduces the value of the difference to be encoded when there is little change in the extended three-dimensional data generation model over time, such as when the target object is almost stationary, thereby improving encoding efficiency. For example, the encoding device 1520 may set RNNt0-2=0 and RNNt3-5=NNt0-2, and reduce the amount of bits by predictive encoding using the extended 3D data generation model of the previous time period as the reference extended 3D data generation model.

[0473] In addition, the difference calculation unit 1523 calculates difference information (hereinafter referred to as differential latent codes) between the latent codes Zt0 to Zt5 for the periods t0 to t2 and t3-5 generated by the extended three-dimensional data generation model acquisition unit 1521 and the latent codes (hereinafter referred to as reference latent codes) generated before each period.

[0474] Here, the differential latent code is, for example, a vector difference value (amount of change) Zt1-Zt0 of the latent code calculated using the latent code Zt1 at time t1 output from the extended 3D data generation model acquisition unit 1521 and the latent code Zt0 at time t0 output from the buffer unit 1512 as the reference latent code. The difference calculation unit 1523 may output the differential latent code to the latent code encoding unit 1525. The differential latent code is then encoded by the latent code encoding unit 1525. In this manner, the encoding device 1520 may reduce the amount of data by predicting the latent code Ztn at time n from the latent code Ztm at time m to calculate a predicted value, and encoding the latent code Ztn at time n and the differential latent code (difference vector) that is the difference between the predicted value and the latent code Ztn. This predictive encoding reduces the value of the difference to be encoded when there is little change in the latent code between times, such as when the target object is almost stationary, thereby improving encoding efficiency.

[0475] Although the encoding device 1520 in the first example predictively encodes the latent code Ztn at time n from the latent code Ztm at time m, this is not necessarily limited to this. For example, the encoding device 1520 may select a reference latent code to be used for prediction from one or more reference latent codes stored in the buffer unit 1522 and perform predictive encoding using the selected reference latent code. In this case, the encoding device 1520 may add information indicating the selected reference latent code (reference latent code information) to the bitstream in order to communicate the selected reference latent code to the decoding device. This allows the encoding device 1520 to select an optimal reference latent code in terms of coding efficiency, thereby improving coding efficiency. Furthermore, the decoding device can appropriately decode a bitstream with improved coding efficiency by decoding the reference latent code.

[0476] When the encoding device 1520 performs predictive encoding by referring to two or more latent codes stored in the buffer unit 1522, the encoding device 1520 may add information indicating two or more reference latent codes to the bitstream. This allows the encoding device 1520 to improve the encoding efficiency of predictive encoding using two or more reference latent codes. Furthermore, the decoding device can appropriately decode the bitstream with improved encoding efficiency.

[0477] In addition, in a case where a reference latent code is not stored in the buffer unit 1522, for example, when encoding the first latent code (first frame) in data order, the encoding device 1520 may encode the latent code to be processed without prediction, without calculating the difference from a predicted value (hereinafter referred to as intra prediction), or may encode by calculating the difference from a predicted value set to 0. Furthermore, when the encoding device 1520 sets a certain period tm-n as a random access point, it may encode the latent code corresponding to the period tm-n using intra prediction, or may encode by calculating the difference from a predicted value set to 0. This allows the decoding device to start decoding the latent code from the first latent code (first frame) in data order or from the random access point, thereby improving functionality during playback.

[0478] A set of multiple latent codes (multiple frames) (hereinafter referred to as a GOF (Group of Frame)) may be defined, and the first frame of the GOF may be coded using intra-prediction, which allows a decoding device to randomly access the first frame of the GOF. Furthermore, decoding the first frame of the GOF can enhance functionality such as fast-forward playback.

[0479] The encoding device 1520 may also add permission information to the bitstream indicating whether or not inter-GOF predictive reference is permitted. For example, if the bitstream includes permission information indicating that inter-GOF predictive reference is prohibited, the decoding device can determine that multiple GOFs can be decoded in parallel. Furthermore, for example, allowing inter-GOF predictive reference can improve encoding efficiency.

[0480] The network model encoding unit 1524 is similar to the network model encoding unit 1513 of the encoding device 1510, but differs in that it encodes the difference information d0-2 and d3-5 of the extended three-dimensional data generation models NNt0-2 and NNt3-5 input from the difference calculation unit 1523 and outputs a bit stream.

[0481] The latent code encoding unit 1525 has the same function as the latent code encoding unit 1514, but differs in that it encodes the differential latent code input from the difference calculation unit 1523 and outputs a bit stream.

[0482] Although the encoding device 1520 has been described with the difference calculation unit 1523, the network model encoding unit 1524, and the latent code encoding unit 1525 separated from each other, this is not necessarily limited thereto, and for example, the difference calculation unit 1523 may be included in the network model encoding unit 1524 or the latent code encoding unit 1525. In other words, the network model encoding unit 1524 or the latent code encoding unit 1525 may perform the processing of the difference calculation unit 1523.

[0483] Although the difference calculation unit 1523 calculates the difference value of the extended 3D data generation model and the difference value of the latent code in the above embodiment, this is not necessarily limited to this. For example, the encoding device may be provided with a network model difference calculation unit that calculates the difference value of the network model and a latent code difference calculation unit that calculates the difference value of the latent code. This allows the difference calculation to be performed in parallel, thereby speeding up the processing.

[0484] The encoding device 1520 may add predictive coding information to the bitstream, which indicates whether the latent code has been encoded by intra prediction or by predictive coding using a reference latent code (hereinafter referred to as inter prediction). This allows the decoding device to appropriately determine whether to decode the latent code using intra prediction or inter prediction by decoding the predictive coding information.

[0485] FIG. 61 is a diagram illustrating a second example of the configuration of the decoding device according to the first embodiment of the present invention.

[0486] The decoding device 1530 includes a network model decoding unit 1531 , a latent code decoding unit 1532 , an adding unit 1533 , a buffer unit 1534 , and a rendering unit 1535 .

[0487] The network model decoding unit 1531 acquires a bit stream, decodes information related to the network model based on the acquired bit stream, and decodes differential information d0-2, d3-5 of the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5 for the period t0 to t2, as well as metadata such as time information.

[0488] The latent code decoding unit 1532 decodes information related to the latent code from the input bit stream and decodes differential latent codes Ztd0 to Ztd5 corresponding to the period from time t0 to t2 and the period from time t3 to t5. The decoding method for the differential latent code may be switched to match the encoding method of the encoding device 1520. For example, if the encoding device 1520 arithmetically encodes the differential latent code, the latent code decoding unit 1532 may decode the differential latent code using arithmetic decoding. If the encoding device 1520 image-encodes the differential latent code, the latent code decoding unit 1532 may decode the differential latent code using image decoding. The latent code decoding unit 1532 may switch the decoding method for the differential latent code using information added to the bit stream. This allows the bit stream to be decoded appropriately.

[0489] The adder 1533 adds, for the corresponding periods, difference information d0-2, d3-5 of the extended three-dimensional data generation models NNt0-2, NNt3-5 decoded by the network model decoding unit 1531 and corresponding to the periods t0 to t2, t3 to t5, to the reference extended three-dimensional data generation models RNNt0-2, RNNt3-5 acquired from the buffer unit 1534, to calculate the extended three-dimensional data generation models NNt0-2, NNt3-5. In this way, the decoding device 1530 may set RNNt0-2 = 0, RNNt3-5 = NNt0-2, and perform predictive decoding on the extended three-dimensional data generation model of the previous time period as the reference extended three-dimensional data generation model.

[0490] In addition, the adder 1533 adds the differential latent codes Ztd0 to Ztd5 corresponding to times t0 to t5 decoded by the latent code decoder 1532 to the reference latent codes RZt0 to RZt5 input from the buffer unit 1534 at each corresponding time to calculate the latent codes Zt0 to Zt5. This allows the addition to be performed in parallel, thereby speeding up the process.

[0491] Note that, in the decoding device 1530 of the second example, the adding unit 1533 is described as being separate from the network model decoding unit 1531 and the latent code decoding unit 1532, but this is not necessarily limited to this, and for example, the adding unit 1533 may be included in the network model decoding unit 1531 or the latent code decoding unit 1532. In other words, the network model decoding unit 1531 may perform the processing of the adding unit 1533, or the latent code decoding unit 1532 may perform the processing of the adding unit 1533.

[0492] In addition, in a case where a reference latent code is not stored in the buffer unit 1534, for example, when decoding the first latent code (first frame) in data order, the decoding device 1530 may perform decoding without prediction (hereinafter referred to as intra prediction) without the adder 1533 adding the difference information and the latent code, or may perform decoding by adding a predicted value set to 0 and the difference information. Furthermore, when the decoding device 1530 sets a certain period tm-n as a random access point, it may decode the latent code corresponding to the period tm-n using intra prediction, or may decode by adding a predicted value set to 0 and the difference information. Furthermore, when predictive coding information indicating that the latent code to be decoded has been coded using intra prediction is included in the bitstream, the latent code may be decoded using intra prediction, or may be decoded by adding a predicted value set to 0 and the difference information. This allows the decoding device 1530 to start decoding the latent code from the first latent code (first frame) in data order, the random access point, or a latent code coded using intra prediction, thereby improving functionality during playback.

[0493] Although the decoding device 1530 in the second example predictively decodes the latent code corresponding to time n from the latent code at time m, this is not necessarily limited to this. For example, the decoding device 1530 may select a reference latent code to be used for prediction from one or more latent codes stored in the buffer unit 1534 and perform predictive decoding using the selected latent code. In this case, the decoding device 1530 may decode information indicating the selected latent code (reference latent code) from the bitstream. This allows the decoding device 1530 to appropriately decode a bitstream with improved coding efficiency by decoding the reference latent code from the bitstream generated by the encoding device 1460, which is optimal in terms of coding efficiency.

[0494] When performing predictive decoding by referring to two or more latent codes stored in the buffer unit 1534, the decoding device 1530 may decode information indicating two or more reference latent codes from the bitstream. This allows the decoding device 1530 to appropriately decode a bitstream in which the coding efficiency of predictive coding is improved by using two or more reference latent codes.

[0495] The rendering unit 1535 is similar to the rendering unit 1518 of the decoding device 1515. The rendering unit 1535 does not necessarily need to generate a moving image, but may generate a still image of specified viewpoint information at a specified time.

[0496] [Variation of First Embodiment] When transmitting the encoded extended 3D data generation model or latent code bit stream via a network, the encoding device may transmit the extended 3D data generation model bit stream and the latent code bit stream separately. For example, if a destination device generates rendering images from time t0 to t9, the encoding device may first transmit the extended 3D data generation model NNt0-9 and the latent code Zt0 related to time t0 so as to be ready for the rendering image generation at time t0. The encoding device may then transmit the latent codes Zt1 to Zt9 in order so as to be ready for the rendering image generation at time t1 to t9 at the receiving device (destination device, e.g., decoding device). This reduces the load on the network transmission capacity and allows the receiving device to generate rendering images by the desired time. The encoding device may transmit the extended 3D data generation model and the latent code related to that time at each random access point, and transmit the latent code at times other than the random access points. This reduces the overall code volume while appropriately inserting random access points.

[0497] In addition, when learning the extended 3D data generation model and the latent code, the information processing device may learn a time sampling interval (time interval) that is short, and the encoding device may transmit a bit stream including a latent code whose sampling interval is set longer than that during learning. For example, during learning, the sampling interval of the latent code may be set in increments of 0.1 seconds, and the sampling interval of the latent code included in the encoded bit stream may be set in increments of 1 second. In this case, the receiving device may use the decoded latent code every second to calculate a latent code every 0.1 seconds using an interpolation process or a filter process. This makes it possible to generate a high-precision extended 3D data generation model and latent code through learning while reducing the amount of code in the generated bit stream.

[0498] The information processing device may assign one or more common latent codes to two or more extended three-dimensional data generation models for learning. For example, the information processing device may simultaneously learn using one latent code common to the extended three-dimensional data generation models for each of space A and space B, thereby generating an extended three-dimensional data generation model for space A, an extended three-dimensional data generation model for space B, and one latent code. In this case, as an example of the operation of the receiving device during rendering, when simultaneously generating rendering images A and B of space A and space B viewed from viewpoint V at time t, the receiving device may acquire image A from the extended three-dimensional data generation model for space A, viewpoint V information, and a latent code related to time t, and acquire image B from the extended three-dimensional data generation model for space B, viewpoint V information, and a latent code related to time t. In this way, by assigning one or more common latent codes to two or more extended three-dimensional data generation models, rendering images of multiple spaces can be generated while reducing the amount of data.

[0499] The information processing device may learn model parameters so that the augmented three-dimensional data generation model can generate multiple attribute information, such as color or reflectance, from a single model, in addition to information related to the shape of the three-dimensional space. This reduces the amount of data in the generated augmented three-dimensional data generation model. Furthermore, the augmented three-dimensional data generation model may hold a separate three-dimensional data generation model for each attribute. Specifically, model parameters may be learned and held separately for each attribute. This makes it possible to obtain an augmented three-dimensional data generation model that is optimal for the attribute information.

[0500] The metadata transmitted to the receiving side may include, for example, information indicating which of one or more viewpoint information is likely to improve the quality of the rendered image. This allows the receiving device to generate high-quality video by selecting viewpoint information that provides a high-quality rendered image at each time point. The metadata may also include information indicating which of one or more latent codes is likely to improve the quality of the rendered image. This allows the receiving device to generate high-quality video by selecting a time point associated with a latent code that provides a high-quality rendered image. The receiving device may generate high-quality video by generating rendered images using latent codes encoded and transmitted as a bitstream. The metadata may also include information indicating which attributes, such as RGB, reflectance, and transparency, the augmented 3D data generation model can output. This allows the receiving device to switch the augmented 3D data generation model to use depending on the application.

[0501] [Supplementary Information] A latent code is assigned to, for example, one image (or object or scene) at a time instant and indicates the characteristics of that image. The latent code can indicate various characteristics, such as appearance, shape, and lighting. As described above, a latent code is an n-dimensional (n is an integer greater than or equal to 2) value and may be expressed as a multidimensional vector. Multidimensional vectors include low-dimensional vectors (2-3 dimensions). A latent code is a concept in machine learning or deep learning, and is information that enables a model to learn the features or patterns of input data and achieve a compact representation. A latent code is used to compress and represent scene movement and appearance changes. Using automatically learned latent information as a code instead of time allows for more expressive recording of geometry or texture movement. Using this latent code allows for smooth representation of temporal changes, which can be useful for creating visual effects. Note that a latent code may also be a one-dimensional value indicating time.

[0502] Times t0, t1, t2, t3, t4, and t5 are consecutive times spaced at equal intervals. For example, if the interval between each time is 1 second and t0 is 0 seconds, then t1 will be 1 second later, t2 will be 2 seconds later, and so on. This is just an example, and the interval between each time may be any value. The interval between each time may be specified as an integer or a decimal. In other words, time tn can also take on a decimal value.

[0503] Although examples of the encoding device or decoding device in one aspect of the present application have been shown for intra-coding or intra-decoding and inter-coding or inter-decoding, respectively, the encoding device or decoding device may be configured to be able to perform either intra-coding or intra-decoding or inter-coding or inter-decoding, and may be used by switching between them. This switching may be applied to encoding or decoding of an extended 3D data generation model, or to encoding or decoding of a latent code. This makes it possible to realize a combination that can further reduce the amount of code.

[0504] The encoding device according to this embodiment may include a circuit and a memory connected to the circuit, and the circuit may use the memory to (1) encode 3D data generation model information capable of generating an image corresponding to a time within a first range from an arbitrary viewpoint into a bitstream, and (2) encode a latent code into the bitstream. This reduces the data size by arithmetically encoding the latent code. Furthermore, a decoding device receiving the bitstream can generate a video from the 3D data generation model.

[0505] The decoding device in this embodiment may be an apparatus including a circuit and a memory connected to the circuit, wherein the circuit uses the memory to acquire (1) 3D data generation model information capable of generating an image corresponding to a time within a first range from an arbitrary viewpoint and (2) a latent code from a bitstream, and generates an image within the first range using the 3D data generation model and the latent code. This allows for generating a moving image from the 3D data generation model.

[0506] The three-dimensional data generation model may be (1) a first network corresponding to a time within a first range, or (2) differential information between the first network and a second network corresponding to a time within a second range prior to the first range.

[0507] The extended three-dimensional data generation model may be generated by inputting first viewpoint information, the latent code at a first time, and an image at the first viewpoint at the first time into a neural network.

[0508] The extended three-dimensional data generation model may include a first model and a second model, and the first model (NNt0-2) may be capable of generating an image corresponding to a time within a range from a first time t0 to a second time t2, and the second model (NNt3-5) may be capable of generating an image corresponding to a time within a range from a third time t3 to a fourth time t5.

[0509] [Example 2] A three-dimensional model generated using a basic NeRF may be composed of multiple networks. The term "network" here refers to a learning model obtained by learning using a neural network. The multiple networks may include, for example, a network trained using sparse sampling points and a network trained using dense sampling points. In this way, the multiple networks are networks that differ in the number of input sampling points or the density of the sampling points. The sampling points may be, for example, three-dimensional points indicating three-dimensional positions.

[0510] Alternatively, the multiple networks may include, for example, a network for outputting geometry information indicating the density of an object, the probability of the object's existence, the coordinates of the geometry, etc., and a network for outputting information (attribute information) associated with the geometry, such as color information, reflectance, normal vector, color coordinates, timestamp, and object ID, based on the geometry information. The multiple networks may include two or more networks. The multiple networks may be three or more networks with different sampling points, or may include two or more networks for outputting geometry information or two or more networks for outputting attribute information.

[0511] The multiple networks may be coded using multiple network coding units, which may code the multiple networks with existing network coding means, such as Neural Network Coding (NNC) of the MPEG standard.

[0512] The information processing device may also apply the encoding method or decoding method described in this embodiment to an extended 3D data generation model that can generate still images viewed from any viewpoint at any time by adding a latent code to a 3D data generation model configured from two or more networks as described above and learning the image, and then generate a moving image by arranging the still images in chronological order. This makes it possible to reduce the amount of code required for the extended 3D data generation model configured from two or more networks and the latent code.

[0513] Next, using NeRF, which is one of the three-dimensional modeling methods, as an example, a method of generating an extended three-dimensional data generation model and a latent code from a plurality of two-dimensional images and encoding the generated extended three-dimensional data generation model (network model) and latent code will be described with reference to Fig. 62. Note that the method described here is just an example, and is not limited to the NeRF method described here, but can also be applied to other NeRF methods or three-dimensional modeling methods.

[0514] FIG. 62 is a block diagram illustrating an example of the configuration of a coding device that codes a plurality of networks in Example 2 of Embodiment 3.

[0515] The encoding device 1540 includes an extended three-dimensional data generation model learning unit 1541, a three-dimensional data generation model encoding unit 1545, and a latent code encoding unit 1549. The encoding device 1540 may further include a bit stream data configuration unit 1552.

[0516] First, we will explain the specific configuration of the extended 3D data generation model learning unit 1541. Specifically, the extended 3D data generation model learning unit 1541 has a first network learning unit 1542, a sampling point determination unit 1543, and a second network learning unit 1544.

[0517] The first network learning unit 1542 uses the input multiple 2D images, the viewpoint information (camera posture) for each of the input multiple 2D images, and the input latent code to learn a 3D data generation model for each viewpoint and time. That is, the first network learning unit 1542 learns the 2D images associated with each viewpoint based on the multiple 2D images and the viewpoint information for each 2D image, thereby generating a 3D data generation model (first network) and a learned latent code (first latent code). The viewpoint information includes the viewpoint when the 2D image was captured and a line-of-sight vector (line-of-sight direction) from the viewpoint. The first network learning unit 1542 may receive input sampling points (first sampling points) for learning. The first sampling points may be, for example, a set of points with large (wide) spacing between points. The coordinates of each point included in the first sampling points may be predetermined coordinates or may be coordinates calculated using a predetermined method. The first network learning unit 1542 outputs a three-dimensional data generation model (first network) and a latent code (first latent code) obtained by learning. The three-dimensional data generation model (first network) generated by the first network learning unit 1542 is a network that outputs density information for sampling points at a time (or time period) represented by the latent code. The first network learning unit 1542 may also output density information for sampling points at a certain time (or time period) obtained during learning.

[0518] Here, density information is information indicating the density of objects at sampling points at a certain time (or time period). For example, density information is set to be high (i.e., higher than a predetermined value) if the object is a person or a desk, low (i.e., lower than a predetermined value) if the object is a light-transmitting object such as glass, and close to 0 if there is no object. Therefore, density information can also be said to be information indicating whether or not an object exists, or information indicating the probability of the object's existence. Density information can also be said to be geometry information.

[0519] The sampling point determination unit 1543 determines the density of objects at a certain time (or time period) at the coordinates indicated by the coarsely sampled first sampling points based on the density information for the first sampling points output from the first network learning unit 1542, and determines second sampling points to be used for learning in the second network learning unit 1544. In determining the second sampling points, for example, if the density of the sampling points is greater than a predetermined density, the sampling point determination unit 1543 may determine that an object exists and may determine to perform more detailed sampling of the periphery of the object whose existence has been determined (the space in which the object is determined to exist and the space surrounding it). In determining the second sampling points, for example, if the density of the sampling points is less than a predetermined density, the sampling point determination unit 1543 may determine that no object exists in the space where the sampling points are located and may determine to perform more coarse sampling in the space or not to perform sampling. The sampling point determination unit 1543 may use, for example, a PDF sampler.

[0520] The meaning of the sampling points output by the sampling point determination unit 1543 varies depending on the density determination method. For example, sampling points output after determining that an object exists can also be considered geometry information indicating the coordinates of the object. Furthermore, depending on the density of the sampling points, it is possible to distinguish between highly transparent objects such as glass and hard materials, and it is possible to remove sampling points in objects (spaces) determined to be highly transparent or hard materials from the sampling points to be extracted. In this way, the sampling points to be extracted can be determined based on the density of the sampling points. Therefore, by extracting sampling points of objects or materials with a specific density that meets specific conditions, the extracted sampling points can be determined as geometry information.

[0521] The sampling point determination unit 1543 may determine sampling points using a predetermined method or parameters, or may use a method or parameters selected from multiple methods or multiple parameters. In this case, information indicating the predetermined method or parameters may be coded as training metadata and stored in the bitstream. As a result, the information indicating the predetermined method or parameters may be notified to the decoding device as training metadata included in the bitstream.

[0522] The second network learning unit 1544 has the same configuration as the first network learning unit 1542. The second network learning unit 1544 learns a three-dimensional model for each viewpoint at a certain time (or time period) using multiple input two-dimensional images, viewpoint information (camera posture) for each of the multiple input two-dimensional images, and second sampling points (detailed sampling points) and initial latent codes output from the sampling point determination unit 1543. That is, the second network learning unit 1544 generates a three-dimensional model (second network) by learning two-dimensional images associated with each viewpoint based on the multiple two-dimensional images, viewpoint information for each two-dimensional image, the second sampling points, and the initial latent code. The second network learning unit 1544 outputs a three-dimensional data generation model (second network) and latent code (second latent code) obtained by learning.

[0523] Here, the second network generated by the second network learning unit 1544 is a network capable of outputting color information and density information. Note that the second network learning unit 1544 may output density information and color information for sampling points obtained during learning, and use the density information and color information for other processing.

[0524] Next, a specific configuration of the three-dimensional data generation model encoding unit 1545 will be described.

[0525] The three-dimensional data generation model encoding unit 1545 includes a first network encoding unit 1546 , a second network encoding unit 1547 , and a metadata encoding unit 1548 .

[0526] The first network coding unit 1546 codes the trained first network generated by the first network training unit 1542. The first network coding unit 1546 outputs coded data obtained by coding the first network.

[0527] The second network coding unit 1547 codes the trained second network generated by the second network training unit 1544. The second network coding unit 1547 outputs coded data obtained by coding the second network.

[0528] The metadata encoding unit 1548 encodes the metadata generated by the sampling point determination unit 1543. The metadata encoding unit 1548 outputs encoded data in which the metadata is encoded.

[0529] In this way, the three-dimensional data generation model encoding unit 1545 generates encoded data in which the first network, the second network, and the metadata are each encoded, and outputs the generated encoded data.

[0530] The three-dimensional data generation model encoding unit 1545 may perform encoding using an existing network encoding method, such as MPEG standard NNC (Neural Network Coding). The trained first network and second network include multiple layers, including an input layer, an intermediate layer, and an output layer of the network, nodes in each layer, weighting coefficients for each node, and transformation functions for each node. Each of the trained first network and second network may include a density network for outputting density information, a color network for outputting color information, a reflectance network for outputting reflectance information, etc. Alternatively, each of the trained first network and second network may include an attribute network for outputting attribute information, such as color information or reflectance information.

[0531] FIG. 63 is a diagram showing an example of coded data of a trained first network in the third embodiment.

[0532] When the trained first network is a network for generating sampling points, it may include at least a density network for outputting density information, and may further include a color network for outputting color information or an attribute network (reflectance network) for outputting other attribute information (e.g., reflectance information).

[0533] FIG. 64 is a diagram showing an example of coded data of the trained second network in the third embodiment.

[0534] The trained second network may be a network for outputting color information or other attribute information for the sampling points. The trained second network may include a density network for outputting density information and a color network for outputting color information. The trained second network may further include an attribute network (reflectance network) for outputting other attribute information (e.g., reflectance information).

[0535] In addition, if the output of attribute information is not required, the trained first network or second network may not include a color network for outputting color information or an attribute network (reflectance network) for outputting other attribute information (e.g., reflectance information).

[0536] The three-dimensional data generation model encoding unit 1545 may generate a predicted value of the second network from the first network and encode the value (prediction residual) obtained by subtracting the predicted value from the second network. This reduces the amount of coding for the second network. For example, the first network may be used as the predicted value. In this case, the value obtained by subtracting the value of the first network from the value of the second network is encoded as the prediction residual for the second network. This reduces the amount of coding when there is a high correlation between the first network and the second network.

[0537] The three-dimensional data generation model encoding unit 1545 may generate a predicted value of the first network from the second network and encode the value (prediction residual) obtained by subtracting the predicted value from the first network. This reduces the amount of coding for the first network. For example, the second network may be used as the predicted value. In this case, the value obtained by subtracting the value of the second network from the value of the first network is encoded as the prediction residual of the first network. This reduces the amount of coding when there is a high correlation between the first network and the second network.

[0538] The latent code encoding unit 1549 has a first latent code encoding unit 1550 and a second latent code encoding unit 1551. The first latent code encoding unit 1550 encodes the first latent code input from the extended three-dimensional data generation model learning unit 1541 and outputs the encoded first latent code to the bit stream data configuration unit 1552. Similarly, the second latent code encoding unit 1551 encodes the second latent code input from the extended three-dimensional data generation model learning unit 1541 and outputs the encoded second latent code to the bit stream data configuration unit 1552.

[0539] As the encoding method of the latent code, the encoding method described with reference to Figures 58 and 60 may be applied, which can reduce the amount of code for the latent code.

[0540] The encoding results of the latent codes may be added to the encoded data of the trained network. For example, the encoding results of the first latent code may be added to the encoded data of the trained first network, and the encoding results of the second latent code may be added to the encoded data of the trained second network. This makes it easier to link the trained networks with their corresponding latent codes and makes management easier.

[0541] The first latent code and the second latent code may be sequentially encoded by a common latent code encoding unit, which eliminates the need for multiple latent code encoding units and reduces the memory size and circuit scale.

[0542] The first latent code that has already been learned may be set as the initial latent code of the second latent code, thereby shortening the learning time of the second latent code.

[0543] The encoding device 1540 may simultaneously learn the first network, the second network, and one latent code to generate a common latent code, thereby eliminating the need to encode multiple latent codes and reducing the amount of coding.

[0544] The initial latent codes of the first latent code and the second latent code may have different values ​​as their initial values, allowing efficient learning of each latent code by setting appropriate initial latent codes for each.

[0545] Alternatively, the latent code encoding unit 1549 may generate a predicted value of the second latent code from the first latent code and encode the value (prediction residual) obtained by subtracting the predicted value from the second latent code. This reduces the amount of coding for the second latent code. For example, the first latent code may be used as the predicted value. In this case, the value obtained by subtracting the value of the first latent code from the second latent code is encoded as the prediction residual for the second latent code. This reduces the amount of coding when the correlation between the first latent code and the second latent code is high.

[0546] The latent code encoding unit 1549 may generate a predicted value of the first latent code from the second latent code and encode the value (prediction residual) obtained by subtracting the predicted value from the first latent code. This reduces the amount of coding for the first latent code. For example, the second latent code may be used as the predicted value. In this case, the value obtained by subtracting the value of the second latent code from the first latent code is encoded as the prediction residual for the first latent code. This reduces the amount of coding when the correlation between the first latent code and the second latent code is high.

[0547] Next, a decoding device 1560 for decoding a plurality of networks will be described. Fig. 65 is a block diagram showing an example of the configuration of a decoding device for decoding a plurality of networks according to the third embodiment.

[0548] The decoding device 1560 includes a bit stream data dividing unit 1561 , a three-dimensional data generation model decoding unit 1562 , a reconstruction unit 1566 , and a latent code decoding unit 1571 .

[0549] The bit stream data division unit 1561 divides the input bit stream into coded data for the first network, the second network, and metadata.

[0550] Next, we will explain the specific configuration of the three-dimensional data generation model decoding unit 1562. The three-dimensional data generation model decoding unit 1562 has a first network decoding unit 1563, a second network decoding unit 1564, and a metadata decoding unit 1565.

[0551] The first network decoding unit 1563 decodes the trained first network based on the encoded data of the first network, and outputs the decoded trained first network.

[0552] The second network decoding unit 1564 decodes the trained second network based on the encoded data of the second network, and outputs the decoded trained second network.

[0553] The metadata decoding unit 1565 decodes the metadata based on the encoded data of the metadata, and outputs the decoded metadata.

[0554] Next, we will explain the specific configuration of the latent code decoding unit 1571. The latent code decoding unit 1571 has a first latent code decoding unit 1572 and a second latent code decoding unit 1573.

[0555] The first latent code decoding unit 1572 decodes and outputs the first latent code.

[0556] The second latent code decoding unit 1573 decodes and outputs the second latent code.

[0557] Note that the decoding method described with reference to Figures 59 and 61 may be applied as the latent code decoding method, thereby making it possible to appropriately decode a bitstream in which the code amount of the latent code has been reduced.

[0558] Next, a specific configuration of the reconstructing unit 1566 will be described. The reconstructing unit 1566 includes a density estimating unit 1567, a sampling point determining unit 1568, an attribute information estimating unit 1569, and a rendering unit 1570.

[0559] The density estimation unit 1567 estimates density information for the first sampling point at time t using the trained first network, the first sampling point, and the first latent code related to time t, and outputs the estimated density information.

[0560] The sampling point determination unit 1568 determines second sampling points based on the density information. The sampling point determination unit 1568 determines second sampling points using parameters included in the metadata in the same manner as the encoding device 1520. The sampling point determination unit 1568 outputs the determined second sampling points.

[0561] The attribute information estimation unit 1569 estimates density information and color information corresponding to the second sampling point at time t using the trained second network, the second sampling point, and the second latent code related to time t. The attribute information estimation unit 1569 outputs the estimated density information and color information. When a sampling point of an arbitrary viewpoint and the second latent code related to time t are input, the attribute information estimation unit 1569 may estimate density information and color information corresponding to the input sampling point of the arbitrary viewpoint at time t and output the estimated density information and color information.

[0562] The rendering unit 1570 performs rendering processing based on the density information and color information for each of the second sampling points, generates a two-dimensional image for each piece of viewpoint information, and outputs the generated two-dimensional image for each piece of viewpoint information.

[0563] The reconstruction unit 1566 may output the second sampling points and the attribute information (density information and color information) corresponding to the second sampling points estimated by the attribute information estimation unit 1569 as they are.

[0564] As shown in this embodiment, when a 3D data generation model is composed of two or more networks, a latent code may be stored for each network. This allows each network to be trained using its own latent code, thereby generating an appropriate latent code for each network. Using the latent code during generation can improve the accuracy of the results output by each network.

[0565] [Encoding device] Fig. 66 is a diagram showing an example of the configuration of an encoding device in Embodiment 3. Fig. 67 is a flowchart showing a first example of an encoding method performed by the encoding device in Embodiment 3.

[0566] The encoding device 1580 includes a circuit 1581 and a memory 1582 connected to the circuit 1581. The encoding device 1580 is a device that realizes the encoding devices 1510, 1520, and 1540.

[0567] The circuit 1581 performs the following operations.

[0568] The circuit 1581 acquires a plurality of three-dimensional data generation models (S1501). The circuit 1581 encodes the plurality of three-dimensional data generation models to generate a bitstream (S1502). The plurality of three-dimensional data generation models includes a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time.

[0569] Here, when viewpoint information including a viewpoint and a line of sight and a latent code are input, each of the plurality of three-dimensional data generation models may output a two-dimensional image of the subject as viewed from the viewpoint and line of sight at the time when the latent code is set. The bit stream includes the latent code. The latent code is set for a time corresponding to each of the plurality of three-dimensional data generation models.

[0570] According to this, since the latent code is encoded, it is possible to reduce the data size of the bit stream generated by encoding a plurality of three-dimensional data generation models.

[0571] For example, a latent code is a value that is set for a two-dimensional image at a particular time and indicates the characteristics of the two-dimensional image at that particular time.

[0572] Latent codes are used, for example, to compress the representation of scene motion and appearance changes. By using automatically learned latent information as a code instead of time, it is possible to more expressively record the movement of geometry or texture. This latent code allows for smooth representation of temporal changes, which can be useful for generating visual effects.

[0573] For example, a latent code is a multidimensional vector having n-dimensional (n is an integer equal to or greater than 2) values.

[0574] In this way, by expressing the latent code as a multidimensional vector, the characteristics of each time point can be expressed in more detail.

[0575] For example, in acquiring a plurality of three-dimensional data generation models (S1501), the circuit 1581 learns using latent codes set for the times corresponding to each of the plurality of three-dimensional data generation models.

[0576] This allows for the automatic learning of latent information, instead of time, using a code, making it possible to record, for example, the movement of geometry or texture more expressively. Furthermore, since learning can be performed to share images of similar subject states even when the time is different, the amount of data in the generated 3D data generation model can be reduced.

[0577] For example, the initial value of the latent code used for learning is set to 0 or a random value, which can reduce the process of calculating the initial value.

[0578] For example, the initial value of the latent code used in learning is set to the value of the latent code corresponding to a time different from the time to which the latent code corresponds, thereby reducing the processing for calculating the initial value.

[0579] For example, the initial value of the latent code used for learning is set to the value of the learned latent code. This allows for highly accurate generation of an extended 3D data generation model or latent code for scenes with little movement.

[0580] For example, the circuit 1581 further predictively encodes the latent code, and the latent code included in the bitstream is a predictively encoded latent code, thereby improving the coding efficiency of the latent code.

[0581] For example, in predictive coding of a latent code, the circuit 1581 encodes the residual between the latent code and the predicted value. The predictively coded latent code is the coded residual. This allows the amount of code for the latent code to be reduced because the residual is coded.

[0582] For example, the circuit 1581 switches between intra-prediction coding and inter-prediction coding in predictive coding of latent codes, thereby enabling the coding method to be changed according to the purpose.

[0583] For example, each of the multiple three-dimensional data generation models is composed of multiple networks including a first network and a second network different from the first network. In (i) obtaining the multiple three-dimensional data generation models (S1501), the circuit 1581 inputs a first initial value of the latent code to the first network and outputs the first latent code, and inputs a second initial value of the latent code to the second network and outputs the second latent code. In (ii) generating a bit stream (S1502), the circuit 1581 generates a bit stream including the first latent code and the second latent code.

[0584] This allows for a latent code to be output for each network, allowing for a latent code to be set according to the network, potentially further reducing the data size of the bitstream generated by encoding multiple 3D data generation models.

[0585] For example, the first initial value and the second initial value are different values.

[0586] This allows each latent code to be learned efficiently by setting an appropriate initial latent code for each.

[0587] For example, circuit 1581 further encodes a first residual of the second latent code and the first prediction based on the first latent code, and the latent code included in the bitstream is the encoded first residual.

[0588] This allows the amount of coding for the second latent code to be reduced. For example, the first latent code may be used as the first predicted value. In this case, the value obtained by subtracting the value of the first latent code from the second latent code is coded as the prediction residual of the second latent code. This allows the amount of coding to be reduced when the correlation between the first latent code and the second latent code is high.

[0589] For example, circuit 1581 further encodes a second residual of the first latent code and a second prediction based on the second latent code, and the latent code included in the bitstream is the encoded second residual.

[0590] This allows the amount of coding for the first latent code to be reduced. For example, the second latent code may be used as the second predicted value. In this case, the value obtained by subtracting the value of the second latent code from the first latent code is coded as the prediction residual of the first latent code. This allows the amount of coding to be reduced when the correlation between the first latent code and the second latent code is high.

[0591] FIG. 68 is a flowchart showing a second example of the encoding method performed by the encoding device in the third embodiment.

[0592] The circuit 1581 may perform the following operations.

[0593] The circuit 1581 acquires a three-dimensional data generation model including a first network and a second network different from the first network (S1511). The circuit 1581 encodes the three-dimensional data generation model to generate a bitstream (S1512). In encoding the three-dimensional data generation model, the circuit 1581 encodes a first predicted value based on the first network and a first residual between the first predicted value and the second network.

[0594] According to this, the second network of the three-dimensional data generation model encodes the first residual obtained using the first predicted value based on the first network, so the amount of code for the second network can be reduced.

[0595] For example, in encoding the three-dimensional data generation model, the circuit 1581 further encodes a second predicted value based on the second network and a second residual between the first network and the second predicted value.

[0596] According to this, the second residual obtained by using the second predicted value based on the second network for the first network of the three-dimensional data generation model is coded, so that the amount of coding for the first network can be reduced.

[0597] [Decoding device] Fig. 69 is a diagram showing an example of the configuration of a decoding device in Embodiment 3. Fig. 70 is a flowchart showing a first example of a decoding method by the decoding device in Embodiment 3.

[0598] The decoding device 1590 includes a circuit 1591 and a memory 1592 connected to the circuit 1591. The decoding device 1590 is a device that realizes the decoding devices 1515, 1530, and 1560.

[0599] The circuit 1591 performs the following operations.

[0600] The circuit 1591 acquires a bitstream (S1521). The circuit 1591 decodes a plurality of three-dimensional data generation models and a latent code from the bitstream (S1522). The plurality of three-dimensional data generation models includes a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time.

[0601] Here, when viewpoint information including a viewpoint and a line of sight and a latent code are input, each of the plurality of three-dimensional data generation models may output a two-dimensional image of the subject as seen from the viewpoint and line of sight at the time when the latent code is set. The latent code is set for the time corresponding to each of the plurality of three-dimensional data generation models.

[0602] This allows a plurality of 3D data generation models to be appropriately decoded from a bitstream with a reduced data size.

[0603] For example, a latent code is a value that is set for a two-dimensional image at a particular time and indicates the characteristics of the two-dimensional image at that particular time.

[0604] This allows the latent code to be utilized to properly decode multiple 3D data generative models from a bitstream with reduced data size.

[0605] For example, a latent code is a multidimensional vector having n-dimensional (n is an integer equal to or greater than 2) values.

[0606] In this way, by expressing the latent code as a multidimensional vector, it is possible to appropriately decode multiple three-dimensional data generation models in which the characteristics of each time are expressed in more detail.

[0607] For example, in acquiring a plurality of three-dimensional data generation models, the circuit 1591 learns using latent codes set for the times corresponding to each of the plurality of three-dimensional data generation models.

[0608] This allows the latent code to be utilized to properly decode multiple 3D data generative models from a bitstream with reduced data size.

[0609] For example, the initial value of the latent code used for learning is set to 0 or a random value, which can reduce the process of calculating the initial value.

[0610] For example, the initial value of the latent code used in learning is set to the value of the latent code corresponding to a time different from the time to which the latent code corresponds, thereby reducing the processing for calculating the initial value.

[0611] For example, the initial value of the latent code used for learning is set to the value of the learned latent code. This allows for highly accurate generation of an extended 3D data generation model or latent code for scenes with little movement.

[0612] For example, the circuit 1591 further predictively decodes the latent code, thereby enabling the latent code with improved coding efficiency to be properly decoded.

[0613] For example, in predictive decoding of a latent code, the circuit 1591 decodes the residual between the latent code and a predicted value, and adds the predicted value and the residual to decode the latent code. This allows the latent code with a reduced code size to be appropriately decoded.

[0614] For example, the circuit 1591 switches between intra-prediction decoding and inter-prediction decoding in predictive decoding of latent codes, thereby enabling the decoding method to be changed according to the purpose.

[0615] For example, each of the multiple three-dimensional data generation models is composed of multiple networks including a first network and a second network different from the first network. The bit stream includes a first latent code and a second latent code. The first latent code is a latent code output by inputting a first initial value of the latent code to the first network. The second latent code is a latent code output by inputting a second initial value of the latent code to the second network.

[0616] This allows the latent code to be appropriately decoded according to the network, and therefore multiple 3D data generative models can be appropriately decoded from a bitstream with a reduced data size.

[0617] For example, the first initial value and the second initial value are different from each other, which allows each efficiently learned latent code to be appropriately decoded.

[0618] For example, circuit 1591 further decodes a first residual between the second latent code and a first predicted value based on the first latent code. Circuit 1591 decodes the second latent code by adding the first predicted value and the first residual.

[0619] This allows the second latent code with a reduced code amount to be appropriately decoded.

[0620] For example, circuit 1591 further decodes a second residual between the first latent code and a second predicted value based on the second latent code. Circuit 1591 decodes the first latent code by adding the second predicted value and the second residual.

[0621] This allows the first latent code with a reduced code amount to be appropriately decoded.

[0622] FIG. 71 is a flowchart showing a second example of the encoding method performed by the decoding device in the third embodiment.

[0623] The circuit 1591 may perform the following operations.

[0624] The circuit 1591 acquires a bitstream (S1531). The circuit 1591 decodes from the bitstream a three-dimensional data generation model including a first network and a second network different from the first network (S1532). In decoding the three-dimensional data generation model, the circuit 1591 decodes a first predicted value based on the first network and a first residual between the first predicted value and the second network.

[0625] This allows the second network with a reduced amount of coding to be appropriately decoded.

[0626] For example, in decoding the three-dimensional data generation model, the circuit 1591 further decodes a second predicted value based on a second network and a second residual between the first network and the second predicted value.

[0627] This allows the first network with the reduced coding amount to be appropriately decoded.

[0628] Fourth Embodiment In a fourth embodiment, a method for encoding and transmitting a three-dimensional model (a learning model for generating three-dimensional data) will be described.

[0629] For example, point cloud data such as point clouds and meshes includes three-dimensional points or line information connecting the points, surface information, attribute information corresponding to the points, attribute information corresponding to the surfaces, etc. Therefore, when the resolution of the points or mesh increases or the area of ​​the points or mesh increases, the amount of data in the point cloud data increases in proportion to the increase in resolution or area.

[0630] When the area of ​​points or meshes is large, even if components of point cloud data such as points or meshes are encoded, the amount of data after encoding is large because the amount of data is large.

[0631] In contrast, a three-dimensional model, which is a learning model for generating three-dimensional data, does not increase the amount of data much even if the number of points or mesh areas increases. A three-dimensional model is a network model that learns two-dimensional data (two-dimensional images) or three-dimensional data (point clouds or meshes) to obtain three-dimensional shapes and attribute information corresponding to the three-dimensional shapes through learning using a neural network or the like. Note that, because a three-dimensional model is a network model for generating three-dimensional data, it can also be called a three-dimensional generative model.

[0632] Therefore, in order to reduce the storage capacity of three-dimensional data or the amount of three-dimensional data to be transmitted, a method for encoding and transmitting a three-dimensional model is required.

[0633] In this embodiment, with regard to the three-dimensional model learning unit (three-dimensional model acquisition unit), three-dimensional model encoding unit (network model encoding unit), and three-dimensional model decoding unit (network model decoding unit) described in the previous embodiment, a modeling method using NeRF (Neural Radiance Fields) as a method for modeling a three-dimensional model, a method for encoding the three-dimensional model, a method for decoding the encoded three-dimensional model, and a method for decoding two-dimensional images and three-dimensional data from the three-dimensional model will be specifically described.

[0634] A 3D model generated using the basic NeRF may be composed of multiple networks. The term "network" here refers to a learning model obtained by learning using a neural network. The multiple networks may include, for example, a network trained using sparse sampling points and a network trained using dense sampling points. In this way, the multiple networks are networks that differ in the number of input sampling points or the density of the sampling points. The sampling points may be, for example, three-dimensional points indicating three-dimensional positions.

[0635] Alternatively, the multiple networks may include, for example, a network for outputting geometry information indicating the density of an object, the probability of the object's existence, the coordinates of the geometry, etc., and a network for outputting information (attribute information) associated with the geometry, such as color information, reflectance, normal vector, color coordinates, timestamp, and object ID, based on the geometry information. The multiple networks may include two or more networks. The multiple networks may be three or more networks with different sampling points, or may include two or more networks for outputting geometry information or two or more networks for outputting attribute information.

[0636] The multiple networks may be coded using multiple network coding units, which may code the multiple networks with existing network coding means, such as Neural Network Coding (NNC) of the MPEG standard.

[0637] Next, a data structure for storing coded data will be described.

[0638] This configuration allows the decoder to identify each type of data from the NeRF encoded bit stream, allowing it to divide the data and decode it by type. Furthermore, the decoder can easily handle (i.e., process) each data unit, enabling it to realize functions such as parallel decoding, random access, partial decoding, and scalable decoding.

[0639] Furthermore, in a three-dimensional model made up of multiple networks, the decoder can identify networks of the same three-dimensional model by assigning the same identification ID to data that make up the same three-dimensional model.

[0640] Fig. 72 is a diagram showing an example of the syntax of metadata in units of sequences in Embodiment 4. Fig. 73 is a diagram showing an example of the syntax of metadata in units of frames.

[0641] A trained network trained with NeRF includes multiple layers, including an input layer, an intermediate layer, an output layer, and the like, nodes in each layer, weight coefficients for each node, a transformation function for each node, and a latent code. Among these, parameters that are the same throughout a sequence are stored in metadata common to the sequence, such as a sequence parameter set (SPS). Note that the number of dimensions of the latent code may be the same throughout the sequence, and therefore may be stored in sequence-common metadata, such as an SPS. For example, if the latent code has n dimensions (n ​​is an integer greater than or equal to 1), the value n indicating the number of dimensions of the latent code may be stored in sequence-common metadata, such as an SPS. As a result, if the number of dimensions of the latent code is the same throughout the sequence, it is set as a sequence-common value, thereby reducing the amount of code in the header.

[0642] Furthermore, parameters common to each frame, each access unit, or multiple frames are stored in metadata common to frames or multiple frames. Note that metadata common to sequences may be set as a VPS (Video Parameter Set) as parameters common to an entire video including one or more sequences.

[0643] For example, configuration information indicating the configuration of a network, such as information on multiple layers of the network, may be stored in the SPS or in frame-level metadata if it is constant in the sequence. The network configuration information may be stored in both the SPS and the frame-level metadata, in which case the frame-level metadata may be used preferentially. Alternatively, the network configuration information may be stored in either the SPS or the frame-level metadata, in which case a flag indicating whether the network configuration information is stored in the SPS or the metadata is stored in the higher-level SPS, and the decoding device 1530 may determine whether the network configuration information is stored in the SPS or the metadata based on the flag.

[0644] Furthermore, for example, if the number of dimensions n of the latent code is constant throughout the sequence, the number of dimensions n of the latent code may be stored in the SPS. This allows for a reduction in the amount of header code when the number of dimensions n of the latent code is constant throughout the sequence. Furthermore, if the number of dimensions n of the latent code changes on a frame-by-frame or multiple-frame basis, the number of dimensions n of the latent code may be stored in metadata common to both the frame and multiple-frame units. In other words, the number of dimensions n of the latent code may be stored in metadata corresponding to the predetermined unit as information common to the predetermined unit. This allows the decoding device to switch the number of dimensions n of the latent code on a frame-by-frame or multiple-frame basis, thereby generating a highly accurate network. Furthermore, the number of dimensions n of the latent code may be stored in both the SPS, which is metadata for the higher-level sequence unit, and the metadata for the lower-level frame unit or multiple-frame unit. In this case, the number of dimensions n stored in the metadata common to the frame unit or multiple-frame unit (i.e., the lower-level unit), may be given priority over the number of dimensions n stored in the metadata common to the higher-level unit. Furthermore, a flag indicating whether the number of dimensions n is stored in the metadata for the lower-level unit or the metadata for the higher-level unit may be stored in the higher-level metadata (SPS). The decoding device may determine whether the number of dimensions n is stored in the metadata of the lower unit or the metadata of the higher unit based on the flag, thereby flexibly adding the number of dimensions n of the latent code to the bitstream according to a predetermined unit.

[0645] In this way, the bit stream includes multiple networks and multiple latent codes. The multiple networks each correspond to multiple times. The multiple networks each correspond to different times. The multiple networks belong to a predetermined unit. That is, there are multiple predetermined units, and multiple networks are classified into each predetermined unit. The multiple latent codes each correspond to multiple times. Dimension number information indicating the number of dimensions of the latent code is provided for each predetermined unit. That is, the number of dimensions may be common to each predetermined unit. The dimension number information indicates the number of dimensions common to multiple latent codes corresponding to multiple networks belonging to the same predetermined unit.

[0646] Next, the data structure of the network coding layer will be described.

[0647] Fig. 74 is a diagram showing an example of the syntax of a data unit for a high-density network in embodiment 4. Fig. 75 is a diagram showing an example of the syntax of a data unit for a low-density network in embodiment 4. Note that high density means that the interval between sampling points is greater than a predetermined value and the sampling points are coarse. Low density means that the interval between sampling points is smaller than a predetermined value and the sampling points are fine.

[0648] Fig. 76 is a diagram showing an example of the configuration of a data unit of a first network in embodiment 4. Fig. 77 is a diagram showing an example of the configuration of a data unit of a second network in embodiment 4. Note that the first network is a network with a high density, and the second network is a network with a lower density than the first network.

[0649] Network parameters obtained as a result of training, such as node weight coefficients and latent codes, may be stored as network data in a network data unit. The network data may include configuration information of the network. The configuration information of the network may include, for example, information for identifying multiple layers including an input layer, an intermediate layer, an output layer, etc. of the network, nodes in each layer, weight coefficients for each node, and transformation functions for each node.

[0650] In addition, among the network data, data related to the output of density information (density network for outputting density information), data related to the output of color information (color network for outputting color information), data related to other attribute information (attribute network for outputting other attribute information (e.g., reflectance information)), and latent codes may be stored together as a group.

[0651] For example, if the trained first network has a density network for outputting density for sampling points, the network data, the density network, and the first latent code are encoded and stored in the payload of the network data unit.

[0652] Also, for example, if the trained second network has a density network for outputting density for sample points and a color network for outputting color information, the network data, the density network, the color network, and the second latent code are encoded and stored in the payload of the network data unit.

[0653] For network encoding, an existing network encoding method such as MPEG standard NNC (Neural Network Coding) may be used. In this case, an NNC data unit may be used as the network data unit. The encoded data defined by NNC may be stored in a data structure defined by NNC. Furthermore, the network data unit (NetworkDataUnit) may be divided into multiple data formats depending on the network configuration.

[0654] The number of dimensions of the latent code may be added to the network data unit (NetworkDataUnit), and the number of dimensions of the latent code may be set for each network. This allows an appropriate number of dimensions of the latent code to be set for each network. For example, by setting the number of dimensions of the latent code to be larger, a network with higher time resolution can be generated, and by setting the number of dimensions of the latent code to be smaller, a network with a smaller code size can be generated.

[0655] Next, the data structure of coded data of an NeRF three-dimensional model will be described. Fig. 78 is a diagram showing an example of the syntax of coded data of an NeRF three-dimensional model in Embodiment 4. Fig. 79 is a diagram showing an example of an NeRF unit type in Embodiment 4. Fig. 80A is a diagram showing another example of the syntax of coded data of an NeRF three-dimensional model in Embodiment 4. Fig. 80B is a diagram showing an example of the syntax of a Latent Data Unit in Embodiment 4. Fig. 81 is a diagram showing an example of the data structure of coded data of an NeRF three-dimensional model in Embodiment 4.

[0656] The coded data of the NeRF 3D model is transmitted, for example, stored in the model_codec_unit in the codec_unit() that handles the coding method in an integrated manner, as described in the data structure of the coded data in the above embodiment. Alternatively, the coded data of the NeRF 3D model may be transmitted as the model_codec_unit without being stored in the codec_unit. The coded data of the NeRF 3D model is referred to herein as the NeRF 3D model Unit.

[0657] The coded data of the NeRF three-dimensional model has a header and a configuration of NeRF 3D model data, and may be configured to store Fine NW Data Unit, Coarse NW Data Unit, NeRF Metadata, SPS, FPS, NPS, SEI, etc. Nerf_unit_type in the header indicates the type of data stored in the NeRF 3D model data. This enables the coding device to generate data that enables the decoding device 1530 to identify the components of the coded data of the NeRF model.

[0658] Here, if the data is a Fine NW Data Unit (first network) or a Coarse NW Data Unit (second network), a 3D sub model ID that identifies the three-dimensional model may be assigned. For example, if the data is a Fine NW Data Unit or a Coarse NW Data Unit in the same three-dimensional model, an ID indicating that the data is a data unit of the same three-dimensional model may be assigned.

[0659] Additionally, the Network Data Unit of the first network may include a first latent code, and the Network Data Unit of the second network may include a second latent code.

[0660] Furthermore, the coded data of the NeRF three-dimensional model may include a 3D model frame id indicating the frame number of the three-dimensional model. Furthermore, in the coded data of the NeRF three-dimensional model, the same frame id may be assigned to data units of the same time frame. Furthermore, in the coded data of the NeRF three-dimensional model, if the 3D model data is data divided into regions, the coded data of the NeRF three-dimensional model may include a space id indicating to which region the data belongs. In other words, the same space id is assigned to data of the same region. Note that this space id is the same as the space id described in the above embodiment.

[0661] Furthermore, the 3D model sub model id and the data unit id described in the above embodiment may be the same. These identifiers enable related data to be associated with each other, and the decoding device 1530 can recognize the associated data.

[0662] Note that a Nerf_unit_type for storing information about the latent code may be defined separately. For example, a Latent Data Unit may be prepared as Nerf_unit_type=6. That is, the bitstream may include Nerf_unit_type as type information indicating the type of data, and among the data included in the bitstream, data indicated by Nerf_unit_type as metadata including a latent code (data with Nerf_unit_type=6) is metadata (Latent Data Unit) including a latent code.

[0663] The Latent Data Unit may include information on the number of dimensions of the latent code, which allows the number of dimensions of the latent code to be appropriately switched for each Latent Data Unit.

[0664] The Latent Data Unit may include information indicating which network the latent code information is related to. That is, the metadata may include network identification information indicating the network to which the latent code corresponds. For example, the Latent Data Unit may include information for referencing a related Fine NW Data Unit or information for referencing a related Coarse NW Data Unit. This allows the latent code included in the Latent Data Unit to be linked to its associated network, and appropriate information can be output using both pieces of information (i.e., the latent code and the network).

[0665] Note that the Latent Data Unit may include a latent data ID (metadata identification information) for identifying the Latent Data Unit (or may be associated with the latent data ID), and the NW data unit (network) may include the latent data ID of the Latent Data Unit to which the NW data unit corresponds. This allows the NW data unit and the Latent Data Unit to be associated with each other.

[0666] For example, in the case where there is a first network and a first latent code corresponding to the first network, the first latent code may be stored in a Latent Data Unit to which latent data id=0 is added, and information ref_latent_data_id indicating the referenced latent data id may be added to the NW data unit in which the first network is stored, and the value of this information may be set to ref_latent_data_id=0, thereby indicating that the first latent code will be used when performing processing using the first network. This makes it possible to link the latent code included in the Latent Data Unit with the network associated therewith, and to output appropriate information using both pieces of information.

[0667] Thus, the bitstream includes a network and metadata including a latent code set for a time corresponding to the network, and the network includes metadata identification information indicating the metadata including the latent code corresponding to the network.

[0668] Note that the Latent Data Unit may include multiple latent data (latent codes) corresponding to multiple times, as shown in Fig. 80B. In other words, one piece of metadata may include multiple latent codes corresponding to multiple networks.

[0669] Next, structural information of the NeRF three-dimensional model (first network or second network) will be described. Fig. 82 is a diagram showing an example of the syntax of the SPS of the NeRF three-dimensional model in the fourth embodiment. Fig. 83 is a diagram showing an example of the syntax of the structural information of the NeRF three-dimensional model in the fourth embodiment. Fig. 84 is a diagram showing an example of the component_type in the fourth embodiment. Fig. 85 is a diagram showing an example of the component coding type in the fourth embodiment.

[0670] The SPS (Sequence Parameter Set) stores structural information of the NeRF that constitutes the sequence corresponding to the SPS. This allows the decoding device 1530 to obtain information on the constituent elements or components of the bitstream in which the SPS is stored, and to start decoding based on this information.

[0671] The SPS includes a field "number_of_component" indicating the number of components constituting the bitstream, and a field "component_type" for each component, which is an identifier for the component. The field "component_type" indicates, for example, that the encoded data is geometry or the density of geometry. Alternatively, the field "component_type" may indicate a type such as those shown in FIG. 84.

[0672] The SPS also includes a component coding type that indicates the encoding method by which the component is encoded. The encoding method indicated by the component coding type is, for example, MPEG G-PCC for point cloud compression, VVC for video codec, or NNC for network compression. Alternatively, the component coding type may indicate an encoding method exemplified in FIG. 85.

[0673] In the example of a bitstream consisting of two networks in this embodiment, the number of components is two, and they may be represented as follows: component0: component_type = 1 or 5, component coding type = 4 component1: component_type = 2 or 6, component coding type = 4

[0674] The above indicates whether component_type of component0 is 1 and component_type of component1 is 2, or whether component_type of component0 is 5 and component_type of component1 is 6.

[0675] As a result, the structural information of the NeRF three-dimensional model is notified to the decoding device 1530, and the decoding device 1530 can start decoding based on the structural information of the NeRF three-dimensional model.

[0676] Note that the component_types 0 to 6 in FIG. 84 are just an example, and not all of them need to be shown, and only some of them may be shown.

[0677] In addition, not only when component coding type == nnc, but also when a component is configured as a network, the structural information of the three-dimensional model of NeRF may include information such as the number of network layers, the number of nodes in each network layer, or the number of dimensions of the latent code. This allows the transmitting device (encoding device) to appropriately convey information about the network to the receiving device (combining device) even when nnc is not used as the codec.

[0678] Next, the reference relationship of the coded data of the NeRF three-dimensional model will be described. Figure 86 is a diagram for explaining the reference relationship of the coded data of the NeRF three-dimensional model in the fourth embodiment.

[0679] For example, the first network (CoarseNW) and the second network (FineNW) in the 3D generative model of frame 0 are assigned the same frame number 3D model frame id to indicate that they are networks corresponding to the same frame 0. Furthermore, the first network (CoarseNW) and the second network (FineNW) are assigned the same model identifier 3D model sub model id to indicate that they are networks corresponding to the same 3D generative model.

[0680] This allows the decoding device 1530 to identify the first network (CoarseNW) and the second network (FineNW) used when generating the 3D model. In addition, the first network (CoarseNW) and the second network (FineNW) are each assigned a ref_nerf_metadata_id, and the decoding device 1530 can perform decoding using the referenced NeRF metadata by referring to NeRF metadata having the same nerf_metadata_id during decoding.

[0681] In addition, when the number of dimensions of the latent code of the first network (Coarse NW) and the latent code of the second network (Fine NW) are the same, the number of dimensions of the common latent code may be included in NeRF Metadata. As a result, when the number of dimensions of the latent code of the first network (Coarse NW) and the latent code of the second network (Fine NW) are the same, it is not necessary to add the number of dimensions of the latent code to the data units of the first network (Coarse NW) and the second network (Fine NW), so the amount of code of the data units of the first network (Coarse NW) and the second network (Fine NW) can be reduced.

[0682] The first network (Coarse NW) may include a latent code corresponding to the first network (Coarse NW) or an ID of the latent code referenced by the first network. The ID of the latent code is a latent code included in NeRF metadata and is an ID for identifying the latent code corresponding to the first network. Similarly, the second network (Fine NW) may include a latent code corresponding to the second network (Fine NW) or an ID of the latent code referenced by the second network. The ID of the latent code is a latent code included in NeRF metadata and is an ID for identifying the latent code corresponding to the second network.

[0683] Thus, the bitstream includes multiple networks, e.g., a first network and a second network, where the first network corresponds to a first latent code and the second network corresponds to a second latent code, and the metadata (NeRF metadata) includes the first latent code and the second latent code, where the first network includes a first identification (id) for referencing the first latent code and the second network includes a second identification (id) for referencing the second latent code.

[0684] The first network (Coarse NW) and the second network (Fine NW) are three-dimensional models for reconstructing three-dimensional data of the same scene, and therefore correspond to a common time, as shown in Fig. 86. The second network (Fine NW) has a higher resolution of sampling points used for learning than the first network (Coarse NW).

[0685] Next, the bitstream division process in the decoding device will be described.

[0686] The bitstream input to the decoding device includes data of various NeRF 3D model units. The decoding device first analyzes the header of the NeRF 3D model unit. If the Nerf_unit_type is Fine NW Data Unit, the decoding device 1530 recognizes that the subsequent data is encoded data of the second network (FineNW) and decodes the data of the second network (FineNW). Note that if a latent code is included in the NW Data Unit, the decoding device may decode the latent code.

[0687] Similarly, the decoding device determines whether the data is a coarse NW Data Unit, NeRF Metadata, Sequence Parameter Set, Frame Parameter Set, or Network Parameter Set using Nerf_unit_type, and decodes the data indicated by each. Note that if Nerf_unit_type indicates that a Latent Data Unit is included in the bitstream (for example, if Nerf_unit_type=6), the decoding device may recognize that the subsequent data is data related to a latent code and decode the latent code data.

[0688] Next, the decoding process of the first network and the second network will be described.

[0689] The decoding device analyzes the 3D model frame ID, 3D model sub model ID, 3D model space ID, and ref_nerf_metadata_id to determine which frame, model, and space the encoded data of the first network or the encoded data of the second network belongs to, and pairs encoded data with the same identifier. The decoding device becomes ready for decoding when all encoded data with the same identifier are collected.

[0690] For example, when a decoding device receives a fine NW Data Unit with 3D model frame id = 0 and 3D model sub model id = 1, it searches for a coarse NW Data Unit with the same ID, and when it finds one, it determines that it can be decoded and starts decoding. ...

Claims

1. An encoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, acquires a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network; and generates a bitstream including the network and metadata including the latent code, wherein the network includes metadata identification information indicating the metadata including the latent code corresponding to the network.

2. The encoding device of claim 1, wherein the bitstream further includes type information indicating the type of data contained in the bitstream, and among the data contained in the bitstream, data indicated by the type information to be metadata including a latent code is metadata including the latent code.

3. The encoding device according to claim 1 or 2, wherein the metadata further includes dimensionality information indicating the number of dimensions of the latent code.

4. The encoding device of claim 3, wherein the bit stream includes: a plurality of networks that include the network and correspond to a plurality of times; a plurality of latent codes that include the latent code and are set for the plurality of times; and the dimension number information indicates a number of dimensions common to the plurality of latent codes.

5. The encoding device of claim 1 or 2, wherein the bitstream includes a plurality of networks including the network, the plurality of networks including a first network corresponding to a first latent code and a second network corresponding to a second latent code, the metadata including the first latent code and the second latent code, the first network including first identification information for referencing the first latent code, and the second network including second identification information for referencing the second latent code.

6. The encoding device according to claim 5, wherein the first latent code and the second latent code correspond to a common time, and the second network has a higher resolution of sampling points used for learning than the first network.

7. The encoding device according to claim 1 or 2, wherein the metadata includes network identification information indicating the network to which the latent code corresponds.

8. A decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, obtains a bitstream including a network constituting a three-dimensional data generation model and metadata including a latent code set for a time corresponding to the network, the network including metadata identification information indicating the metadata including the latent code corresponding to the network, and decodes the network based on the bitstream.

9. The decoding device of claim 8, wherein the bitstream further includes type information indicating the type of data contained in the bitstream, and among the data contained in the bitstream, data indicated by the type information to be metadata including a latent code is metadata including the latent code.

10. The decoding device according to claim 8 or 9, wherein the metadata further includes dimensionality information indicating the number of dimensions of the latent code.

11. The decoding device described in claim 10, wherein the bit stream includes: a plurality of networks including the network and corresponding to a plurality of times; a plurality of latent codes including the latent code and set for the plurality of times; and the dimension number information indicates the number of dimensions common to the plurality of latent codes.

12. A decoding device as described in claim 8 or 9, wherein the bit stream includes a plurality of networks including the network, the plurality of networks including a first network corresponding to a first latent code and a second network corresponding to a second latent code, the metadata includes the first latent code and the second latent code, the first network includes first identification information indicating the first latent code, and the second network includes second identification information indicating the second latent code.

13. The decoding device according to claim 12, wherein the first latent code and the second latent code correspond to a common time, and the second network has a higher resolution of sampling points used for learning than the first network.

14. The decoding device according to claim 8 or 9, wherein the metadata includes network identification information indicating the network to which the latent code corresponds.

15. An encoding method executed by an encoding device, comprising: acquiring a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network; generating a bitstream including the network and metadata including the latent code; and the network including metadata identification information indicating the metadata including the latent code corresponding to the network.

16. A decoding method executed by a decoding device, comprising: acquiring a bitstream including a network constituting a three-dimensional data generation model and metadata including a latent code set for a time corresponding to the network; the network including metadata identification information indicating the metadata including the latent code corresponding to the network; and decoding the network based on the bitstream.

Images