Encoding apparatus, decoding apparatus, encoding method, and decoding method

Abstract

The encoding apparatus acquires a network configuring a three-dimensional data generation model and a latent code set for a time corresponding to the network (S1601), and generates a bit stream including the acquired network and metadata including the acquired latent code (S1602). The network includes metadata identification information indicating metadata including the latent code corresponding to the network. Thus, the network includes metadata identification information indicating metadata including the latent code corresponding to the network, so that a decoding apparatus that acquires the bit stream can determine the metadata including the latent code corresponding to the network based on the metadata identification information. Therefore, the decoding apparatus can output appropriate information based on the network and the latent code.

Description

Technical Field

[0001] This disclosure relates to encoding devices, decoding devices, encoding methods, and decoding methods. Background Technology

[0002] Devices and services utilizing 3D data are expected to become increasingly common in a wide range of fields, including computer vision, mapping information, surveillance, infrastructure inspection, and image distribution, which enable autonomous movement of vehicles or robots. 3D data is acquired through various methods, such as distance sensors like rangefinders, stereo cameras, or combinations of multiple monocular cameras.

[0003] As one method of representing 3D data, there is the so-called point cloud method, which uses a group of points in 3D space to represent the shape of a 3D structure. In a point cloud, the position and color of the point group are stored. It is envisioned that point clouds will become the mainstream method of representing 3D data, but the data volume of point groups is extremely large. Therefore, in the accumulation or transmission of 3D data, similar to 2D moving images (for example, MPEG-4 AVC or HEVC, which are standardized through MPEG), data compression based on encoding is necessary.

[0004] In addition, point cloud compression is partially supported by publicly available libraries that perform point cloud association processing (such as Point Cloud Library).

[0005] In addition, there are known technologies that use three-dimensional map data to retrieve and display facilities located around a vehicle (for example, see Patent Document 1).

[0006] Existing technical documents Patent documents Patent Document 1: International Publication No. 2014 / 020663 Non-patent literature Non-patent document 1: ISO / IEC 15938-17:2022 (Information technology - Multimedia content description interface - Part 17: Compression of neural networks for multimedia content description and analysis (https / / www.iso.org / standard / 78480.html)) Summary of the Invention

[0007] The problem that the invention aims to solve The purpose of this disclosure is to provide an encoding device, etc., capable of reducing the data size of a bitstream generated by encoding multiple three-dimensional data generation models.

[0008] Methods for solving problems An encoding apparatus according to one aspect of this disclosure includes: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, acquires a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network, and generates a bitstream containing the network and metadata, the metadata containing the latent code, and the network containing metadata identification information indicating metadata containing the latent code corresponding to the network.

[0009] One aspect of the decoding apparatus disclosed herein includes: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, acquires a bitstream containing a network constituting a three-dimensional data generation model and metadata containing a latent code set for a time corresponding to the network, the network containing metadata identification information indicating metadata containing the latent code corresponding to the network, and decodes the network based on the bitstream.

[0010] Furthermore, these general or specific methods can be implemented either through systems, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0011] Invention Effects The encoding apparatus and the like disclosed herein can generate a bit stream of information that can be appropriately output. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating an example of the configuration of a three-dimensional data encoding and decoding system according to Embodiment 1.

[0013] Figure 2 This is a diagram showing the composition of the point group data in Implementation Method 1.

[0014] Figure 3 This is a diagram illustrating an example of the structure of a data file that describes information about point group data in Implementation 1.

[0015] Figure 4 This is a diagram showing the composition of the three-dimensional mesh data in Implementation Method 1.

[0016] Figure 5 This is a diagram illustrating an example of the structure of a data file containing information about three-dimensional mesh data in Implementation 1.

[0017] Figure 6 This is a diagram used to illustrate the three-dimensional model in Implementation Method 1.

[0018] Figure 7 This is a diagram representing the types of three-dimensional data in Implementation Method 1.

[0019] Figure 8 This is a diagram used to illustrate the encoding process of three-dimensional data in Implementation Method 1.

[0020] Figure 9 This is a diagram used to illustrate the decoding process of three-dimensional data in Implementation Method 1.

[0021] Figure 10 This is a two-dimensional schematic diagram representing the tiles and slices of the three-dimensional data in Implementation Method 1.

[0022] Figure 11 This is a block diagram illustrating an example of the functional configuration of the server and terminal in Implementation Method 1.

[0023] Figure 12 This is a block diagram illustrating another example of the data generation unit of the server in Implementation Method 1.

[0024] Figure 13 This is a diagram used to illustrate the relationship between the three-dimensional space and the encoded data in Implementation Method 1.

[0025] Figure 14 This is a diagram illustrating an example of the syntax of the encoding method unit in Implementation Method 1.

[0026] Figure 15 This is a diagram illustrating an example of the syntax of the encoded point group in Implementation Method 1.

[0027] Figure 16 This is a diagram illustrating an example of the syntax of the encoding grid in Implementation 1.

[0028] Figure 17 This is a diagram illustrating an example of the syntax for encoding a three-dimensional model in Implementation Method 1.

[0029] Figure 18 This is a diagram illustrating an example of the syntax for three-dimensional data information in Implementation Method 1.

[0030] Figure 19 This is a diagram used to illustrate the data structure of the coded point group in Implementation Method 1.

[0031] Figure 20 This is a diagram used to illustrate the data structure of the coded grid in Implementation 1.

[0032] Figure 21 This is a diagram used to illustrate the data structure of the encoded three-dimensional model in Implementation Method 1.

[0033] Figure 22 This is a diagram illustrating an example of multiple three-dimensional spaces in Implementation Method 1, presented in two dimensions.

[0034] Figure 23 This is a diagram showing an example of a bounding box in Implementation 1.

[0035] Figure 24 This is a diagram illustrating an example of the syntax for three-dimensional spatial information in Implementation 1.

[0036] Figure 25 This is a flowchart illustrating an example of partial decoding in Implementation 1.

[0037] Figure 26 This is a diagram illustrating an example of a three-dimensional spatial region that becomes part of the decoded object in Implementation 1.

[0038] Figure 27 This is a diagram illustrating an example of the data structure of the partially decoded group of encoded points in Implementation 1.

[0039] Figure 28 This is a diagram illustrating an example of the data structure of the partially decoded encoded grid in Implementation 1.

[0040] Figure 29 This is a diagram illustrating an example of the data structure of the partially decoded encoded 3D model in Implementation 1.

[0041] Figure 30 This is a diagram illustrating an example of the configuration of the decoding device in Embodiment 1.

[0042] Figure 31 This is a flowchart illustrating an example of a decoding method of the decoding device in Embodiment 1.

[0043] Figure 32 This is a flowchart illustrating another example of a decoding method using a decoding device.

[0044] Figure 33 This is a diagram illustrating an example of the configuration of an encoding device.

[0045] Figure 34 This is a flowchart illustrating an example of an encoding method using an encoding device.

[0046] Figure 35 This is a diagram used to illustrate the processing during the learning of the three-dimensional generative model in Implementation Method 2.

[0047] Figure 36 This diagram illustrates the process of generating a still image of a subject from any viewpoint using a three-dimensional generative model in Implementation 2.

[0048] Figure 37 This is a diagram illustrating the motion image generation method using the three-dimensional data generation model of Example 1 in Embodiment 2.

[0049] Figure 38 This is a diagram illustrating the first example of the configuration of the encoding device in Embodiment 1 of Embodiment 2.

[0050] Figure 39 This is a diagram illustrating the first example of the configuration of the decoding device in Embodiment 1 of Embodiment 2.

[0051] Figure 40 This is a second example of the configuration of the encoding device in Embodiment 1 of Embodiment 2.

[0052] Figure 41 This is a second example of the configuration of the decoding device in Embodiment 1 of Embodiment 2.

[0053] Figure 42 This is a diagram illustrating the motion image generation method using the extended three-dimensional data generation model of Embodiment 2 in Implementation 2.

[0054] Figure 43 This is a diagram illustrating the first example of the configuration of the encoding device in Embodiment 2 of Implementation 2.

[0055] Figure 44 This is a diagram illustrating the first example of the configuration of the decoding device in Embodiment 2 of Implementation 2.

[0056] Figure 45 This is a second example of the configuration of the encoding device in Embodiment 2 of Implementation 2.

[0057] Figure 46 This is a second example of the configuration of the decoding device in Embodiment 2 of Implementation 2.

[0058] Figure 47 This is a diagram illustrating a motion image generation method using an extended three-dimensional data generation model, which is used to explain a variation of Embodiment 2.

[0059] Figure 48 This is a diagram illustrating a motion image generation method using a three-dimensional data generation model, which is used to explain a variation of Embodiment 2.

[0060] Figure 49 This is a diagram illustrating an example of the configuration of the encoding device in Embodiment 2.

[0061] Figure 50 This is a flowchart illustrating an example of the encoding method of the encoding device in Embodiment 2.

[0062] Figure 51 This is a diagram illustrating an example of the configuration of the decoding device in Embodiment 2.

[0063] Figure 52 This is a flowchart illustrating an example of a decoding method of the decoding device in Embodiment 2.

[0064] Figure 53 This is a diagram illustrating an example of the configuration of an encoding device.

[0065] Figure 54 This is a diagram illustrating an example of the configuration of a decoding device.

[0066] Figure 55 This is a diagram used to illustrate the processing during the learning of the three-dimensional data generation model in Implementation Method 3.

[0067] Figure 56 This diagram illustrates the process of generating a still image of a subject observed from any viewpoint using a three-dimensional data generation model in Implementation 3.

[0068] Figure 57 This is a diagram illustrating the motion image generation method using the extended three-dimensional data generation model of Example 1 in Embodiment 3.

[0069] Figure 58 This is a diagram illustrating the first example of the configuration of the encoding device in Embodiment 1 of Implementation 3.

[0070] Figure 59 This is a diagram illustrating the first example of the configuration of the decoding device in Embodiment 1 of Embodiment 3.

[0071] Figure 60 This is a second example of the configuration of the encoding device in Embodiment 1 of Embodiment 3.

[0072] Figure 61 This is a second example of the configuration of the decoding device in Embodiment 1 of Embodiment 3.

[0073] Figure 62 This is a block diagram illustrating an example of the configuration of an encoding apparatus for encoding multiple networks in Embodiment 2 of Implementation 3.

[0074] Figure 63 This is a diagram showing an example of the encoded data of the learned first network in Implementation 3.

[0075] Figure 64 This is a diagram representing an example of the encoded data of the learned second network in Implementation 3.

[0076] Figure 65This is a block diagram illustrating an example of the configuration of a decoding device that decodes multiple networks in Embodiment 3.

[0077] Figure 66 This is a diagram illustrating an example of the configuration of the encoding device in Embodiment 3.

[0078] Figure 67 This is a flowchart illustrating the first example of the encoding method of the encoding device in Embodiment 3.

[0079] Figure 68 This is a flowchart illustrating a second example of the encoding method of the encoding device in Embodiment 3.

[0080] Figure 69 This is a diagram illustrating an example of the configuration of the decoding device in Embodiment 3.

[0081] Figure 70 This is a flowchart illustrating the first example of the decoding method of the decoding device in Embodiment 3.

[0082] Figure 71 This is a flowchart illustrating the second example of the encoding method of the decoding device in Embodiment 3.

[0083] Figure 72 This is a diagram illustrating an example of the syntax of metadata for sequence units in implementation 4.

[0084] Figure 73 This is a diagram illustrating an example of the syntax for the metadata of a frame unit in Implementation 4.

[0085] Figure 74 This is a diagram illustrating an example of the syntax of data units in a high-density network in Implementation 4.

[0086] Figure 75 This is a diagram illustrating an example of the syntax of data units in a low-density network in Implementation 4.

[0087] Figure 76 This is a diagram illustrating an example of the configuration of the data units of the first network in Implementation Method 4.

[0088] Figure 77 This is a diagram illustrating an example of the configuration of the data units of the second network in Implementation Method 4.

[0089] Figure 78 This is a diagram illustrating an example of the syntax of the encoded data of the three-dimensional model of NeRF in Implementation 4.

[0090] Figure 79 This is a diagram illustrating an example of the cell type of NeRF in Implementation 4.

[0091] Figure 80AThis is a diagram illustrating another example of the syntax of the encoded data of the three-dimensional model of NeRF in Implementation 4.

[0092] Figure 80B This is a diagram illustrating an example of the syntax of the Latent Data Unit in Implementation 4.

[0093] Figure 81 This is a diagram illustrating an example of the data structure of the encoded data of the three-dimensional model of NeRF in Implementation 4.

[0094] Figure 82 This is a diagram illustrating an example of the SPS syntax of the three-dimensional model of NeRF in Implementation 4.

[0095] Figure 83 This is a diagram illustrating an example of the syntax of the structural information of the three-dimensional model of NeRF in Implementation 4.

[0096] Figure 84 This is a diagram representing an example of component_type in implementation method 4.

[0097] Figure 85 This is a diagram illustrating an example of the component coding type in implementation method 4.

[0098] Figure 86 This is a diagram illustrating the reference relationship of the encoded data of the three-dimensional model of NeRF in Implementation 4.

[0099] Figure 87 This is a diagram illustrating an example of how the data of a frame in Implementation 4 is divided into three three-dimensional spaces.

[0100] Figure 88 This is a diagram illustrating an example of an ID assigned to the segmented data in Implementation 4.

[0101] Figure 89 This is a diagram used to illustrate the first example of the encoding method in Implementation Method 4.

[0102] Figure 90 This is a diagram illustrating the first example of the output of the decoding device in Embodiment 4.

[0103] Figure 91 This is a diagram illustrating the second example of the encoding method in Implementation Method 4.

[0104] Figure 92 This is a diagram illustrating the second example of the output of the decoding device in Embodiment 4.

[0105] Figure 93 This is a diagram illustrating the third example of the encoding method in Implementation Method 4.

[0106] Figure 94 This is a diagram illustrating the third example of the output of the decoding device in Embodiment 4.

[0107] Figure 95 This is a diagram illustrating the fourth example of the encoding method in Implementation Method 4.

[0108] Figure 96 This is a diagram illustrating the fourth example of the output of the decoding device in Embodiment 4.

[0109] Figure 97 This is a diagram illustrating the fifth example of the encoding method in Implementation Method 4.

[0110] Figure 98 This is a diagram illustrating the fifth example of the output of the decoding device in Embodiment 4.

[0111] Figure 99 This diagram illustrates the data exchange between the decoding unit and the control unit in Embodiment 4.

[0112] Figure 100 This is a diagram used to illustrate the consistency points in implementation method 4.

[0113] Figure 101 This is a diagram illustrating an example of a bitstream containing multiple networks in Implementation 4.

[0114] Figure 102 This is a diagram illustrating an example of the syntax of the layered structure of multiple networks in Implementation 4.

[0115] Figure 103 This is a block diagram illustrating an example of the configuration of a modified encoding device according to Embodiment 4.

[0116] Figure 104 This is a block diagram illustrating an example of the configuration of a modified version of the decoding device according to Embodiment 4.

[0117] Figure 105 This is a diagram illustrating an example of the configuration of the encoding device in Embodiment 4.

[0118] Figure 106 This is a flowchart illustrating the first example of the encoding method of the encoding device in Embodiment 4.

[0119] Figure 107 This is a diagram illustrating an example of the configuration of the decoding device in Embodiment 4.

[0120] Figure 108 This is a flowchart illustrating the first example of the decoding method of the decoding device in Embodiment 4.

[0121] Figure 109This is a flowchart illustrating the second example of the encoding method of the encoding device in Embodiment 4.

[0122] Figure 110 This is a flowchart illustrating a second example of the decoding method of the decoding device in Embodiment 4.

[0123] Figure 111 This is a flowchart illustrating the third example of the encoding method of the encoding device in Embodiment 4.

[0124] Figure 112 This is a flowchart illustrating the third example of the decoding method of the decoding device in Embodiment 4.

[0125] Figure 113 This is a flowchart illustrating the fourth example of the encoding method of the encoding device in Embodiment 4.

[0126] Figure 114 This is a flowchart illustrating the fourth example of the decoding method of the decoding device in Embodiment 4.

[0127] Figure 115 This is a flowchart illustrating the fifth example of the encoding method of the encoding device in Embodiment 4.

[0128] Figure 116 This is a flowchart illustrating the fifth example of the decoding method of the decoding device in Embodiment 4. Detailed Implementation

[0129] The encoding apparatus of the first aspect of this disclosure includes: a circuit; and a memory connected to the circuit, wherein, in operation, the circuit acquires a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network, and generates a bit stream containing the network and metadata, wherein the metadata contains the latent code, and the network contains metadata identification information indicating metadata containing the latent code corresponding to the network.

[0130] Therefore, the network contains metadata identification information, which represents metadata containing the latent code corresponding to the network. Thus, the decoding device that has obtained the bitstream can determine the metadata containing the latent code corresponding to the network based on the metadata identification information. Consequently, the decoding device can output appropriate information based on the network and the latent code.

[0131] In the encoding apparatus of the second aspect of this disclosure, in the encoding apparatus of the first aspect, the bit stream further includes category information, which indicates the category of data contained in the bit stream, and the data contained in the bit stream that, according to the category information, contains metadata of the latent code is metadata of the latent code.

[0132] Therefore, metadata can be used to represent the data contained in a bitstream using category information.

[0133] In the encoding apparatus of the third aspect of this disclosure, in the encoding apparatus of the first or second aspect, the metadata further includes dimension information representing the dimension of the latent code.

[0134] Therefore, it is possible to appropriately switch the dimension of the latent code for each metadata.

[0135] The fourth type of encoding apparatus of this disclosure is in the third type of encoding apparatus, wherein the bit stream comprises: a plurality of networks including the network and corresponding to a plurality of times; and a plurality of potential codes including the potential codes and set for the plurality of times, wherein the dimension information represents a common dimension among the plurality of potential codes.

[0136] Therefore, since dimensionality information is common across multiple potential codes, the amount of dimensionality information encoded can be reduced.

[0137] In the fifth aspect of the encoding apparatus of this disclosure, in any of the first to third aspects of the encoding apparatus, the bit stream includes a plurality of networks including the network, the plurality of networks including a first network corresponding to a first potential code and a second network corresponding to a second potential code, the metadata including the first potential code and the second potential code, the first network including first identification information for referring to the first potential code, and the second network including second identification information for referring to the second potential code.

[0138] Therefore, the decoding device that has obtained the bitstream can determine the first potential code corresponding to the first network and the second potential code corresponding to the second network. Thus, the decoding device can appropriately output information using the first network and the first potential code, and also can appropriately output information using the second network and the second potential code.

[0139] The sixth type of encoding apparatus of this disclosure is in the fifth type of encoding apparatus, wherein the first latent code and the second latent code correspond to a common time, and the second network has a higher resolution of sampling points for learning compared to the first network.

[0140] In the encoding apparatus of the seventh aspect of this disclosure, in any of the encoding apparatuses of the first to fourth aspects, the metadata includes network identification information representing the network corresponding to the latent code.

[0141] Therefore, by establishing a correspondence between the latent code contained in the metadata and the network corresponding to that latent code, the bitstream decoding device can output appropriate information based on the network and the latent code.

[0142] The decoding apparatus of the eighth aspect of this disclosure includes: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, acquires a bit stream containing a network constituting a three-dimensional data generation model and metadata containing a latent code set for a time corresponding to the network, the network containing metadata identification information indicating metadata containing the latent code corresponding to the network, and decodes the network based on the bit stream.

[0143] Therefore, the network contains metadata identification information, which represents metadata containing the latent code corresponding to the network. Thus, the decoding device that has obtained the bitstream can determine the metadata containing the latent code corresponding to the network based on the metadata identification information. Consequently, the decoding device can output appropriate information based on the network and the latent code.

[0144] The decoding apparatus of the ninth aspect of this disclosure, in the decoding apparatus of the eighth aspect, further includes category information in the bit stream, which indicates the category of data contained in the bit stream, and the data in the bit stream that, as indicated by the category information, contains metadata of the latent code is metadata containing the latent code.

[0145] Therefore, the decoding device can determine that the data contained in the bitstream is metadata based on the type information.

[0146] In the decoding apparatus of the 10th method of this disclosure, in the decoding apparatus of the 8th or 9th method, the metadata further includes dimension information representing the dimension of the latent code.

[0147] Therefore, the decoding device can appropriately switch the dimension of the latent code according to each metadata, and can output information appropriately.

[0148] The decoding apparatus of the 11th aspect of this disclosure, in the decoding apparatus of the 10th aspect, the bit stream includes: a plurality of networks including the network and corresponding to a plurality of times; and a plurality of potential codes including the potential codes and set for the plurality of times, wherein the dimension information represents the common dimension among the plurality of potential codes.

[0149] Therefore, since the dimensional information is common across multiple potential codes, the decoding device can obtain a bitstream with reduced coding volume of the dimensional information.

[0150] The decoding apparatus of the 12th method of this disclosure is a decoding apparatus of any of the 8th to 11th methods, wherein the bit stream includes a plurality of networks including the network, the plurality of networks including a first network corresponding to a first potential code and a second network corresponding to a second potential code, the metadata includes the first potential code and the second potential code, the first network includes first identification information representing the first potential code, and the second network includes second identification information representing the second potential code.

[0151] Therefore, the decoding device that has obtained the bitstream can determine the first potential code corresponding to the first network and the second potential code corresponding to the second network. Thus, the decoding device can appropriately output information using the first network and the first potential code, and also can appropriately output information using the second network and the second potential code.

[0152] The decoding apparatus of the 13th aspect of this disclosure is in the decoding apparatus of the 12th aspect, wherein the first latent code and the second latent code correspond to a common time, and the second network has a higher resolution of sampling points for learning compared to the first network.

[0153] In the decoding apparatus of the 14th aspect of this disclosure, in any of the decoding apparatuses of the 8th to 11th aspects, the metadata includes network identification information representing the network corresponding to the latent code.

[0154] Therefore, by establishing a correspondence between the latent code contained in the metadata and the network corresponding to that latent code, the bitstream decoding device can output appropriate information based on the network and the latent code.

[0155] The 15th aspect of the present disclosure is an encoding method executed by an encoding device, which obtains a network constituting a three-dimensional data generation model and a latent code set for a time corresponding to the network, and generates a bit stream containing the network and metadata, wherein the metadata contains the latent code, and the network contains metadata identification information, which indicates metadata containing the latent code corresponding to the network.

[0156] Therefore, the network contains metadata identification information, which represents metadata containing the latent code corresponding to the network. Thus, the decoding device that has obtained the bitstream can determine the metadata containing the latent code corresponding to the network based on the metadata identification information. Consequently, the decoding device can output appropriate information based on the network and the latent code.

[0157] The decoding method of the 16th aspect of this disclosure is a decoding method executed by a decoding device, which obtains a bitstream containing a network constituting a three-dimensional data generation model and metadata, the metadata containing a potential code set for a time corresponding to the network, the network containing metadata identification information, the metadata identification information indicating metadata containing the potential code corresponding to the network, and decoding the network based on the bitstream.

[0158] Therefore, the network contains metadata identification information, which represents metadata containing the latent code corresponding to the network. Thus, the decoding device that has obtained the bitstream can determine the metadata containing the latent code corresponding to the network based on the metadata identification information. Consequently, the decoding device can output appropriate information based on the network and the latent code.

[0159] Furthermore, these general or specific methods can be implemented either through systems, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0160] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below represent specific examples of this disclosure. The numerical values, shapes, materials, constituent elements, arrangement positions of constituent elements, connection methods, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Additionally, constituent elements in the following embodiments that are not described in the independent claims representing the highest-level concept will be described as arbitrary constituent elements.

[0161] (Implementation Method 1) The configuration of the three-dimensional data encoding and decoding system of this embodiment will be described. Figure 1 This is a diagram illustrating an example of the configuration of the three-dimensional data encoding and decoding system of this embodiment. For example... Figure 1 As shown, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 1001, a three-dimensional data decoding system 1002, a sensor terminal 1003, and an external connection unit 1004.

[0162] The 3D data encoding system 1001 generates encoded data or reused data by encoding 3D data. Furthermore, the 3D data encoding system 1001 can be a single device or a system implemented by multiple devices. Additionally, the 3D data encoding device may include a portion of the multiple processing units included in the 3D data encoding system 1001.

[0163] The 3D data encoding system 1001 includes a 3D data generation system 1011, a prompting unit 1012, an encoding unit 1013, a multiplexing unit 1014, an input / output unit 1015, and a control unit 1016. The 3D data generation system 1011 includes a sensor information acquisition unit 1017 and a 3D data generation unit 1018.

[0164] The sensor information acquisition unit 1017 acquires sensor signals from the sensor terminal 1003 and outputs the sensor signals to the three-dimensional data generation unit 1018. The three-dimensional data generation unit 1018 generates three-dimensional data based on the sensor signals and outputs the three-dimensional data to the encoding unit 1013.

[0165] The prompting unit 1012 prompts the user with sensor signals or three-dimensional data. For example, the prompting unit 1012 displays information or images based on sensor signals or three-dimensional data.

[0166] The encoding unit 1013 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 1014. The additional information may include, for example, sensor signals.

[0167] The multiplexing unit 1014 generates multiplexed data by multiplexing the encoded data, control information, and additional information input from the encoding unit 1013. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0168] The input / output unit 1015 (e.g., a communication unit or interface) outputs multiplexed data to the outside. Alternatively, the multiplexed data is stored in an internal memory or other storage unit. The control unit 1016 (or application execution unit) controls each processing unit. That is, the control unit 1016 performs control such as encoding and multiplexing. The control unit 1016 can also perform control such as demultiplexing, decoding, or prompting.

[0169] Alternatively, sensor signals can be input to the encoding unit 1013 or the multiplexing unit 1014. Furthermore, the input / output unit 1015 can directly output 3D data or encoded data to the outside.

[0170] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 1001 is input to the three-dimensional data decoding system 1002 via the external connection unit 1004.

[0171] The 3D data decoding system 1002 generates 3D data by decoding encoded or multiplexed data. Furthermore, the 3D data decoding system 1002 can be a single device or a system comprised of multiple devices. Additionally, the 3D data decoding device may include a portion of the multiple processing units included in the 3D data decoding system 1002.

[0172] The 3D data decoding system 1002 includes a sensor information acquisition unit 1021, an input / output unit 1022, a demultiplexing unit 1023, a decoding unit 1024, a prompting unit 1025, a user interface 1026, and a control unit 1027.

[0173] The sensor information acquisition unit 1021 acquires sensor signals from the sensor terminal 1003.

[0174] The input / output unit 1022 acquires the transmission signal, decodes the multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexing unit 1023.

[0175] The demultiplexing unit 1023 obtains encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 1024.

[0176] The decoding unit 1024 reconstructs the point group data by decoding the encoded data.

[0177] The prompting unit 1025 prompts the user with dot group data. For example, the prompting unit 1025 displays information or images based on the dot group data. The user interface 1026 obtains instructions based on the user's operation. The control unit 1027 (or application execution unit) controls each processing unit. That is, the control unit 1027 performs demultiplexing, decoding, and prompting control, etc.

[0178] Furthermore, the input / output unit 1022 can directly acquire point group data or encoded data from external sources. Additionally, the prompting unit 1025 can acquire additional information such as sensor signals and provide prompts based on this additional information. Furthermore, the prompting unit 1025 can also provide prompts based on user instructions obtained from the user interface 1026.

[0179] The sensor terminal 1003 generates the information obtained from the sensor, namely the sensor signal. The sensor terminal 1003 is a terminal equipped with a sensor or camera, such as a moving object like a car, a flying object like an airplane, a portable terminal, or a camera.

[0180] Sensor signals that can be acquired by sensor terminal 1003 include, for example: (1) the distance between sensor terminal 1003 and an object or the reflectivity of an object obtained from a LIDAR, millimeter-wave radar, or infrared sensor; (2) signals representing the distance between a camera and an object or the reflectivity of an object obtained from multiple monocular camera images or stereo camera images. Additionally, sensor signals may also include sensor posture, orientation, gyroscope (angular velocity), position (GPS information or altitude), velocity, or acceleration. Furthermore, sensor signals may also include temperature, air pressure, humidity, or magnetism.

[0181] The external connection unit 1004 is realized through communication with integrated circuits (LSI or IC), external storage units, cloud servers via the Internet, or broadcasting.

[0182] Next, the point group data will be explained. Figure 2 It is a graph representing the composition of point group data. Figure 3 This is a diagram illustrating the structure of a data file that records information about a group of data.

[0183] Point cluster data comprises data from multiple points. Each point's data includes location information (3D coordinates) and attribute information related to that location. A cluster of these points is called a point cluster. For example, a point cluster represents the 3D shape of an object.

[0184] Position information, such as three-dimensional coordinates, is sometimes referred to as geometric information. Additionally, the data for each point can also contain attribute information across multiple attribute categories. Attribute categories could include, for example, color or reflectivity.

[0185] One attribute can be mapped to one location, or multiple attributes with different attribute categories can be mapped to one location. Furthermore, multiple mappings can be established between attribute categories and one location.

[0186] Figure 3 The example data file shown illustrates a one-to-one correspondence between location information and attribute information, illustrating the location and attribute information of N points constituting the point group data.

[0187] Positional information includes, for example, information along the x, y, and z axes. Attribute information includes, for example, RGB color information. Representative data files include .ply files, etc.

[0188] Next, the three-dimensional mesh data will be explained. Figure 4 It is a diagram representing the composition of three-dimensional mesh data. Figure 5 This is a diagram illustrating an example of the structure of a data file that contains information about three-dimensional mesh data.

[0189] 3D mesh data is a data format used in Computer Graphics (CG) that represents the three-dimensional shape of an object through a collection of face information. These face information represent polygons such as triangles or quadrilaterals. 3D mesh data is also known as polygonal data or polygonal meshes.

[0190] The constituent elements are a group of three-dimensional points, vertices of the multiple three-dimensional points forming the group, edges connecting two vertices of the multiple three-dimensional points, and a set of faces enclosed by the multiple edges. A group of three-dimensional points is a collection of points that contains positional information in three-dimensional space and attribute information corresponding to that positional information. Furthermore, a three-dimensional point can also be simply referred to as a point.

[0191] Vertices can also possess attributes such as color, reflectivity, and normal vectors specific to a 3D point. The relationships between vertices forming an edge or face can also be represented by connectivity. Furthermore, vertices can also be represented by position. The face and back faces can be represented by the orientation of the normal vectors to a 3D point. Additionally, vertices can also possess surface-specific attribute information.

[0192] Grid data files can take the form of object files, for example. Figure 5 In the mesh data file shown, the position information G(1)~G(N) of the N vertices that make up the mesh and the attribute information A(1)~A(N) of the vertices are represented as vertex information. In the mesh data file, the vertex information may also not include attribute information.

[0193] Furthermore, attribute information does not necessarily have to correspond one-to-one with vertices. Figure 5 The mesh data file shows an example of three-dimensional mesh data with M attribute information A2.

[0194] Face information is represented by a combination of vertex indices. n[1,3,4] represents the face of a triangle formed by vertices n=1, n=3, and n=4.

[0195] Additionally, m[2,4,6] indicates that the attribute information in attribute information A2 with m=2, m=4, and m=6 correspond to three vertices respectively. Furthermore, an example of a face consisting of three vertices is shown here, but for a face, any number of vertices greater than or equal to 3 is acceptable, not limited to 3. For example, if the face is a quadrilateral, the number of vertices is 4; if the face is a polygon, the number of vertices is the same as the number of vertices constituting the polygon.

[0196] Furthermore, attribute information A2 can be represented by a file different from the mesh data file, or it can include its pointer information. For example, attribute information can be stored in a two-dimensional attribute map file, and the attribute map filename and the two-dimensional coordinates in the attribute map can be represented by attribute information A2 from the mesh data file. In this way, attribute information A2 can be contained in the mesh data file or represented by a file different from the mesh data file; regardless of the method used, attribute information for three-dimensional points can be specified.

[0197] Next, the three-dimensional model will be explained. Figure 6 It is a diagram used to illustrate a three-dimensional model.

[0198] A 3D model is a model generated based on 2D or 3D data.

[0199] The 3D model learning unit 1031 generates a network model, i.e., a 3D model, by learning 2D data (2D images) or 3D data (point groups or meshes) and using neural networks to learn 3D shapes and corresponding attribute information.

[0200] The 3D model learning unit 1031 can also generate a 3D model based on a 2D image by learning using NeRF (Neural Radiance Fields). Alternatively, the 3D model learning unit 1031 can generate a 3D model after transforming a 2D image into 3D data through photogrammetry. The 3D model can also be generated using 3D data obtained from a sensor (distance sensor).

[0201] Three-dimensional model data consists of the elements that constitute a three-dimensional model, containing information representing the structure of the network model, feature quantities, etc. For example, three-dimensional model data contains information related to the constituent elements of a neural network. This information includes, for example, multiple layers such as input layer, intermediate layers, and output layer, nodes in each layer, weight coefficients for nodes, and transformation functions for nodes.

[0202] The 3D model encoding unit 1032 can encode 3D model data and transmit the encoded 3D model data.

[0203] The 3D model decoding unit 1033 receives the transmitted encoded 3D model data and decodes the 3D model based on the encoded 3D model data.

[0204] The rendering reconstruction unit 1034 reconstructs (generates) two-dimensional data (two-dimensional images) or three-dimensional data (point groups or meshes) based on the decoded three-dimensional model. For example, when using a three-dimensional model modeled via NeRF, the rendering reconstruction unit 1034 obtains viewpoint position or view vector information, generates rendered two-dimensional data (two-dimensional images) based on the three-dimensional model and the viewpoint position or view vector, and outputs the two-dimensional data. The generated two-dimensional data represents a two-dimensional image of a three-dimensional object observed from the viewpoint position or from the viewpoint indicated by the view vector. The three-dimensional object is a three-dimensional object that serves as the source of the two-dimensional or three-dimensional data input to the three-dimensional model learning unit 1031.

[0205] Next, the types of three-dimensional data will be explained. Figure 7 It is a graph representing the types of three-dimensional data. For example... Figure 7 As shown, there are static and dynamic objects in 3D data.

[0206] Static objects are 3D data at any given time (a specific moment). Dynamic objects are 3D data that changes over time. Hereinafter, the point group data at a specific moment will be referred to as a PCC frame or frame. Additionally, grid data at any given time will be referred to as a grid frame or frame.

[0207] The object can be three-dimensional data with the area restricted to some extent, such as typical image data, or it can be three-dimensional data with the area unrestricted, such as map information.

[0208] In addition, there are points of various densities, which can include sparse point clusters (sparse grid data) and dense point clusters (dense grid data).

[0209] The details of each processing unit are described below. Sensor information is obtained through various methods such as distance sensors like LIDAR or rangefinders, stereo cameras, or combinations of multiple monocular cameras. The 3D data generation unit 1018 generates point group data based on the sensor information obtained by the sensor information acquisition unit 1017. The 3D data generation unit 1018 generates position information (geometric information) as point group data and adds attribute information specific to that position information to the position information.

[0210] The 3D data generation unit 1018 can also process point group data during the generation of position information or the addition of attribute information. For example, the 3D data generation unit 1018 can reduce the amount of data by deleting point groups with overlapping positions. Furthermore, the 3D data generation unit 1018 can transform the position information (position shift, rotation, or normalization, etc.) and process the point group data to generate mesh data. Additionally, the 3D data generation unit 1018 can also render attribute information.

[0211] In addition, Figure 1 In this system, the three-dimensional data generation system 1011 is included in the three-dimensional data encoding system 1001, but it can also be set independently outside the three-dimensional data encoding system 1001.

[0212] The encoding unit 1013 generates encoded data by encoding the three-dimensional data based on a predefined encoding method. Regarding encoding methods, there are G-PCC (encoding method using position information), V-PCC (encoding method using video codec), Draco (grid coding method), and V-DMC (grid coding method). The encoding method is not limited to these methods; for example, it can also be a method for encoding dynamic grids, or other methods combining these methods.

[0213] The decoding unit 1024 decodes the three-dimensional data by decoding the encoded data based on a predefined encoding method.

[0214] The multiplexing unit 1014 generates multiplexed data by multiplexing encoded data using existing multiplexing methods. The generated multiplexed data is transmitted or stored. In addition to multiplexing encoded data of 3D data, the multiplexing unit 1014 also multiplexes other media such as images, audio, subtitles, applications, and files, or reference time information. Furthermore, the multiplexing unit 1014 can also multiplex sensor information or attribute information associated with point group data.

[0215] As multiplexing methods or file formats, there are ISOBMFF, ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0216] The demultiplexing unit 1023 extracts encoded data, other media, and time information from the multiplexed data.

[0217] The input / output unit 1015 transmits multiplexed data using a method that matches the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 1015 can communicate with other devices via the Internet, or with storage units such as cloud servers.

[0218] Use HTTP, FTP, TCP, or UDP as the communication protocol. You can use either the PULL or PUSH communication method.

[0219] Either wired or wireless transmission can be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), or coaxial cable can be used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), or millimeter wave can be used.

[0220] In addition, broadcast methods can be used, such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3.

[0221] Next, the process of dividing 3D data into more than one 3D data segment will be explained. Figure 8 It is a diagram used to illustrate the encoding and processing of three-dimensional data. Figure 9 This is a diagram used to illustrate the decoding process of three-dimensional data.

[0222] like Figure 8 As shown, the data segmentation unit 1041 segments the three-dimensional data into one or more three-dimensional spaces, generating one or more segmented three-dimensional data (i.e., one or more segmented three-dimensional data). The encoding unit 1042 can encode the one or more segmented three-dimensional data to generate encoded data. The data segmentation unit 1041 and the encoding unit 1042 are components of an encoding device and can be included in that encoding device or in different devices.

[0223] One or more 3D spaces can be individually labeled as tiles or spaces. A 3D space is, for example, a bounding box. Additionally, the 3D data contained within each of the segmented 3D spaces can also be labeled as slices. A slice is segmented 3D data, including any of the following: a group of points with location information (Geometry) or attribute information (Attribute), a mesh, or a 3D model. Each slice in the multiple slices is encoded by the encoding unit 1042 according to each constituent element and output as encoded data. The encoded data includes the encoded slices.

[0224] like Figure 9As shown, in the decoding process, the decoding unit 1051 decodes one or more segmented 3D data (one or more slices) based on the encoded data. The data combining unit 1052 combines the one or more segmented 3D data to restore (generate) 3D data. The decoding unit 1051 and the data combining unit 1052 are components of a decoding device and can be included in that decoding device or in different devices. The one or more segmented 3D data decoded by the decoding unit 1051 may not be combined. The decoding unit 1051 may also decode a portion of the one or more segmented 3D data based on a portion of the encoded data and output the decoded portion of the segmented 3D data. In this case, the decoding device may not have the data combining unit 1052.

[0225] Figure 10 It is a two-dimensional schematic representation of tiles and slices of three-dimensional data.

[0226] When encoding multiple slices, the encoding device can encode using the dependencies between the slices or without using dependencies. When encoding without dependencies, the encoding device can encode each slice independently, reducing processing time by using parallel processing to encode multiple slices. Similarly, when the decoding device encodes multiple slices without dependencies, it can decode each slice independently, reducing processing time by using parallel processing to decode multiple slices. Furthermore, the decoding device can reduce the amount of processing by partially decoding only a portion of the multiple slices.

[0227] When encoding using dependency relationships, the encoding device signals the identifier representing the dependency relationship and encodes the dependent data sequentially. When multiple slices are encoded using dependency relationships, the decoding device decodes the dependent data sequentially based on the identifier.

[0228] In 3D data segmentation, any number of segments and any segmentation method can be used. The shape of an object can also be determined, and multiple 3D points can be segmented for each object. Alternatively, segmentation can be based on the number of 3D points contained in a tile; that is, an upper limit can be determined for the number of 3D points a tile can contain. Furthermore, map information or location information can be used to segment 3D data based on whether it is contained in 3D space (tile information). Multiple tile shapes can also overlap.

[0229] By dividing 3D data into multiple segmented 3D data in this way, parallel processing in adaptive encoding and decoding corresponding to content or objects becomes possible.

[0230] Next, a method for selecting prompts or transmitted 3D data from multiple 3D data sets will be explained.

[0231] The server stores multiple 3D data sets for the same space. For example, the server stores point cluster data and mesh data for the same space. The server is an example of an encoding device. The terminal, based on its purpose, switches the 3D data obtained from the server and displays the switched 3D data. The terminal can also be a terminal that parses 3D data. In this case, the terminal can also switch the 3D data to be displayed based on purposes such as parsing or displaying, or user operations. The terminal is an example of a decoding device.

[0232] During the switching of 3D data, it is also possible to switch between using point clusters or grids as the 3D data. Additionally, it is possible to switch between transmitting point clusters or transmitting grids as the 3D data. For example, the terminal can send the user's selection result to the server, receive (download) 3D data based on that selection result from the server, and provide prompts for the received 3D data. The 3D data (point clusters or grids) may or may not be encoded in the server. If the 3D data is encoded, the terminal can receive the encoded 3D data from the server, decode the 3D data based on the received encoded 3D data, and provide prompts for the decoded 3D data.

[0233] Next, the configuration of server 1070 and terminal 1090 will be explained. Figure 11 This is a block diagram illustrating an example of the functional composition of a server and a terminal.

[0234] The server 1070 includes a data generation unit 1071, a synchronization unit 1075, a point group coding unit 1076, a grid coding unit 1077, a model coding unit 1078, a multiplexing unit 1079, and a data extraction unit 1080.

[0235] The data generation unit 1071 generates three-dimensional data based on at least one of two-dimensional data and three-dimensional data. The generated three-dimensional data includes at least two of point group data, mesh data, and three-dimensional model data. The data generation unit 1071 has a point group generation unit 1072, a mesh generation unit 1073, and a model generation unit 1074. The data generation unit 1071 only needs to have at least two of the point group generation unit 1072, mesh generation unit 1073, and model generation unit 1074. The point group generation unit 1072 generates point group data based on at least one of two-dimensional data and three-dimensional data. The mesh generation unit 1073 generates mesh data based on at least one of two-dimensional data and three-dimensional data. The model generation unit 1074 generates three-dimensional model data by performing machine learning based on at least one of two-dimensional data and three-dimensional data.

[0236] The two-dimensional data input to the data generation unit 1071 can also be two-dimensional images acquired by a camera. The three-dimensional data input to the data generation unit 1071 can, for example, be point data of spaces such as a building site, factory, or office acquired by a sensor such as LiDAR. The data generation unit 1071 can also generate color information as attribute information for each point contained in the point data of the three-dimensional data using the two-dimensional image of the two-dimensional data. The three-dimensional data generated by the data generation unit 1071 can also be divided into arbitrary spaces. Point data, mesh data, and three-dimensional model data can also be divided into arbitrary spaces respectively.

[0237] The synchronization unit 1075 acquires the spatial position or time (reproduction time, decoding time, acquisition time, etc.) of the point group data, mesh data, and 3D model data generated by the data generation unit 1071. The time of each data point is the reproduction time, decoding time, acquisition time, etc. Alternatively, the synchronization unit 1075 may not acquire the synchronization of the point group data, mesh data, and 3D model data, but instead generate synchronization information for acquiring synchronization. Furthermore, the synchronization unit 1075 may perform processing to acquire the synchronization of at least two types of 3D data from the point group data, mesh data, and 3D model data generated by the data generation unit 1071, or generate synchronization information (synchronization signal) for acquiring synchronization; alternatively, it may not perform the processing for acquiring the synchronization of all three types of 3D data (synchronization processing).

[0238] The point group encoding unit 1076 encodes the point group data that has been synchronized by the synchronization unit 1075. Alternatively, the point group encoding unit 1076 may not encode the point group data. The point group data may be pre-encoded or encoded upon request from the terminal 1090.

[0239] The grid encoding unit 1077 encodes the grid data that has been synchronized by the synchronization unit 1075.

[0240] The model encoding unit 1078 encodes the three-dimensional model data that has been synchronized by the synchronization unit 1075.

[0241] The multiplexing unit 1079 uses a prescribed format or a prescribed multiplexing method to multiplex the encoded point group data (coded point group), the encoded mesh data (coded mesh data), the encoded 3D model data, and the synchronization information. Alternatively, multiplexing based on the multiplexing unit 1079 may not be performed. In this case, the server 1070 may not have the multiplexing unit 1079.

[0242] The data extraction unit 1080 extracts a portion of the multiplexed 3D data corresponding to the request from the terminal 1090 and sends the extracted portion of 3D data to the terminal 1090. Alternatively, data extraction based on the data extraction unit 1080 may not be performed. In this case, the server 1070 may not have the data extraction unit 1080. Without data extraction based on the data extraction unit 1080, the server 1070 can send the 3D data multiplexed by the multiplexing unit 1079 to the terminal 1090. Furthermore, even without multiplexing based on the multiplexing unit 1079, the server 1070 can send encoded point group data (encoded point group), encoded mesh data (encoded mesh), encoded 3D model data (encoded 3D model), and synchronization information to the terminal 1090, or send a bitstream containing encoded point group data (encoded point group), encoded mesh data (encoded mesh), encoded 3D model data (encoded 3D model), and synchronization information to the terminal 1090.

[0243] The terminal 1090 includes a control unit 1091, a decoding unit 1092, and a prompting unit 1093.

[0244] The control unit 1091 will send a request for a portion of the 3D data to the server 1070. The control unit 1091 can also process user operations to determine a portion of the 3D data.

[0245] The decoding unit 1092 decodes a portion of the 3D data based on the bit stream (encoded data) obtained from the server 1070.

[0246] The prompting unit 1093 renders a portion of the decoded 3D data to provide a prompt.

[0247] Figure 11 The data generation unit 1071 can also be... Figure 12 The data generation unit 1110 shown is used to achieve this. Figure 12 This is a block diagram representing another example of the data generation department of a server.

[0248] The data generation unit 1110 includes a point group generation unit 1111, a grid generation unit 1112, and a model generation unit 1113.

[0249] The point group generation unit 1111 has the same function as the point group generation unit 1072. The point group generation unit 1111 acquires point group data from the point group sensor 1101 and a two-dimensional image from the camera 1102, and generates point group data based on the point group data and the two-dimensional image. The point group data generated by the point group generation unit 1111 includes position information of each point and attribute information (such as color information) extracted from the two-dimensional image, wherein the attribute information corresponds to each point indicated by the position information.

[0250] The mesh generation unit 1112 generates mesh data based on the point group data generated by the point group generation unit 1111.

[0251] The model generation unit 1113 has the same function as the model generation unit 1074. The model generation unit 1113 acquires point group data from the point group sensor 1101 and two-dimensional images from the camera 1102, and performs machine learning based on the point group data and the two-dimensional images to generate three-dimensional model data.

[0252] like Figure 11 As explained, point cluster data, grid data, and 3D model data can also be generated independently. For example... Figure 12 As explained, grid data can also be generated from point cluster data. Furthermore, point cluster data can also be generated from grid data.

[0253] Mesh can be generated from point groups, and point groups can also be generated from mesh.

[0254] Furthermore, point cluster data, mesh data, and 3D model data can be generated by the server 1070, or by sensors or a terminal 1090 equipped with sensors. Sensors include, for example, a point cluster sensor 1101 and a camera 1102.

[0255] Next, the relationship between three-dimensional space and coded data will be explained. Figure 13 It is a diagram used to illustrate the relationship between three-dimensional space and coded data.

[0256] As mentioned above, three-dimensional data includes, for example, point group data, grid data, and any of the three-dimensional models.

[0257] like Figure 13As shown, when 3D data is segmented into three 3D data units in three 3D spaces (tiles or spaces), the encoding device encodes each of the three segmented 3D data units separately and adds a header to perform data unitization. In the header, signaling notifies (assigns) the identifier (Space_ID) of the space to which the encoded data of the data unit belongs and the identifier (DataUnit_ID) of the data unit.

[0258] The data unit is further given a header containing the identifier of the data unit or the length information of the data unit, and the encoding method unit is generated through unitization.

[0259] Next, the syntax of the encoding method unit will be explained. Figure 14 This is a diagram representing an example of the syntax of an encoding unit. Figure 15 This is a diagram representing an example of the syntax of a group of encoded points. Figure 16 This is a diagram representing an example of the syntax of a coding grid. Figure 17 This is a diagram representing an example of the syntax for encoding a three-dimensional model.

[0260] `unit_type` indicates the category of the data unit stored in the encoding mode unit. Therefore, the category of the data unit stored in the encoding mode unit is specified.

[0261] length represents the length of a data unit.

[0262] data() represents the body of a data unit.

[0263] exist Figure 15 In this context, when `unit_type` is 0, it indicates that the data unit is the location information (geometric information) of a group of encoded points. When `unit_type` is 1, it indicates that the data unit is the attribute information of a group of encoded points. When `unit_type` is 2, it indicates that the data unit is the metadata of a group of encoded points.

[0264] exist Figure 16 In this context, when unit_type is 0, it indicates that the data unit is the location information (geometric information) of the encoded mesh. When unit_type is 1, it indicates that the data unit is the attribute information of the encoded mesh. When unit_type is 2, it indicates that the data unit is the metadata of the encoded mesh.

[0265] exist Figure 17 In this context, when `unit_type` is 0, it indicates that the data unit is element 1 encoding the 3D model. When `unit_type` is 1, it indicates that the data unit is element 2 encoding the 3D model. When `unit_type` is 2, it indicates that the data unit is metadata encoding the 3D model.

[0266] in addition, Figures 15-17 The syntax shown is an example and is not limited to the above configuration. These syntaxes can use a portion of the syntax, or types (categories) not mentioned above, and the order of the syntax's constituent elements can be rearranged. For example, in the syntax of the encoding mode unit, it can also be as follows: Figure 14 The configuration of the encoding scheme unit, which is common to multiple encoding schemes, is shown in that way. Figures 15-17 The unit_type, length, and data() shown are shown.

[0267] Additionally, a header can be assigned to the encoding method unit to indicate its category. Encoding unit categories include, for example, `point_cloud_codec_unit` for point cloud data, `mesh_codec_unit` for mesh data, and `model_codec_unit` for 3D model data. This allows for the comprehensive processing of multiple encoding methods.

[0268] Figure 18 This is a diagram illustrating an example of syntax for representing three-dimensional data information.

[0269] In syntax, when multiple encoding methods are saved in a single format, the number of 3D data items contained in that format (number_of_3Dformat) and the type of 3D data (format_type) can also be represented to save data in each format. Therefore, it is possible to comprehensively process multiple encoding methods or 3D data, and to recognize multiple encoding methods or 3D data.

[0270] 3Ddata_info represents the format structure information for storing multiple 3D data sets.

[0271] number_of_3Dformat indicates the number of 3D formats used.

[0272] `format_type` indicates the type of format of the saved 3D data. For example, `format_type` can be defined as a number and its corresponding format as follows: `format_type` = 0, indicating the saved 3D data is in point cloud format. `format_type` = 1, indicating the saved 3D data is in mesh format. `format_type` = 2, indicating the saved 3D data is in G-PCC format (g-pcc). `format_type` = 3, indicating the saved 3D data is in V-DMC format (v-dmc). `format_type` = 4, indicating the saved 3D data is in 3D model format.

[0273] Next, the data structure of the encoded data of multiple three-dimensional data will be explained according to each type of three-dimensional data. Figure 19 It is a diagram used to illustrate the data structure of a group of coded points. Figure 20 It is a diagram used to illustrate the data structure of the coded grid. Figure 21 It is a diagram used to illustrate the data structure of a coded 3D model.

[0274] For each type of three-dimensional data, the encoding device divides the three-dimensional data into multiple three-dimensional data according to each of the multiple spatial regions, and encodes the multiple three-dimensional data (i.e. multiple segmented three-dimensional data) separately to generate encoded data.

[0275] Each encoded data is assigned a header, and at least one of the data_unit_id and space_id is stored.

[0276] Here, `data_unit_id` is an identifier that identifies a data unit within the encoded data and is unique within the encoded data. Additionally, `space_id` represents identification information for a spatial region. If either `data_unit_id` or `space_id` is common across multiple 3D datasets, it represents the same value across all three-dimensional datasets.

[0277] exist Figures 19-21 In the example, data units with data_unit_id=0 in the encoded point group, data units with data_unit_id=3 in the encoded mesh, and data units with data_unit_id=0 in the encoded 3D model are all assigned space_id=1. This means that the 3D data is contained in a common 3D space represented by Space_ID#1.

[0278] Data, including headers, can be contained in bitstream structures such as data units or encoding methods, or stored in file formats specified by ISOBMFF, such as various BOXes.

[0279] Next, the three-dimensional spatial information will be explained. Figure 22 It is a diagram that represents an example of multiple three-dimensional spaces in a two-dimensional way. Figure 23 This is a diagram representing an example of a bounding box. Figure 24 This is a diagram illustrating an example of syntax for representing three-dimensional spatial information.

[0280] In the syntax of three-dimensional spatial information, 3Dspace_info represents information about the segmented three-dimensional space. 3Dspace_info can be used for partial decoding.

[0281] number_of_space represents the number of three-dimensional spaces after partitioning.

[0282] space_id represents the identifier of the segmented three-dimensional space.

[0283] Three-dimensional spatial information includes bounding box information as a specification Figure 23 Information about the bounding box shown.

[0284] The bounding box information includes bounding_box_xyz and bounding_box_whd.

[0285] `bounding_box_xyz` represents the coordinates of the reference point of the bounding box. Figure 23 In the example, the coordinates of x, y, and z (x0, y0, z0) are used to represent it.

[0286] `bounding_box_whd` represents the size of the bounding box. Figure 23 In the example, it can be represented by width w, height h, and depth d (w0, h0, d0).

[0287] Additionally, the three-dimensional spatial information may include an identifier for each data unit of the encoded data. Furthermore, the three-dimensional spatial information may also omit this identifier; that is, the identifier may not be notified by signaling.

[0288] pointcloud_id represents the identifier of the data unit of the coded point group in the space corresponding to space_id.

[0289] mesh_id represents the identifier of the data cell of the encoded grid in the space corresponding to space_id.

[0290] model_id represents the identifier of the data unit of the coded 3D model corresponding to space_id.

[0291] Furthermore, even when the data unit shows the data_unit_id instead of the space_id, the identifier of each encoded data unit can still be stored in the information of each space in the representation of the three-dimensional spatial information. This allows for the establishment of a correspondence between the three-dimensional spatial information and the segmented three-dimensional encoded data.

[0292] Alternatively, if the space_id is shown in the data unit, the three-dimensional spatial information can be mapped to an identifier for each data unit of the encoded data using the space_id. In this case, it is also possible not to store the identifier for each data unit of the encoded data.

[0293] Alternatively, the 3D spatial information of the point group data and the grid data can be commonalized by making the segmentation method, the origin of each segmented space, and the size of the bounding box the same in both the grid data and the point group data. Alternatively, the same 3D spatial information can be used separately in the point group data and the grid data. This allows for the commonalization of 3D spatial information across different types of 3D data, and the use of the same 3D spatial information. By commonalizing 3D spatial information, switching between different types of 3D data (e.g., switching prompts or switching transmissions) becomes easier. Furthermore, in formats that integrate multiple 3D data, it is not necessary to set 3D spatial information for each 3D data; instead, one set of 3D spatial information can be used for each 3D data, thus reducing the amount of 3D spatial information.

[0294] In addition to point group data and mesh data, it is also possible to synchronize the three-dimensional spatial information of the three-dimensional model with other types of three-dimensional data, and to make the three-dimensional spatial information of other types of three-dimensional data common.

[0295] Next, the relationship between the data structure of 3D data and partial decoding will be explained. Figure 25 This is a flowchart illustrating an example of partial decoding. Figure 26 This is a diagram representing an example of a three-dimensional spatial region that is partly decoded. Figure 27 This is a diagram representing an example of a data structure for a group of encoded points that has been partially decoded. Figure 28 This is a diagram representing an example of a data structure for a partially decoded encoded grid. Figure 29 This is a diagram representing an example of the data structure of a partially decoded encoded 3D model.

[0296] In partial decoding, firstly, the decoding device determines the three-dimensional spatial region of the object to be partially decoded (S1001).

[0297] Next, the decoding device uses three-dimensional spatial information (3Dspace_info) to determine the region that overlaps with the three-dimensional spatial region of the object based on the bounding box information of multiple three-dimensional spatial regions, and obtains the space_id corresponding to the determined region (S1002).

[0298] Next, the decoding device obtains a data unit with the obtained space_id from the encoded data and decodes it (S1003). Thus, the decoding device performs partial decoding, decoding only a portion of the three-dimensional data. In partial decoding, the decoding device does not decode the entire three-dimensional data, but only a portion of it.

[0299] For example, such as Figure 26 As shown, when the three-dimensional space region of the object being partially decoded is shown in thick lines, the space_id of the obtained three-dimensional space is determined to be #2 based on the three-dimensional space information.

[0300] Then, as Figures 27-29 As shown, the encoded data of various three-dimensional data were used to establish corresponding data units with Space_id=#2 and then decoded.

[0301] In addition, the decoding device can also obtain the data unit ID from the three-dimensional spatial information instead of obtaining the space_id, obtain the data unit with the obtained data unit ID, and perform partial decoding.

[0302] In the above embodiments, point group data, mesh data, and 3D model data are exemplified as 3D data representing 3D objects, but the methods are not limited to these. For example, a 3D object may also be represented by multiple groups, each containing line-of-sight information representing a line of sight and a 2D image obtained when viewing the 3D object from that line of sight. That is, data containing these multiple groups can also be processed as a type of 3D data. In addition, 3D data may also be data in other formats such as Gaussian splatting data.

[0303] Figure 30 This is a diagram illustrating an example of the configuration of a decoding device. Figure 31 This is a flowchart illustrating an example of a decoding method performed by a decoding device.

[0304] The decoding device 1130 includes a circuit 1131 and a memory 1132 connected to the circuit 1131.

[0305] Circuit 1131 performs the following actions.

[0306] Circuit 1131 acquires encoded data (S1021), the encoded data including: encoding method information (format), indicating one encoding method containing first data representing a three-dimensional object and second data representing the three-dimensional object; and identification information, indicating the three-dimensional space containing the three-dimensional object. Next, based on the encoded data, circuit 1131 decodes the first data and the second data corresponding to the three-dimensional space (S1022). Next, circuit 1131 renders the first data to generate first prompt data for prompting (S1023). Next, circuit 1131 renders the second data to generate second prompt data for prompting (S1024). Next, circuit 1131 switches from the generated second prompt data to the first prompt data for prompting (S1025). Furthermore, the first prompt data and the second prompt data are, for example, two-dimensional data or three-dimensional data generated by the rendering reconstruction unit 1034.

[0307] Therefore, based on the first and second data corresponding to the three-dimensional space, first and second prompt data are generated, and prompts are given by switching from the second prompt data to the first prompt data. This allows for prompting in a way that does not produce spatial deviation during the switching between the two data representing the three-dimensional object. Thus, the first and second prompt data can be appropriately prompted.

[0308] For example, the first data is point group data representing the three-dimensional object.

[0309] Therefore, since the prompt is given by switching from the second prompt data to the first prompt data based on the point group data, the prompt can be given in a way that does not produce spatial deviation in the switching between the two data representing the 3D object.

[0310] For example, the second data is mesh data representing the three-dimensional object.

[0311] Therefore, by switching from the second cue data based on grid data to the first cue data, the cue can be switched in a way that does not produce spatial deviation in the switching between the two data representing the 3D object.

[0312] For example, the second data is a 3D model data representing the 3D object. The 3D model data represents a machine learning model obtained by performing machine learning on multiple sets of viewpoints and 2D images.

[0313] Therefore, by switching from the second cue data based on the 3D model data to the first cue data, the cue can be switched in a way that does not produce spatial deviation when switching between the two data representing the 3D object.

[0314] For example, the second data is a two-dimensional image obtained when the three-dimensional object is viewed from a specified line of sight.

[0315] Therefore, by switching from the second cue data based on the two-dimensional image to the first cue data, it is possible to provide cueing in a way that does not produce spatial deviation in the switching between the two data representing the three-dimensional object.

[0316] For example, the circuit also receives a switching request for prompt data from the user. In the prompt, the circuit switches from the second prompt data to the first prompt data according to the switching request.

[0317] Therefore, it is possible to switch at a time specified by the user.

[0318] For example, the circuit also receives an operation from the user to change the style of the prompt. In the prompt, the circuit changes the style of the prompt according to the operation, switching from the second prompt data to the first prompt data based on the change.

[0319] Therefore, it can switch at timed intervals corresponding to user actions.

[0320] For example, in the acquisition process, the circuit acquires the encoded data from the encoding device via a communication network. In the prompting process, the circuit switches from the second prompting data to the first prompting data based on the bandwidth of the communication network.

[0321] Therefore, it can switch according to the bandwidth of the communication network. For example, when the bandwidth of the communication network changes from less than the specified bandwidth to more than the specified bandwidth, it can switch from the second prompt data to the first prompt data to provide a prompt.

[0322] For example, in the prompt, the circuit switches from the second prompt data to the first prompt data based on the capabilities of the circuit that is available.

[0323] Therefore, it is possible to switch according to the capability of the available circuit. For example, when the capability of the available circuit changes from less than the specified capability to more than the specified capability, it is possible to switch from the second prompt data to the first prompt data to provide a prompt.

[0324] For example, the encoded data includes synchronization information for synchronizing the coordinate system of the first data with the coordinate system of the second data. The circuit, in the prompt, provides prompts for both the first and second prompt data based on the synchronization information.

[0325] Therefore, it is possible to switch from the second prompt data to the first prompt data based on matching the coordinate systems of the first and second prompt data. Thus, it is possible to provide prompts in a way that minimizes spatial deviation during the switching between the two data representing a 3D object.

[0326] For example, the circuit further determines whether to synchronize the coordinate system of the first data with the coordinate system of the second data. If the circuit determines that it is necessary to synchronize the coordinate system of the first data with the coordinate system of the second data, the circuit provides a prompt in the prompt, based on the synchronization information, for both the first prompt data and the second prompt data.

[0327] Therefore, synchronous processing can be performed when needed and skipped when not needed. This could potentially reduce the processing load.

[0328] For example, the first data and the second data have a common structure in the first data and the second data, respectively.

[0329] Therefore, it is possible to reduce the amount of encoded data. Therefore, it is possible to reduce communication capacity.

[0330] For example, the encoded data includes spatial information for determining the three-dimensional space containing the three-dimensional object. The circuit also obtains an object region representing a portion of the three-dimensional space. Based on the spatial information, the circuit determines first overlapping data, which is a portion of the first data and overlaps with the object region. In the decoding, the circuit decodes the determined first overlapping data.

[0331] Therefore, for example, the amount of data acquired can be reduced by acquiring only the first overlapping data. This reduces communication capacity. Furthermore, for example, only the first overlapping data can be decoded. This reduces processing load.

[0332] Alternatively, circuit 1131 can also be like Figure 32 The decoding method shown in the flowchart is followed. Figure 32 This is a flowchart illustrating another example of a decoding method performed by a decoding device.

[0333] Circuit 1131 decodes the encoding information representing the three-dimensional object and representing a second encoding method different from the first encoding method of the first data (S1031). Circuit 1131 decodes the second data representing the second encoding method as indicated by the encoding information (S1032). The second data is used to generate second prompt data for prompting.

[0334] Therefore, by decoding the second data of the second encoding method represented by the encoding method information obtained through decoding, the second data for generating the second prompt data can be obtained.

[0335] Figure 33 This is a diagram illustrating an example of the configuration of an encoding device. Figure 34 This is a flowchart illustrating an example of an encoding method performed by an encoding device.

[0336] The encoding device 1140 includes a circuit 1141 and a memory 1142 connected to the circuit 1141.

[0337] Circuit 1141 performs the following actions.

[0338] Circuit 1141 generates encoding information representing the three-dimensional object and a second encoding method different from the first encoding method of the first data (S1041). Circuit 1141 generates second data representing the second encoding method as indicated by the encoding information (S1042). Circuit 1141 generates a bitstream containing the encoding information and the second data (S1043). The second data is used to generate second prompt data for prompting purposes.

[0339] Therefore, since a bitstream containing encoding method information and second data is generated, the decoding device that obtained the bitstream can obtain the second data for generating the second prompt data for appropriate prompts.

[0340] (Implementation Method 2) This paper describes a method for generating still images of a subject (three-dimensional object) observed from any viewpoint in still space using a learning-based model, i.e., a three-dimensional data generation model.

[0341] Figure 35 This is a diagram used to illustrate the processing during the learning of the three-dimensional generative model in Implementation Method 2. Figure 36 This diagram illustrates the process of generating a still image of a subject from any viewpoint using a three-dimensional generative model in Implementation 2.

[0342] Information processing devices acquire 3D data generation models through learning, thereby enabling the generation of still images observed from any viewpoint in static space. For example, there are 3D data generation models generated using methods such as Neural Radiance Fields (NeRF).

[0343] During learning, for example, the information processing device acquires learning data containing an image of viewpoint A (correct value) obtained from any viewpoint A, and viewpoint information (camera pose, etc.) of viewpoint A when the image was obtained. The viewpoint information may include viewpoint A and the direction of the line of sight from viewpoint A. The information processing device, for example, uses an evaluation function 1402 to optimize the parameters of the network included in the 3D data generation model, such that the difference between the generated image of viewpoint A output from the 3D data generation model 1401 by inputting the viewpoint information from the aforementioned learning data and the image of viewpoint A as the input image corresponding to viewpoint A is minimized. By performing this learning process using multiple learning data corresponding to multiple different viewpoints, the information processing device can obtain a more accurate 3D data generation model. Learning processing is performed on the learning data corresponding to each of the multiple viewpoints. That is, the same processing as the learning processing for viewpoint A is performed on each viewpoint.

[0344] During generation, if the information processing device inputs viewpoint information, such as viewpoint B, into the learned 3D data generation model 1403, it outputs a generated image of viewpoint B. If it inputs viewpoint information of viewpoint Z, which is different from viewpoint B, it outputs a generated image of viewpoint Z. The viewpoint information of viewpoint B may include viewpoint B and the viewing direction from viewpoint B. The viewpoint information of viewpoint Z may include viewpoint Z and the viewing direction from viewpoint Z.

[0345] Thus, by learning and obtaining a three-dimensional data generation model 1403, it is possible to generate still images observed from any viewpoint in static space. However, this alone cannot generate moving images.

[0346] In addition, Figure 36 The example shown is a 3D data generation model that generates an image of a given viewpoint if that viewpoint information is input. However, this is not a limitation, and the data output from the 3D data generation model can be in any form. For example, the 3D data generation model could also be a network model that outputs learned 3D data of the object space as point cluster data or mesh data. Thus, users can stereoscopically view 3D data such as point cluster data or mesh data in the object space, and can also use the point cluster data or mesh data to measure the dimensions of objects in the object space as 3D data output.

[0347] [Example 1] Figure 37This diagram illustrates the motion image generation method using the three-dimensional data generation model of Embodiment 1 in Embodiment 2. Furthermore, this embodiment describes an example of the configuration of an apparatus and a method for encoding or decoding the three-dimensional data generation models NNt0~NNt5 generated corresponding to times t0~t5. However, it is not limited to this; it can also be applied to an apparatus and method for encoding or decoding the three-dimensional data generation models at any time within any period.

[0348] This embodiment illustrates a method for generating a moving image of an object (subject) observed from any viewpoint using a three-dimensional data generation model. In this method, for example... Figure 37 As shown, by obtaining 3D data generation models corresponding to each time point, still images of an object observed from any viewpoint at each time point can be generated. By arranging the generated still images in chronological order, motion images can be generated. More specifically, in generating motion images for times t0 to t5, multiple 3D data generation models NNt0 to NNt5 corresponding to times t0 to t5 are generated through learning. The viewpoint information (camera pose, etc.) of the viewpoint A from which the motion images are to be generated is input into the generated 3D data generation models NNt0 to NNt5 corresponding to times t0 to t5. Thus, the generated images of viewpoint A at times t0 to t5 are output by the 3D data generation models NNt0 to NNt5. By concatenating these images in time, motion images of the object observed from viewpoint A at times t0 to t5 can be generated.

[0349] However, in this case, maintaining multiple 3D data generation models corresponding to multiple time points requires either a large storage capacity for storing the data of these multiple 3D data generation models on a storage device or a large network bandwidth for transmitting the data of these multiple 3D data generation models over a network. Therefore, the data size can also be reduced by using, for example, NNC (Neural Network Coding) in the MPEG (Moving Picture Experts Group) standard specification to encode the data of the multiple 3D data generation models corresponding to multiple time points. In this disclosure, a method for more effectively compressing this data is described.

[0350] NNC is shown in Non-Patent Document 1.

[0351] Figure 38 This is a diagram illustrating the first example of the configuration of the encoding device in Embodiment 1 of Embodiment 2.

[0352] The encoding device 1420 includes a three-dimensional data generation model acquisition unit 1421, a buffer unit 1422, and a network model encoding unit 1423.

[0353] The 3D data generation model acquisition unit 1421 acquires learning data from times t0 to t5, and uses this learning data to generate 3D data generation models NNt0 to NNt5 for times t0 to t5 through learning. The learning data includes multiple viewpoint images obtained by photographing an object from one or more viewpoint positions along one or more viewing directions at each time t0 to t5, and one or more viewpoint information representing one or more viewpoint positions and one or more viewing directions corresponding to the multiple viewpoint images. The one or more viewpoint information may also be the camera position and pose when each of the multiple viewpoint images was photographed. Furthermore, the learning data is not limited to this and may also include information obtained from other sensors. For example, the learning data may also include point cluster data and depth images obtained using LiDAR or TOF sensors at each time. This improves the accuracy of the 3D data generation model obtained through learning.

[0354] The buffer unit 1422 stores the three-dimensional data generation model at time t generated by the three-dimensional data generation model acquisition unit 1421. The buffer unit 1422 is implemented by a storage device such as a memory. The three-dimensional data generation model at time t stored in the buffer unit 1422 can also be used as an initial model when the three-dimensional data generation model acquisition unit 1421 acquires (generates) the three-dimensional data generation model after time t through learning. As a result, the learning time can be shortened and the accuracy of the three-dimensional data generation model after time t can be improved.

[0355] Furthermore, the buffer unit 1422 can also store multiple 3D data generation models corresponding to multiple time points. Therefore, for example, an initial model can be generated based on the multiple 3D data generation models stored in the buffer unit 1422, for example, through averaging or other processing. The 3D data generation model acquisition unit 1421 learns 3D data generation models after time t using this initial model, and can obtain a high-precision 3D data generation model. Furthermore, if the 3D data generation model acquisition unit 1421 does not refer to past 3D data generation models during learning, the encoding device 1420 may not need to include the buffer unit 1422. This reduces the amount of storage used as the buffer unit 1422.

[0356] The network model encoding unit 1423 encodes the three-dimensional data generation models NNt0~NNt5 obtained by the three-dimensional data generation model acquisition unit 1421 and outputs a bit stream.

[0357] Furthermore, as a network model encoding method, the data size can be reduced, for example, by using NNC data encoding from the MPEG standard. That is, the network model encoding unit 1423 uses NNC to encode the three-dimensional data generation models NNt0 to NNt5 and appends the encoding result to the bitstream. In other words, the network model encoding unit 1423 generates encoded data as the encoding result and generates a bitstream containing the encoded data.

[0358] Specifically, the network model encoding unit 1423 first encodes the 3D data generation model NNt0 at time t0 using NNC, and appends the encoding result to the bitstream. Next, the network model encoding unit 1423 encodes the 3D data generation model NNt1 at time t1 using NNC, and appends the encoding result to the bitstream. In this way, the network model encoding unit 1423 can also reduce the amount of encoding by sequentially encoding the 3D data generation model at each time using NNC and appending each encoding result to the bitstream.

[0359] Furthermore, at this time, the network model encoding unit 1423 can also append time information, representing the time corresponding to the encoded 3D data generation model, as metadata to the bitstream. Thus, by decoding and referencing the metadata contained in the bitstream, the decoding device can determine the time corresponding to the decoded 3D data generation model and appropriately generate motion images of the object from any viewpoint.

[0360] In addition, metadata is not limited to time information; it can also include information related to the acquisition (generation) of learning data, or information required by the decoding device to generate motion images.

[0361] For example, the network model encoding unit 1423 may also attach information related to the camera's frame rate when acquiring (generating) learning data as metadata. Thus, the decoding device can decode the frame rate of the generated motion image from the bitstream and appropriately set that frame rate.

[0362] Alternatively, the network model encoding unit 1423 may append the frame number corresponding to each time moment as metadata to the bitstream instead of the time information, and use other parameters to associate each frame number with the time information. For example, the network model encoding unit 1423 may append the time information and frame rate of the first frame as metadata, and the decoding device may calculate the time information of each frame based on this metadata, thereby reducing the amount of encoding corresponding to the time information of each frame.

[0363] Furthermore, the network model encoding unit 1423 can also append viewpoint information from viewpoint images used during learning to the bitstream. Thus, the decoding device can, for example, generate high-quality motion images by preferentially selecting viewpoints close to the viewpoint positions corresponding to the images used during learning. This is because the closer the viewpoint position or time is to the time of learning, the more likely the 3D data generation model is to generate higher-quality viewpoint images.

[0364] Figure 39 This is a diagram illustrating the first example of the configuration of the decoding device in Embodiment 1 of Embodiment 2.

[0365] The decoding device 1425 includes a network model decoding unit 1426 and a rendering unit 1427.

[0366] The network model decoding unit 1426 acquires the bit stream and, based on the acquired bit stream, decodes the three-dimensional data generation model NNt0~NNt5 and metadata such as time information for times t0~t5.

[0367] The rendering unit 1427 uses the 3D data generation models NNt0~NNt5 decoded by the network model decoding unit 1426 and metadata such as time information to generate motion images of viewpoint A based on viewpoint information specified by the user or system. Specifically, the rendering unit 1427 inputs the viewpoint information of viewpoint A into the 3D data generation model NNt0 at time t0 to generate image IMGt0 of viewpoint A at time t0. Next, it inputs the viewpoint information of viewpoint A into the 3D data generation model NNt1 at time t1 to generate image IMGt1 of viewpoint A at time t1. The rendering unit 1427 applies the generation processing of these images at each time to each time t2~t5 to generate images IMGt2~IMGt5 of viewpoint A at times t2~t5. Furthermore, the rendering unit 1427 uses images IMGt0~IMGt5 and metadata such as time information to generate motion images of objects observed from viewpoint A at times t0~t5. The moving image may include, for example, images IMGt0~IMGt5 and cue time information for calculating the cue times of images IMGt0~IMGt5 based on times t0~t5.

[0368] Furthermore, the viewpoint information can change according to time. For example, viewpoint information of viewpoint A can be input into the 3D data generation model NNt0~NNt3 at times t0~t3, and viewpoint information of viewpoint B can be input into the 3D data generation model NNt4~NNt5 at times t4~t5. Thus, the rendering unit 1427 generates multiple images of the object observed from viewpoint A at times t0~t3, and generates multiple images of the object observed from viewpoint B at times t4~t5. In other words, the rendering unit 1427 can generate motion images of the observed object, where the viewpoint switches from viewpoint A to viewpoint B at time t4.

[0369] Furthermore, the rendering unit 1427 does not necessarily need to generate moving images; it can also generate still images with specified viewpoint information at a specified time. Thus, the user can switch between generating moving images and generating still images depending on the application.

[0370] Furthermore, the rendering unit 1427 is not limited to generating moving or still images based on a 3D data generation model. For example, the rendering unit 1427 can also generate point group data or mesh data based on a 3D data generation model, and output the generated point group data or mesh data as dynamic point group data or dynamic mesh data. Thus, users can use dynamic 3D data of audiovisual objects such as HMDs (Head Mount Displays), and can also use the dynamic 3D data to measure the amount of motion of objects.

[0371] Figure 40 This is a second example of the configuration of the encoding device in Embodiment 1 of Embodiment 2.

[0372] The encoding device 1430 includes a three-dimensional data generation model acquisition unit 1431, a buffer unit 1432, a difference calculation unit 1433, and a network model encoding unit 1434.

[0373] The three-dimensional data generation model acquisition unit 1431 is the same as the three-dimensional data generation model acquisition unit 1421 of the encoding device 1420.

[0374] The buffer unit 1432 is the same as the buffer unit 1422 of the encoding device 1420, but it differs from the buffer unit 1422 in that it inputs the three-dimensional data generation model stored in the memory or the like as a reference three-dimensional data generation model into the difference calculation unit 1433.

[0375] The difference calculation unit 1433 calculates difference information, which represents the difference between the three-dimensional data generation models NNt0~NNt5 generated by the three-dimensional data generation model acquisition unit 1431 at times t0~t5 and the three-dimensional data generation models (hereinafter referred to as reference three-dimensional data generation models) generated by the three-dimensional data generation model acquisition unit 1431 before each time step. Here, the difference information may include the difference in the weight parameters of the nodes of each network model, etc. For example, the difference calculation unit 1433 obtains the three-dimensional data generation model NNt5 at time t5 from the three-dimensional data generation model acquisition unit 1431 and obtains the three-dimensional data generation model NNt4 at time t4 from the buffer unit 1432 as a reference three-dimensional data generation model.

[0376] Alternatively, the difference calculation unit 1433 can use the three-dimensional data generation model NNt5 and the three-dimensional data generation model NNt4, for example, to calculate the difference (change) between the weight parameters of the nodes in the network model NNt5 and the weight parameters of the nodes in the network model NNt4, and input the difference information representing this difference to the network model encoding unit 1434. Thus, the difference information is encoded by the network model encoding unit 1434. That is, the encoding device 1430 can also reduce the amount of data by performing predictive encoding on the difference between the predicted value and the information related to the network model in the three-dimensional data generation model NNt5 predicted by the three-dimensional data generation model NNt4. Through such predictive encoding, for example, in cases where the changes in the three-dimensional data generation model are small over time, such as when the object hardly moves, the value of the encoded difference becomes smaller, thus improving encoding efficiency. For example, the encoding device 1430 can also be set to RNNt0 = 0 and RNNtn = NNt(n-1) (n is an integer value from 1 to 5), using the previous three-dimensional data generation model as a reference three-dimensional data generation model, and reducing the number of bits through predictive coding.

[0377] Furthermore, in the second example, the encoding device 1430 performs predictive encoding on the information related to the network model in the three-dimensional data generation model NNt5 based on the information related to the network model in the three-dimensional data generation model NNt4, but is not limited to this. For example, the encoding device 1430 may select a reference three-dimensional data generation model for prediction from one or more three-dimensional data generation models stored in the buffer 1432, and perform predictive encoding using the selected three-dimensional data generation model. In this case, the encoding device 1430 may append information representing the selected three-dimensional data generation model (reference three-dimensional data generation model information) to the bitstream in order to pass the selected three-dimensional data generation model to the decoding device. Thus, the encoding device 1430 can select the optimal reference three-dimensional data generation model from the viewpoint of encoding efficiency, thereby improving encoding efficiency. Furthermore, by decoding the reference three-dimensional data generation model information, the decoding device can appropriately decode the bitstream, which has improved encoding efficiency.

[0378] Furthermore, when the encoding device 1430 performs predictive coding with reference to two or more three-dimensional data generation models stored in the buffer 1432, it can also append information representing the two or more reference three-dimensional data generation models to the bitstream. Thus, the encoding device 1430 can use two or more reference three-dimensional data generation models to improve the coding efficiency of predictive coding. Moreover, the decoding device can appropriately decode the bitstream with improved coding efficiency.

[0379] Furthermore, when the reference 3D data generation model is not stored in the buffer 1432, for example, when encoding the initial 3D data generation model (initial frame) in data order, the encoding device 1430 may encode the 3D data generation model of the processing object without calculating the difference from the predicted value (hereinafter referred to as intra-frame prediction), or it may encode by calculating the difference from the predicted value set to 0. Additionally, when the encoding device 1430 sets a certain time t as a random access point, it can encode the 3D data generation model corresponding to time t through intra-frame prediction, or it may encode by calculating the difference from the predicted value set to 0. Therefore, the decoding device can start decoding the 3D data generation model from the initial 3D data generation model (initial frame) or the random access point in data order, improving the functionality during playback.

[0380] Furthermore, a set of multiple 3D data generation models (multiple frames) can be defined (hereinafter referred to as GOF (Group of Frame)). The first frame of the GOF can also be encoded through intra-frame prediction. Thus, the decoding device can randomly access the first frame of the GOF. In addition, by decoding the first frame of the GOF, functionality such as fast-forward playback can be improved.

[0381] Furthermore, the encoding device 1430 may also append permission information indicating whether inter-GOF prediction referencing is permitted to the bitstream. For example, if the bitstream contains permission information indicating that inter-GOF prediction referencing is prohibited, the decoding device can determine that multiple GOFs can be decoded in parallel. Additionally, for example, by permitting inter-GOF prediction referencing, encoding efficiency can be improved.

[0382] The network model encoding unit 1434 is the same as the network model encoding unit 1423 of the encoding device 1420, but it differs in that it encodes the differential information d0~d5 of the three-dimensional data generation model NNt0~NNt5 input from the differential calculation unit 1433 and outputs a bit stream.

[0383] Furthermore, the encoding device 1430 includes a difference calculation unit 1433 and a network model encoding unit 1434 separately, but it is not limited to this. For example, it may be configured such that the difference calculation unit 1433 is included within the network model encoding unit 1434. That is, the network model encoding unit 1434 may also perform the processing of the difference calculation unit 1433.

[0384] Furthermore, the encoding device 1430 may also append prediction coding information to the bitstream, indicating whether the 3D data generation model was encoded using intra-frame prediction or using a reference 3D data generation model for prediction coding (hereinafter referred to as inter-frame prediction). Thus, by decoding the prediction coding information, the decoding device can appropriately determine whether intra-frame prediction or inter-frame prediction should be used to decode the 3D data generation model.

[0385] Figure 41 This is a second example of the configuration of the decoding device in Embodiment 1 of Embodiment 2.

[0386] The decoding device 1435 includes a network model decoding unit 1436, an addition unit 1437, a buffer unit 1438, and a rendering unit 1439.

[0387] The network model decoding unit 1436 acquires the bit stream and, based on the acquired bit stream, decodes the differential information d0~d5 and other metadata such as time information of the three-dimensional data generation model NNt0~NNt5 at times t0~t5.

[0388] The addition unit 1437 adds the difference information d0~d5 of the three-dimensional data generation model corresponding to times t0~t5, which is decoded by the network model decoding unit 1436, and the reference three-dimensional data generation model RNNt0~RNNt5 obtained from the buffer unit 1438 at the corresponding times to calculate the three-dimensional data generation model NNt0~NNt5. In this way, the decoding device 1435 can also be set to RNNt0 = 0 and RNNtn = NNt(n-1) (n is a value of 1~5), and use the three-dimensional data generation model of the previous time as the reference three-dimensional data generation model for prediction decoding.

[0389] Furthermore, the decoding device 1435 in the second example separately describes the addition unit 1437 and the network model decoding unit 1436, but it is not limited to this. For example, it could also be a structure in which the addition unit 1437 is included within the network model decoding unit 1436. That is, the network model decoding unit 1436 can also perform the processing of the addition unit 1437.

[0390] Furthermore, if the buffer 1438 does not store a reference 3D data generation model, for example, when decoding the initial 3D data generation model (the opening frame) in data order, the decoding device 1435 may perform decoding without adding the difference information to the reference 3D data generation model via the addition unit 1437 and without prediction (hereinafter referred to as intra-frame prediction), or it may add the prediction value set to 0 to the difference information for decoding. Additionally, if the decoding device 1435 sets a certain time t as a random access point, it can decode the 3D data generation model corresponding to time t using intra-frame prediction, or it may add the prediction value set to 0 to the difference information for decoding. Furthermore, if the bitstream contains prediction encoding information indicating that the 3D data generation model to be decoded has been encoded using intra-frame prediction, it can decode the 3D data generation model using intra-frame prediction, or it may add the prediction value set to 0 to the difference information for decoding. Therefore, the decoding device 1435 can begin decoding the 3D data generation model from the 3D data generation model that starts with the data sequence (starting frame), random access points, or 3D data generation models that have been encoded by intra-frame prediction, thereby improving the functionality during reproduction.

[0391] In addition, the decoding device 1435 in the second example performs predictive decoding on information related to the network model in the three-dimensional data generation model NNt5 based on information related to the network model in the three-dimensional data generation model NNt4, but is not limited to this. For example, the decoding device 1435 may also select a reference three-dimensional data generation model for prediction from one or more three-dimensional data generation models stored in the buffer 1438, and perform predictive decoding using the selected three-dimensional data generation model. In this case, the decoding device 1435 may also decode the information representing the selected three-dimensional data generation model (reference three-dimensional data generation model information) from the bitstream. Thus, the decoding device 1435 decodes the reference three-dimensional data generation model information from the bitstream generated by the encoding device 1430, which selects the reference three-dimensional data generation model that is optimal from the viewpoint of encoding efficiency, thereby enabling appropriate decoding of the bitstream that improves encoding efficiency.

[0392] Furthermore, when performing predictive decoding with reference to two or more three-dimensional data generation models stored in the buffer 1438, the decoding device 1435 can also decode information representing two or more reference three-dimensional data generation models from the bitstream. Thus, the decoding device 1435 can appropriately decode the bitstream, which improves the coding efficiency of predictive coding, using two or more reference three-dimensional data generation models.

[0393] The rendering unit 1439 is the same as the rendering unit 1427 of the decoding device 1425. The rendering unit 1439 does not necessarily need to generate moving images; it can also generate still images with specified viewpoint information at a specified time.

[0394] [Example 2] Figure 42 This diagram illustrates the motion image generation method using the extended three-dimensional data generation model of Embodiment 2 in Embodiment 2. Furthermore, this embodiment describes an example of the configuration and method of an apparatus for encoding or decoding extended three-dimensional data generation models NNt0-2 and NNt3-5, generated corresponding to periods t0-t2 and t3-t5 respectively, but it is not limited to this; it can also be applied to an apparatus and method for encoding or decoding extended three-dimensional data generation models in any period.

[0395] This embodiment illustrates a method for generating a moving image of an object (subject) observed from any viewpoint using a three-dimensional data generation model. In this method, for example, as... Figure 42In this way, by obtaining a three-dimensional data generation model (hereinafter referred to as the extended three-dimensional data generation model) capable of generating images from any viewpoint within a certain time range (period), it is possible to generate still images of objects observed from any viewpoint at any time within each period. By arranging the generated still images in chronological order, moving images can be generated. Similar to the three-dimensional data generation model in Example 1, the extended three-dimensional data generation model is, for example, a three-dimensional data generation model generated by methods such as NeRF.

[0396] More specifically, when generating motion images from time t0 to t5, an extended 3D data generation model NNt0-2 capable of representing the period from t0 to t2 and an extended 3D data generation model NNt3-5 capable of representing the period from t3 to t5 are generated through learning. The viewpoint information (camera pose, etc.) of the viewpoint A from which the motion images are to be generated is input into the generated extended 3D data generation models NNt0-2 and NNt3-5. Thus, the generated images of viewpoint A from time t0 to t5 are output by the extended 3D data generation models NNt0-2 and NNt3-5. By concatenating these images temporally, motion images from time t0 to t5, in which the object is observed from viewpoint A, can be generated.

[0397] However, in this case, maintaining the extended 3D data generation model corresponding to each period (time period) requires either a large storage capacity to store the extended 3D data generation model data in a storage device or a large network bandwidth to transmit the data of multiple 3D data generation models over a network. Therefore, the data size can also be reduced by using, for example, NNC (Neural Network Coding) in the MPEG (Moving Picture Experts Group) standard to encode the extended 3D data generation model corresponding to each period. In this disclosure, a method for more effectively compressing this data is described.

[0398] Furthermore, based on the above configuration, the information processing device can generate any viewpoint image at any time within the period t0-t5. For example, when acquiring the extended 3D data generation model NNt0-2, the information processing device generates the extended 3D data generation model NNt0-2 by using multi-viewpoint images captured at times t0, t1, and t2 as learning data, and by machine learning based on the camera poses corresponding to the multi-viewpoints. Moreover, when generating a motion image of viewpoint A, the information processing device can generate not only viewpoint images A at times t0, t1, and t2, but also images of any viewpoint at times t0.5 and t1.5 between times t0, t1, and t2. Time t0.5 is the time between time t0 and time t1, and time t1.5 is the time between time t1 and time t2.

[0399] Therefore, not only the time corresponding to the image during learning, the information processing device can also generate an image of any viewpoint corresponding to a time offset from the time corresponding to the image during learning, thus enabling the generation of motion images of viewpoint A at a high frame rate.

[0400] Furthermore, as learning data for the extended 3D data generation model NNt0-2, the information processing device can learn not only the learning data at times t0, t1, and t2, but also, for example, the learning data at time t3. Thus, it is possible to generate viewpoint images from any viewpoint after time t2 with high precision, such as an image from any viewpoint at time t2.5.

[0401] Furthermore, as learning data for the extended 3D data generation model NNt3-5, the information processing device can learn not only the learning data corresponding to times t3, t4, and t5, but also, for example, supplement the learning data corresponding to times t2 and t6. Thus, the information processing device can generate images from any viewpoint before time t3 or from any viewpoint after time t5 with high precision. Moreover, as a switching point of the extended 3D data generation model, for example, in the above example, when generating a viewpoint image at time 2.5 between time t2 and t3, which is the switching point between the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5, the information processing device can also generate viewpoint images at time t2.5 using both the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5, and generate the average image of the two generated viewpoint images at time t2.5 as the viewpoint image at time t2.5. Thus, a high-precision viewpoint image at time t2.5 can be generated.

[0402] In this way, by specifying the time and viewpoint information within the period corresponding to the extended three-dimensional data generation model, the information processing device can generate an image of the object observed from the specified viewpoint at the specified time.

[0403] Figure 43 This is a diagram illustrating the first example of the configuration of the encoding device in Embodiment 2 of Implementation 2.

[0404] The encoding device 1450 includes an extended three-dimensional data generation model acquisition unit 1451, a buffer unit 1452, and a network model encoding unit 1453.

[0405] The extended 3D data generation model acquisition unit 1451 acquires learning data for each period t0~t2 and t3~t5, from time t0 to t5. Using the acquired learning data for each period, it generates an extended 3D data generation model NNt0-2 for period t0~t2 and an extended 3D data generation model NNt3-5 for period t3~t5 through learning. The learning data includes multiple viewpoint images of an object captured from one or more viewpoint positions along one or more viewing directions at each time t0~t5, and one or more viewpoint information representing one or more viewpoint positions and one or more viewing directions corresponding to the multiple viewpoint images. The one or more viewpoint information may be the position and pose of the camera when capturing each of the multiple viewpoint images. Furthermore, the learning data is not limited to this and may also include information obtained from other sensors. For example, the learning data may also include point cluster data and depth images acquired at each time using a LiDAR or TOF sensor. As a result, the accuracy of the extended 3D data generation model obtained through learning can be improved.

[0406] The buffer unit 1452 stores the extended three-dimensional data generation model for the period tm-n, from time tm (m is an integer) to time tn (n is an integer greater than m), generated by the extended three-dimensional data generation model acquisition unit 1451. The buffer unit 1452 is implemented using a storage device such as a memory. The extended three-dimensional data generation model for the period tm-n stored in the buffer unit 1452 can also be used as an initial model when the extended three-dimensional data generation model acquisition unit 1451 acquires (generates) the extended three-dimensional data generation model for the period after tm-n through learning. As a result, the learning time can be shortened and the accuracy of the extended three-dimensional data generation model for the period after tm-n can be improved.

[0407] Furthermore, the buffer unit 1452 can also store multiple extended 3D data generation models corresponding to multiple periods. Thus, for example, an initial model can be generated based on the multiple extended 3D data generation models stored in the buffer unit 1452, for example, through averaging or other processing. The extended 3D data generation model acquisition unit 1451 learns extended 3D data generation models for periods after period tm-n by using this initial model, and can obtain a high-precision extended 3D data generation model. Furthermore, if the extended 3D data generation model acquisition unit 1451 does not refer to extended 3D data generation models of past periods during learning, the encoding device 1450 may not need to include the buffer unit 1452. This reduces the amount of storage used as the buffer unit 1452.

[0408] The network model encoding unit 1453 encodes the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5 obtained by the extended three-dimensional data generation model acquisition unit 1451 and outputs a bit stream.

[0409] Furthermore, as a network model encoding method, the data size can be reduced, for example, by using NNC data encoding from the MPEG standard. That is, the network model encoding unit 1453 uses NNC to encode the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5, and appends the encoding result to the bitstream. In other words, the network model encoding unit 1453 generates encoded data as the encoding result and generates a bitstream containing the encoded data.

[0410] Specifically, the network model encoding unit 1453 first uses NNC to encode the extended 3D data generation model NNt0-2 for periods t0 to t2, and appends the encoding result to the bitstream. Next, the network model encoding unit 1453 uses NNC to encode the extended 3D data generation model NNt3-5 for periods t3 to t5, and appends the encoding result to the bitstream. In this way, the network model encoding unit 1453 can sequentially encode the extended 3D data generation model for each period using NNC and append each encoding result to the bitstream, thereby reducing the amount of encoding.

[0411] Furthermore, at this time, the network model encoding unit 1453 can also append time information, representing the period to which the encoded extended 3D data generation model corresponds, as metadata to the bitstream. Thus, by decoding and referring to the metadata contained in the bitstream, the decoding device can determine which period the decoded extended 3D data generation model corresponds to, and can appropriately generate motion images of the object from any viewpoint.

[0412] Furthermore, the network model encoding unit 1453 can also generate information as time information indicating which period of viewpoint image the extended 3D data generation model can generate, and append the generated time information as metadata to the bitstream. Thus, in the decoding device, by decoding this metadata, the period during which the extended 3D data generation model can generate viewpoint images can be determined, and motion images can be generated appropriately.

[0413] In addition, metadata is not limited to time information; it can also include information related to the acquisition (generation) of learning data, or information required by the decoding device to generate motion images.

[0414] For example, the network model encoding unit 1453 may also attach information related to the camera's frame rate when acquiring (generating) the learning data as metadata. Thus, the decoding device can decode the frame rate of the generated motion image from the bitstream and appropriately set that frame rate.

[0415] Alternatively, the network model encoding unit 1453 may append the frame number corresponding to each period as metadata to the bitstream instead of the time information, and use other parameters to associate each frame number with the time information. For example, the network model encoding unit 1453 may append the time information and frame rate of the first frame as metadata, and the decoding device may calculate the time information of each frame based on this metadata, thereby reducing the amount of encoding corresponding to the time information of each frame.

[0416] Furthermore, the network model encoding unit 1453 can also append viewpoint information from viewpoint images used in the learning process, or time information indicating the time when the viewpoint image was captured, to the bitstream. Thus, for example, the decoding device can generate high-quality motion images by preferentially selecting viewpoints close to the viewpoint positions corresponding to images used in the learning process, or times close to the times corresponding to images used in the learning process. This is because the closer the viewpoint position or time is to the time of learning, the more likely the extended 3D data generation model is to generate higher-quality viewpoint images.

[0417] Figure 44 This is a diagram illustrating the first example of the configuration of the decoding device in Embodiment 2 of Implementation 2.

[0418] The decoding device 1455 includes a network model decoding unit 1456 and a rendering unit 1457.

[0419] The network model decoding unit 1456 acquires the bit stream and, based on the acquired bit stream, decodes metadata such as the extended three-dimensional data generation model NNt0-2 during period t0~t2 and the extended three-dimensional data generation model NNt3-5 during period t3~t5, as well as the time information corresponding to these extended three-dimensional data generation models NNt0-2 and NNt3-5.

[0420] The rendering unit 1457 uses metadata such as the extended 3D data generation models NNt0-2 and NNt3-5 decoded by the network model decoding unit 1456 and time information to generate motion images of viewpoint A based on viewpoint information specified by the user or system. Specifically, the rendering unit 1457 inputs the viewpoint information of viewpoint A and the time within the period t0~t2 into the extended 3D data generation model NNt0-2, and generates images IMGt0 of viewpoint A at time t0, IMGt1 of viewpoint A at time t1, and IMGt2 of viewpoint A at time t2. The rendering unit 1457 applies the image generation processing of the period t0~t2 to the extended 3D data generation model NNt3-5 for the period t3~t5 to generate images IMGt3~IMGt5 of viewpoint A at times t3~t5. Furthermore, the rendering unit 1457 uses metadata such as images IMGt0~IMGt5 and time information to generate motion images of the object observed from viewpoint A at times t0~t5. The motion image may, for example, include images IMGt0~IMGt5 and prompt time information for calculating prompt times based on the images IMGt0~IMGt5 at times t0~t5.

[0421] Furthermore, the viewpoint information can change according to time. For example, viewpoint information of viewpoint A can be input into the extended 3D data generation model NNt0-2 during the period t0~t2, and viewpoint information of viewpoint B can be input into the extended 3D data generation model NNt3-5 during the period t3~t5. As a result, the rendering unit 1457 generates multiple images of the object observed from viewpoint A during time t0~t2, and generates multiple images of the object observed from viewpoint B during time t3~t5. That is, the rendering unit 1457 can generate motion images of the object observed, with the viewpoint switching from viewpoint A to viewpoint B at time t3.

[0422] Furthermore, the rendering unit 1457 does not necessarily need to generate moving images; it can also generate still images with specified viewpoint information at a specified time. Thus, the user can switch between generating moving images and generating still images depending on the application.

[0423] Furthermore, the rendering unit 1457 is not limited to generating moving or still images based on the extended 3D data generation model. For example, the rendering unit 1457 can also generate point group data or mesh data that the extended 3D data generation model can represent, and output the generated point group data or mesh data as dynamic point group data or dynamic mesh data. Thus, users can use dynamic 3D data of audiovisual objects such as HMDs (Head Mount Displays), and can also use the dynamic 3D data to measure the amount of motion of objects.

[0424] Figure 45 This is a second example of the configuration of the encoding device in Embodiment 2 of Implementation 2.

[0425] The encoding device 1460 includes an extended three-dimensional data generation model acquisition unit 1461, a buffer unit 1462, a difference calculation unit 1463, and a network model encoding unit 1464.

[0426] The extended three-dimensional data generation model acquisition unit 1461 is the same as the extended three-dimensional data generation model acquisition unit 1451 of the encoding device 1450.

[0427] The buffer unit 1462 is the same as the buffer unit 1452 of the encoding device 1450, but it differs from the buffer unit 1452 in that it inputs the extended three-dimensional data generation model stored in the memory or the like as a reference extended three-dimensional data generation model into the differential calculation unit 1463.

[0428] The difference calculation unit 1463 calculates difference information, which represents the difference between the extended 3D data generation model NNt0-2 generated by the extended 3D data generation model acquisition unit 1461 for periods t0 to t2 and the extended 3D data generation model NNt3-5 for periods t3 to t5, and the extended 3D data generation model generated by the extended 3D data generation model acquisition unit 1461 before each period (hereinafter referred to as the reference extended 3D data generation model). Here, the difference information may include the difference of the weight parameters of the nodes of each network model, etc. For example, the difference calculation unit 1463 obtains the extended 3D data generation model NNt3-5 for periods t3 to t5 from the extended 3D data generation model acquisition unit 1461, and obtains the extended 3D data generation model NNt0-2 for periods t0 to t2 from the buffer unit 1462 as the reference extended 3D data generation model.

[0429] The difference calculation unit 1463 can also use extended 3D data generation models NNt3-5 and NNt0-2, for example, to calculate the difference (change) between the weight parameters of the nodes in the network model of extended 3D data generation model NNt3-5 and the weight parameters of the nodes in the network model of extended 3D data generation model NNt0-2, and input the difference information representing the difference to the network model encoding unit 1464. Thus, the difference information is encoded by the network model encoding unit 1464. That is, the encoding device 1460 can also reduce the amount of data by predictive encoding, which encodes the difference between the predicted value and the information related to the network model in extended 3D data generation model NNt3-5 based on the extended 3D data generation model NNt0-2. Through such predictive encoding, for example, in cases where the changes in the extended 3D data generation model are small over time, such as when the object hardly moves, the value of the encoded difference becomes smaller, thus improving encoding efficiency. For example, the encoding device 1460 can also be set to RNNt0-2 = 0 and RNNt3-5 = NNt0-2, and the extended three-dimensional data generation model of the previous time period can be used as a reference extended three-dimensional data generation model to reduce the number of bits through predictive coding.

[0430] Furthermore, in the second example, the encoding device 1460 performs predictive encoding on information related to the network model in the extended 3D data generation model NNt3-5 based on information related to the network model in the extended 3D data generation model NNt0-2, but is not limited to this. For example, the encoding device 1460 may select a reference extended 3D data generation model for prediction from one or more extended 3D data generation models stored in the buffer 1462, and perform predictive encoding using the selected extended 3D data generation model. In this case, in order to pass the selected extended 3D data generation model to the decoding device, the encoding device 1460 may also append information representing the selected extended 3D data generation model (reference extended 3D data generation model information) to the bitstream. Thus, the encoding device 1460 can select the optimal reference extended 3D data generation model from the viewpoint of encoding efficiency, thereby improving encoding efficiency. Furthermore, by decoding the reference extended 3D data generation model information, the decoding device can appropriately decode the bitstream with improved encoding efficiency.

[0431] Furthermore, when the encoding device 1460 performs predictive coding with reference to two or more extended three-dimensional data generation models stored in the buffer 1462, it can also append information representing the two or more reference extended three-dimensional data generation models to the bitstream. Thus, the encoding device 1460 can use two or more reference extended three-dimensional data generation models to improve the coding efficiency of predictive coding. Furthermore, the decoding device can appropriately decode the bitstream with improved coding efficiency.

[0432] Furthermore, when the reference extended 3D data generation model is not stored in the buffer 1462, for example, when encoding the initial extended 3D data generation model (initial frame) in data order, the encoding device 1460 can encode the extended 3D data generation model of the processing object without calculating the difference from the predicted value (hereinafter referred to as intra-frame prediction), or it can encode by calculating the difference from the predicted value set to 0. Additionally, when the encoding device 1460 sets a certain period tm-n as a random access point, it can encode the extended 3D data generation model corresponding to the period tm-n through intra-frame prediction, or it can encode by calculating the difference from the predicted value set to 0. Therefore, the decoding device can decode the extended 3D data generation model starting from the initial extended 3D data generation model (initial frame) or the random access point in data order, improving the functionality during playback.

[0433] Furthermore, a set of multiple extended 3D data generation models (multiple frames) can be defined (hereinafter referred to as GOF (Group of Frame)). The first frame of the GOF can also be encoded through intra-frame prediction. Thus, the decoding device can randomly access the first frame of the GOF. In addition, by decoding the first frame of the GOF, functionality such as fast-forward playback can be improved.

[0434] Furthermore, the encoding device 1460 may also append permission information indicating whether inter-GOF prediction referencing is permitted to the bitstream. For example, if the bitstream contains permission information indicating that inter-GOF prediction referencing is prohibited, the decoding device can determine that multiple GOFs can be decoded in parallel. Additionally, for example, by permitting inter-GOF prediction referencing, encoding efficiency can be improved.

[0435] The network model encoding unit 1464 is the same as the network model encoding unit 1453 of the encoding device 1450, but it differs in that it encodes the differential information d0-2 and d3-5 of the extended three-dimensional data generation models NNt0-2 and NNt3-5 input from the differential calculation unit 1463 and outputs a bit stream.

[0436] Furthermore, the encoding apparatus 1460 includes a difference calculation unit 1463 and a network model encoding unit 1464 separately, but it is not limited to this. For example, it may be configured to include the difference calculation unit 1463 within the network model encoding unit 1464. That is, the network model encoding unit 1464 may also perform the processing of the difference calculation unit 1463.

[0437] Furthermore, the encoding device 1460 may also append prediction coding information to the bitstream, indicating whether the extended 3D data generation model was encoded using intra-frame prediction or using a reference extended 3D data generation model for prediction coding (hereinafter referred to as inter-frame prediction). Thus, by decoding the prediction coding information, the decoding device can appropriately determine whether intra-frame prediction or inter-frame prediction should be used to decode the extended 3D data generation model.

[0438] Figure 46 This is a second example of the configuration of the decoding device in Embodiment 2 of Implementation 2.

[0439] The decoding device 1465 includes a network model decoding unit 1466, an addition unit 1467, a buffer unit 1468, and a rendering unit 1469.

[0440] The network model decoding unit 1466 acquires the bit stream and, based on the acquired bit stream, decodes the differential information d0-2, d3-5, and time information of the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5 during the period t0~t2.

[0441] The addition unit 1467 adds the difference information d0-2, d3-5 of the extended three-dimensional data generation models NNt0-2 and NNt3-5 corresponding to periods t0~t2 and t3~t5, decoded by the network model decoding unit 1466, and the reference extended three-dimensional data generation models RNNt0-2 and RNNt3-5 obtained from the buffer unit 1468 to the corresponding periods, and calculates the extended three-dimensional data generation models NNt0-2 and NNt3-5. In this way, the decoding device 1465 can also be set to RNNt0-2 = 0 and RNNt3-5 = NNt0-2, and use the extended three-dimensional data generation model of the previous time period as the reference extended three-dimensional data generation model for prediction decoding.

[0442] Furthermore, the decoding device 1465 in the second example separately describes an addition unit 1467 and a network model decoding unit 1466, but it is not limited to this. For example, it could be configured to include the addition unit 1467 within the network model decoding unit 1466. That is, the network model decoding unit 1466 could also perform the processing of the addition unit 1467.

[0443] Furthermore, if the buffer 1468 does not store a reference extended 3D data generation model, for example, when decoding the initial extended 3D data generation model (the first frame) in data order, the decoding device 1465 may perform decoding without adding the differential information to the reference extended 3D data generation model by the addition unit 1467 and without prediction (hereinafter referred to as intra-frame prediction), or it may add the prediction value set to 0 to the differential information for decoding. Additionally, if the decoding device 1465 sets a certain period tm-n as a random access point, it can decode the extended 3D data generation model corresponding to period tm-n through intra-frame prediction, or it may add the prediction value set to 0 to the differential information for decoding. Furthermore, if the bitstream contains prediction encoding information indicating that the extended 3D data generation model of the decoding target has been encoded through intra-frame prediction, it can decode the extended 3D data generation model through intra-frame prediction, or it may add the prediction value set to 0 to the differential information for decoding. Therefore, the decoding device 1465 can start decoding the extended three-dimensional data generation model from the extended three-dimensional data generation model (starting frame) that begins in data order, random access points, or extended three-dimensional data generation models that have been encoded by intra-frame prediction, thereby improving the functionality during reproduction.

[0444] Furthermore, in the second example, the decoding device 1465 performs predictive decoding on information related to the network model in the extended 3D data generation model NNt3-5 based on information related to the network model in the extended 3D data generation model NNt0-2, but is not limited to this. For example, the decoding device 1465 may also select a reference extended 3D data generation model for prediction from one or more extended 3D data generation models stored in the buffer 1468, and perform predictive decoding using the selected extended 3D data generation model. In this case, the decoding device 1465 may also decode information representing the selected extended 3D data generation model (reference extended 3D data generation model information) from the bitstream. Thus, the decoding device 1465 decodes the reference extended 3D data generation model information from the bitstream generated by the encoding device 1460, which selects the reference extended 3D data generation model that is optimal from the viewpoint of encoding efficiency, thereby enabling appropriate decoding of the bitstream with improved encoding efficiency.

[0445] Furthermore, when performing predictive decoding with reference to two or more extended three-dimensional data generation models stored in the buffer 1468, the decoding device 1465 can also decode information representing two or more reference extended three-dimensional data generation models from the bitstream. Thus, the decoding device 1465 can appropriately decode the bitstream that improves the coding efficiency of predictive coding using two or more reference extended three-dimensional data generation models.

[0446] The rendering unit 1469 is the same as the rendering unit 1427 of the decoding device 1425. The rendering unit 1469 does not necessarily need to generate moving images; it can also generate still images with specified viewpoint information at a specified time.

[0447] [Variation Example] Additionally, the encoding device 1460 may include information related to the number of images that the extended 3D data generation model can generate (i.e., the upper limit of the number of images) in the metadata attached to the bitstream of the extended 3D data generation model during the period tm-n. Thus, the decoding device 1465 can know the number of images that the decoded extended 3D data generation model can generate, for example, by appropriately setting the frame rate of the motion picture to be generated, and also, for example, by calculating the number of delayed frames until the motion picture is displayed.

[0448] Additionally, the encoding device 1460 can also append information indicating the time unit (i.e., the smallest time unit) to which the extended 3D data generation model can generate viewpoint images as time information appended to the bitstream. For example, as time information, the encoding device 1460 can append information such as whether the viewpoint image can be generated up to a time unit of 1 msec or 1 μmsec to the bitstream. Thus, the decoding device 1465 can determine the time unit to which the viewpoint information is generated and can accordingly generate high frame rate motion images or 3D data.

[0449] Furthermore, the encoding device 1460 can also append information about the extended 3D data generation model to the metadata of the bitstream attached to the extended 3D data generation model during the period tm-n. For example, by appending the timing information or viewpoint information of the image used for learning as metadata to the bitstream, the encoding device 1460 can, by decoding the metadata, know the timing or viewpoint information at which the extended 3D data generation model can generate viewpoint images with high quality, thereby enabling the production of high-quality motion pictures.

[0450] Furthermore, the width of the viewpoint image generated by the extended 3D data generation model can also be... Figure 47 The width can be switched dynamically as shown. Specifically, the width can also be switched according to the subject. Figure 47 This is a diagram illustrating a motion image generation method using an extended three-dimensional data generation model, which is used to explain a variation of Embodiment 2.

[0451] For example, in scenes with many stationary objects in the subject (scenes where the number of stationary objects in multiple subjects is a first or more, or scenes where the volume (area) occupied by stationary objects in multiple subjects is a first or more), the encoding device 1460 can generate an extended 3D data generation model with a long duration capable of generating viewpoint images with high image quality by expanding the width of the learning data used during learning (i.e., extending the duration). For example, in scenes with many moving objects in the subject (scenes where the number of moving objects in multiple subjects is a first or more, or scenes where the volume (area) occupied by moving objects in multiple subjects is a first or more), the encoding device 1460 can generate an extended 3D data generation model with a narrow duration capable of generating viewpoint images with high image quality, even for moving objects, by narrowing the width of the learning data used during learning.

[0452] Alternatively, the encoding device 1460 can use learning data for a certain period tm-n (e.g., a Group of Frames (GOF) representing a set of frames within period tm-n in a learning image) to generate an extended 3D data generation model NNtm-n for that period tm-n. In this case, the encoding device 1460 buffers the learning image frames for period tm-n to generate the extended 3D data generation model and performs compressed transmission, thus generating a transmission delay of the GOF size. The encoding device 1460 can also append information related to this transmission delay, such as the number of GOF frames and the number of delayed frames, to the bitstream. Therefore, the decoding device 1465 can obtain the delay information by decoding the bitstream and can appropriately reproduce the motion image or 3D data taking the delay into account.

[0453] Furthermore, the above embodiments illustrate an example of generating a still image of an arbitrary viewpoint at a given time or period using a 3D data generation model or an extended 3D data generation model, but are not necessarily limited to this. For example, other 3D data generation models, such as... Figure 48 As shown, it generates (outputs) 3D data such as point cluster data or mesh data at a specific moment within a certain period. This allows users to perform dimensional measurements of objects or obtain 3D data with higher audiovisual detail. Figure 48 This is a diagram used to illustrate a motion image generation method based on a modified example of embodiment 2 involving a three-dimensional data generation model.

[0454] Furthermore, encoding devices 1420 and 1460 can also include recommended output formats corresponding to use cases in the metadata of the bitstream, representing output formats such as image, point group data, and grid data. This allows the user to select the recommended output format based on the use case.

[0455] Furthermore, encoding devices 1420 and 1460 can also attach more than one viewpoint information to the metadata of the bitstream appended to the 3D data generation model or the extended 3D data generation model. For example, encoding devices 1420 and 1460 can consider including recommended viewpoint information for audiovisual objects or user viewpoint information when acquiring learning data in the metadata. Thus, decoding devices 1425 and 1465 can generate motion graphics or 3D data using viewpoint information selected from more than one viewpoint information appended to the bitstream based on user intent, etc.

[0456] Alternatively, a default viewpoint can be predetermined based on more than one viewpoint. Alternatively, if no user-specified viewpoint is provided, the decoding devices 1425 and 1465 can use the predetermined default viewpoint to generate motion images or 3D data. Thus, the decoding devices 1425 and 1465 can automatically generate motion images or 3D data even without user specification.

[0457] As an example of using this implementation method, there are the following usage methods.

[0458] First, the encoding devices 1420 and 1460 use cameras or sensors to acquire data of a dynamic object that they want to send to a distance, and use the data of the dynamic object as learning data to generate a three-dimensional data generation model or an extended three-dimensional data generation model of the dynamic object.

[0459] Next, the encoding devices 1420 and 1460 encode the three-dimensional data generation model or the extended three-dimensional data generation model using the encoding method described in this embodiment, and transmit the bit stream containing the encoding result to a remote location.

[0460] Then, decoding devices 1425 and 1465 decode the bitstream received from a distance, and use the decoded 3D data of the dynamic object to generate a 3D model or an extended 3D data generation model to generate motion images or 3D data from any viewpoint. The generated 3D data can then be used for appreciation or measurement purposes. In this way, this embodiment can also be applied to all use cases of remotely sharing information in a certain space.

[0461] Furthermore, when there are more than one object in a certain space that needs to be sent to a distant location, the 3D data generation modeling, encoding and transmission, decoding, and rendering processes described in this embodiment can be applied to each object separately. For example, dynamic objects in the foreground and static objects in the background existing in a certain space can be generated and modeled in 3D data and encoded and transmitted separately. As a result, the optimal 3D data generation modeling or encoding method can be applied to each object, thereby improving encoding efficiency.

[0462] Furthermore, it is not necessarily limited to this; multiple objects can also be treated as a single object, and the 3D data generation, modeling, encoding, transmission, decoding, and rendering processes described in this embodiment can be applied to each object separately. This allows for the transmission of multiple objects to a remote location while minimizing processing overhead.

[0463] Figure 49 This is a diagram illustrating an example of the configuration of the encoding device in Embodiment 2. Figure 50 This is a flowchart illustrating an example of the encoding method of the encoding device in Embodiment 2.

[0464] The encoding device 1470 includes circuitry 1471 and memory 1472. The encoding device 1470 is a device that implements the encoding devices 1420 and 1460.

[0465] Circuit 1471 performs the following actions.

[0466] Circuit 1471 acquires a first three-dimensional data generation model (e.g., three-dimensional data generation model NNt0) corresponding to a first time point (e.g., time t0) and a second three-dimensional data generation model (e.g., three-dimensional data generation model NNt1) corresponding to a second time point (e.g., time t1) (S1401). Circuit 1471 generates a bitstream by encoding the acquired first and second three-dimensional data generation models (S1402). The first and second three-dimensional data generation models output two-dimensional images of the subject as viewed from the viewpoint and the viewing direction, respectively, when input with viewpoint information including the viewpoint and the viewing direction.

[0467] Therefore, it is possible to generate a bitstream containing a first three-dimensional data generation model that generates a two-dimensional image corresponding to the first moment based on arbitrary viewpoint information and a second three-dimensional data generation model that generates a two-dimensional image corresponding to the second moment. Thus, it is possible to generate a compressed bitstream of the motion image data obtained from an arbitrary viewpoint. Therefore, it is possible to reduce the storage capacity used to store the motion image data obtained from an arbitrary viewpoint, or the network bandwidth used to transmit the data.

[0468] For example, the first three-dimensional data generation model and the second three-dimensional data generation model are learning models that use neural networks.

[0469] For example, the bit stream contains first-time information representing the first time point and second-time information representing the second time point.

[0470] For example, the bitstream includes a first frame number corresponding to the first time point and a second frame number corresponding to the second time point.

[0471] For example, the bitstream contains frame rate information related to the frame rate of multiple learning images used in the generation of the first 3D data generation model and the second 3D data generation model. The multiple learning images are 2D images obtained by capturing images at multiple different timings.

[0472] For example, the bitstream contains viewpoint information, which includes the viewpoint and line-of-sight direction of multiple learning images used in the generation of the first 3D data generation model and the second 3D data generation model.

[0473] For example, the plurality of learning images are two-dimensional images obtained by photographing the subject from mutually different viewpoints and viewing directions. The viewpoint information includes the mutually different viewpoints and viewing directions.

[0474] For example, circuit 1471 calculates differential information representing the difference between the first three-dimensional data generation model and the second three-dimensional data generation model in the encoding of the second three-dimensional data generation model. The bitstream contains the differential information.

[0475] For example, the difference includes a difference with corresponding weight parameters established for the nodes contained in the first three-dimensional data generation model and the second three-dimensional data generation model.

[0476] For example, the bitstream includes reference target information, which indicates that the difference information is calculated with reference to the first three-dimensional data generation model.

[0477] For example, the first time step corresponds to a random access point. The first 3D data generation model is encoded either by intra-frame prediction or by inter-frame prediction with a prediction value of 0.

[0478] For example, the first 3D data generation model and the second 3D data generation model are contained in one of a plurality of sets. The first 3D data generation model is the first in the data order among the plurality of 3D data generation models contained in the set.

[0479] For example, the bitstream contains licensing information indicating whether the 3D data generation model is permitted to reference other 3D data generation models in the respective encodings of the plurality of 3D data generation models.

[0480] For example, the first three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt0-2) corresponds to the first period (e.g., period t0~t2) that includes the first time point (e.g., time t0). The second three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt3-5) corresponds to the second period (e.g., period t3~t5) that includes the second time point (e.g., time t3).

[0481] For example, the multiple first learning images used in the generation of the first three-dimensional data generation model are two-dimensional images obtained by taking pictures at multiple different timings during the first period.

[0482] For example, when the first three-dimensional data generation model is input with a time contained in the first period, it outputs a two-dimensional image of the subject at the input time.

[0483] For example, the bitstream contains information indicating the upper limit of the number of images that the first 3D data generation model can generate.

[0484] For example, the bitstream contains first information associated with the plurality of first learning images. The first information includes multiple viewpoints and multiple line-of-sight directions corresponding to the plurality of first learning images, as well as multiple different timings.

[0485] For example, the first period or the second period is dynamically determined based on the subject.

[0486] For example, circuit 1471 saves the generated first three-dimensional data generation model in memory 1472. Based on the first three-dimensional data generation model saved in memory 1472, circuit 1471 generates the second three-dimensional data generation model.

[0487] For example, circuit 1471 stores the generated first 3D data generation model and the second 3D data generation model in memory 1472. Based on the first 3D data generation model and the second 3D data generation model stored in memory 1472, circuit 1471 generates an initial model. Based on the initial model, circuit 1471 generates a third 3D data generation model (e.g., 3D data generation model NNt2) corresponding to the third time step (e.g., time t2).

[0488] Figure 51 This is a diagram illustrating an example of the configuration of the decoding device in Embodiment 2. Figure 52 This is a flowchart illustrating an example of a decoding method of the decoding device in Embodiment 2.

[0489] The decoding device 1480 includes circuitry 1481 and memory 1482. The decoding device 1480 is a device that implements the decoding devices 1425 and 1465.

[0490] Circuit 1481 performs the following actions.

[0491] Circuit 1481 acquires the bitstream (S1411). Circuit 1481 decodes from the bitstream a first three-dimensional data generation model (e.g., three-dimensional data generation model NNt0) corresponding to a first time point (e.g., time t0) and a second three-dimensional data generation model (e.g., three-dimensional data generation model NNt1) corresponding to a second time point (e.g., time t1) (S1412). When the first three-dimensional data generation model and the second three-dimensional data generation model are input with viewpoint information including the viewpoint and the viewing direction, they output two-dimensional images of the subject as viewed from the viewpoint and the viewing direction, respectively.

[0492] Therefore, based on the compressed bitstream of motion image data obtained from any viewpoint, it is possible to decode the first three-dimensional data generation model that generates a two-dimensional image corresponding to the first moment and the second three-dimensional data generation model that generates a two-dimensional image corresponding to the second moment, based on arbitrary viewpoint information. Thus, it is possible to appropriately decode the bitstream that reduces the storage capacity for storing the motion image data obtained from any viewpoint or the network bandwidth for transmitting the data.

[0493] For example, the first three-dimensional data generation model and the second three-dimensional data generation model are learning models that use neural networks.

[0494] For example, the bit stream contains first-time information representing the first time point and second-time information representing the second time point.

[0495] For example, the bitstream includes a first frame number corresponding to the first time point and a second frame number corresponding to the second time point.

[0496] For example, the bitstream contains frame rate information related to the frame rate of multiple learning images used in the generation of the first 3D data generation model and the second 3D data generation model. The multiple learning images are 2D images obtained by capturing images at multiple different timings.

[0497] For example, the bitstream contains viewpoint information, which includes the viewpoint and line-of-sight direction of multiple learning images used in the generation of the first 3D data generation model and the second 3D data generation model.

[0498] For example, the plurality of learning images are two-dimensional images obtained by photographing the subject from mutually different viewpoints and viewing directions. The viewpoint information includes the mutually different viewpoints and viewing directions.

[0499] For example, the bitstream contains differential information representing the difference between the first three-dimensional data generation model and the second three-dimensional data generation model.

[0500] For example, the difference includes a difference with corresponding weight parameters established for the nodes contained in the first three-dimensional data generation model and the second three-dimensional data generation model.

[0501] For example, the bitstream includes reference target information, which indicates that the difference information is calculated with reference to the first three-dimensional data generation model.

[0502] For example, the first time step corresponds to a random access point. The first 3D data generation model is encoded either by intra-frame prediction or by inter-frame prediction with a prediction value of 0.

[0503] For example, the first 3D data generation model and the second 3D data generation model are contained in one of a plurality of sets. The first 3D data generation model is the first in the data order among the plurality of 3D data generation models contained in the set.

[0504] For example, the bitstream contains licensing information indicating whether the 3D data generation model is permitted to reference other 3D data generation models in the respective encodings of the plurality of 3D data generation models.

[0505] For example, the first three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt0-2) corresponds to the first period (e.g., period t0~t2) that includes the first time point (e.g., time t0). The second three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt3-5) corresponds to the second period (e.g., period t3~t5) that includes the second time point (e.g., time t3).

[0506] For example, the multiple first learning images used in the generation of the first three-dimensional data generation model are two-dimensional images obtained by taking pictures at multiple different timings during the first period.

[0507] For example, when the first three-dimensional data generation model is input with respect to the time included in the first period, it outputs a two-dimensional image of the subject at the input time.

[0508] For example, the bitstream contains information indicating the upper limit of the number of images that the first 3D data generation model can generate.

[0509] For example, the bitstream contains first information associated with the plurality of first learning images. The first information includes multiple viewpoints and multiple line-of-sight directions corresponding to the plurality of first learning images, as well as multiple different timings.

[0510] For example, the first period or the second period is dynamically determined based on the subject.

[0511] For example, circuit 1471 saves the generated first three-dimensional data generation model in memory 1472. Based on the first three-dimensional data generation model saved in memory 1472, circuit 1471 generates the second three-dimensional data generation model.

[0512] For example, circuit 1471 stores the generated first 3D data generation model and the second 3D data generation model in memory 1472. Based on the first 3D data generation model and the second 3D data generation model stored in memory 1472, circuit 1471 generates an initial model. Based on the initial model, circuit 1471 generates a third 3D data generation model (e.g., 3D data generation model NNt2) corresponding to the third time step (e.g., time t2).

[0513] (other) In one embodiment, a method for generating motion images from a predetermined viewpoint is disclosed. The generation of motion images is achieved, for example, by a device including a memory and circuitry connected to the memory. In one example of this device, a three-dimensional data generation model (Neural Network) generated through learning is stored in the memory, and the circuitry retrieves the stored three-dimensional data generation model (Neural Network) and generates motion images based on the three-dimensional data generation model. Alternatively, the three-dimensional data generation model or an extended three-dimensional data generation model may not be stored in memory. For example, encoding devices 1420 and 1460 may also retrieve specified information from a URL on a specified network and obtain the three-dimensional data generation model based on that specified information.

[0514] Figure 53 This is a diagram illustrating an example of the configuration of an encoding device.

[0515] The encoding device 1490 includes a processor 1491 and a memory 1492.

[0516] Processor 1491 is a circuit that performs information processing and is capable of accessing memory 1492. For example, processor 1491 is a dedicated or general-purpose electronic circuit that encodes a 3D data generation model. Processor 1491 can also be a processor like a CPU. Alternatively, processor 1491 can be an assembly of multiple electronic circuits. Furthermore, for example, processor 1491 can also function as multiple components of the aforementioned encoding device, excluding the component for storing information.

[0517] Memory 1492 is a dedicated or general-purpose memory that stores information used by processor 1491 to encode the 3D data generation model. Memory 1492 can be an electronic circuit or connected to processor 1491. Alternatively, memory 1492 can be contained within processor 1491. Alternatively, memory 1492 can be an assembly of multiple electronic circuits. Alternatively, memory 1492 can be a disk or optical disk, or it can be a storage device or recording medium. Alternatively, memory 1492 can be non-volatile memory or volatile memory.

[0518] For example, memory 1492 may store the encoded 3D data generation model, or it may store the stream corresponding to the encoded 3D data generation model. Additionally, memory 1492 may also store the program used by processor 1491 to encode the 3D data generation model.

[0519] Furthermore, in the encoding device 1490, it is possible to omit all of the aforementioned components of the encoding device, and it is also possible to omit all of the aforementioned processes. A portion of the components may be included in other devices, and a portion of the aforementioned processes may be performed by other devices.

[0520] Figure 54 This is a diagram illustrating an example of the configuration of a decoding device.

[0521] The decoding device 1495 includes a processor 1496 and a memory 1497.

[0522] Processor 1496 is a circuit that performs information processing and is capable of accessing memory 1497. For example, processor 1496 is a dedicated or general-purpose electronic circuit for decoding streams. Processor 1496 can also be a processor like a CPU. Alternatively, processor 1496 can be an assembly of multiple electronic circuits. Furthermore, for example, processor 1496 can also function as multiple components of the aforementioned decoding device, excluding the component for storing information.

[0523] Memory 1497 is a dedicated or general-purpose memory that stores information used by processor 1496 to decode the stream. Memory 1497 can be an electronic circuit or connected to processor 1496. Alternatively, memory 1497 can be contained within processor 1496. Alternatively, memory 1497 can be an assembly of multiple electronic circuits. Alternatively, memory 1497 can be a magnetic disk or optical disk, or it can be a storage device or recording medium. Alternatively, memory 1497 can be non-volatile memory or volatile memory.

[0524] For example, the memory 1497 can store a 3D data generation model or a stream. Furthermore, the memory 1497 can also store a program for the processor 1496 to decode the stream.

[0525] Furthermore, in the decoding device 1495, it is possible to omit all of the aforementioned components of the decoding device, and also to omit all of the aforementioned processes. A portion of the components may be included in other devices, and a portion of the aforementioned processes may be performed by other devices.

[0526] (Implementation Method 3) This section describes other methods for generating still images of a subject (3D object) observed from any viewpoint in still space using a learning-based model, i.e., a 3D data generation model.

[0527] In the above embodiments, examples of methods are shown, including: generating an extended 3D data generation model capable of generating images from any viewpoint within a certain time range (period); generating still images observed from any viewpoint at any time; generating moving images by arranging them in chronological order; and encoding or decoding the extended 3D data generation model. In this embodiment, an example is shown of generating a more accurate extended 3D data generation model by also learning a latent code Zt at time t when the extended 3D data generation model is obtained through learning; and encoding the extended 3D data generation model and the latent code.

[0528] Figure 55 This is a diagram used to illustrate the processing during the learning of the three-dimensional data generation model in Implementation Method 3. Figure 56 This diagram illustrates the process of generating a still image of a subject observed from any viewpoint using a three-dimensional data generation model in Implementation 3.

[0529] Similar to the embodiments described above, the information processing device acquires a three-dimensional data generation model through learning, thereby enabling the generation of still images observed from any viewpoint in static space. For example, there are three-dimensional data generation models generated by methods such as Neural Radiance Fields (NeRF).

[0530] During learning, for example, the information processing device acquires learning data containing, for instance, an image of viewpoint A (correct value) acquired from any viewpoint A at any time t, viewpoint information of viewpoint A at the time the image was acquired (camera pose, etc.), and a latent code Zt corresponding to time t. The viewpoint information may include viewpoint A and the direction of the line of sight from viewpoint A. The information processing device, for example, uses an evaluation function 1502 to optimize the parameters of the network included in the extended 3D data generation model 1501 and the latent code, such that the difference between the generated image of viewpoint A output from the extended 3D data generation model 1501 by inputting the viewpoint information from the aforementioned learning data and the image of viewpoint A as the input image of viewpoint A at time t is minimized. In other words, the information processing device learns the 3D data generation model at time t using the latent code Zt set for any time t.

[0531] The information processing device can obtain a more accurate 3D data generation model by using more than one learning data corresponding to more than one time point and more than one viewpoint. Learning processing is performed on the learning data corresponding to each of the multiple viewpoints. That is, the same processing as that performed on viewpoint A is performed on each viewpoint.

[0532] Furthermore, the latent code can be a one-dimensional value or a multi-dimensional vector with n dimensions (n ​​being an integer greater than 2). For example, a latent code with 256 dimensions can be represented as a vector of 256 values, such as the latent code at time t, Zt = (value 0t, value 1t, ..., value 255t), representing information from 256 values ​​from value 0t to value 255t. Thus, by representing the latent code as a multi-dimensional vector, the characteristics at each time point can be further represented.

[0533] Furthermore, the initial value of the latent code used for learning can be set to 0 or a random value. When the latent code is a multi-dimensional vector, all components can be set to 0 or random values. This reduces the processing required to calculate the initial value.

[0534] Furthermore, the initial values ​​for the latent codes used for learning are not limited to those described above; they can also be set to the values ​​of the already generated extended 3D data generation model and its associated latent codes. That is, the initial values ​​of the latent codes can be set to the values ​​of latent codes corresponding to times different from the time corresponding to the latent code itself, or they can be set to the values ​​of the already learned latent codes. Specifically, when learning the extended 3D data generation model and latent codes for times t5 to t9, if the extended 3D data generation model and latent codes for times t0 to t4 have already been learned, the model parameters of the already learned extended 3D data generation model for times t0 to t4 can be used as the initial values ​​for learning the extended 3D data generation model for times t5 to t9, and the latent codes for times t0 to t4 can be used as the initial values ​​for learning the latent codes for times t5 to t9. This allows for the generation of extended 3D data generation models and latent codes with high accuracy.

[0535] Furthermore, when there is little change in the object, such as minimal motion between times t0~t4 and t5~t9, the values ​​of the already generated extended 3D data generation model and its associated latent code can be set to initial values. This allows for the high-precision generation of extended 3D data generation models and latent codes in scenes with minimal motion.

[0536] Next, during generation, if the learned extended 3D data generation model 1503 is input with viewpoint information for viewpoint B and the latent code Zt0 corresponding to time t0, the generated image of viewpoint B at time t0 is output. If the input is viewpoint information for viewpoint Z (different from viewpoint B) and the latent code Zt4 corresponding to time t4, the generated image of viewpoint Z at time t4 is output. The viewpoint information for viewpoint B can include viewpoint B and the viewing direction from viewpoint B. The viewpoint information for viewpoint Z can include viewpoint Z and the viewing direction from viewpoint Z.

[0537] In this way, the information processing device learns and acquires an extended three-dimensional data generation model 1503 and a latent code, thereby enabling it to generate still images observed from any viewpoint within a certain time range. Furthermore, the information processing device can generate multiple still images corresponding to multiple times and generate moving images by arranging them in chronological order.

[0538] In addition, Figure 56The example shown is an extended 3D data generation model that generates an image of a viewpoint at a given moment when viewpoint information and a latent code are input. However, this is not a limitation, and the data output from the extended 3D data generation model can be in any form. For example, the extended 3D data generation model can also be a network model (network) that outputs a 3D model of the object space at a given moment, obtained through learning, in the form of point cluster data or mesh data. Thus, the user can stereoscopically view 3D data such as point cluster data or mesh data of the object space at a given moment, or can use point cluster data or mesh data to measure the size of objects in the object space at a given moment as 3D data output. Furthermore, the viewpoint image, generated image, and still image in this embodiment can also represent 2D images.

[0539] [Example 1] Figure 57 This diagram illustrates the motion image generation method using the extended three-dimensional data generation model of Embodiment 1 in Implementation Method 3. Furthermore, in this embodiment, an example of the configuration and method of an apparatus for encoding or decoding extended three-dimensional data generation models NNt0-2 and NNt3-5, and latent codes Zt0-Zt2 and Zt3-Zt5, which correspond to the periods t0-t2 and t3-t5 respectively, will be described. However, this is not necessarily limited to this; the apparatus and method can also be applied to encoding or decoding extended three-dimensional data generation models for any period.

[0540] This embodiment illustrates a method for generating a moving image of an object (subject) observed from any viewpoint using a three-dimensional data generation model. In this method, for example, as... Figure 57 Therefore, by acquiring a 3D data generation model capable of generating images from any viewpoint within a certain time range (hereinafter referred to as an extended 3D data generation model), it is possible to generate still images of objects observed from any viewpoint at any time within each period, and to generate moving images by arranging multiple generated still images in chronological order. An extended 3D data generation model is, for example, a 3D data generation model generated using methods such as NeRF.

[0541] More specifically, when generating motion images from time t0 to t5, an extended 3D data generation model NNt0-2 and latent codes Zt0-Zt2 representing the period from t0 to t2, and an extended 3D data generation model NNt3-5 and latent codes Zt3-Zt5 representing the period from t3 to t5 are generated through learning. The viewpoint information (camera pose, etc.) of the viewpoint A from which the motion images are to be generated, and the latent code Zt0-5 corresponding to time t0 to t5, are input into the generated extended 3D data generation models NNt0-2 and NNt3-5. Thus, the generated images of viewpoint A from time t0 to t5 are output by the extended 3D data generation models NNt0-2 and NNt3-5. By concatenating these images temporally, motion images of objects observed from viewpoint A from time t0 to t5 can be generated.

[0542] Furthermore, in this embodiment, an example is used to illustrate the encoding and decoding of extended three-dimensional data generation models NNt0-2 and latent codes Zt0-Zt2, as well as extended three-dimensional data generation models NNt3-5 and latent codes Zt3-Zt5, generated during periods t0-t2 and t3-t5 respectively, but it is not limited to this and can also be applied to the encoding and decoding of extended three-dimensional data generation models for any period.

[0543] However, in this case, maintaining the extended 3D data generation model and latent code corresponding to each period (time period) requires either a large storage capacity for storing the extended 3D data generation model data in a storage device or a large network bandwidth for transmitting the data of multiple 3D data generation models over a network. Therefore, the data size can be reduced by, for example, encoding the extended 3D data generation model corresponding to each period using NNC (Neural Network Coding) in the MPEG (Moving Picture Experts Group) standard. Alternatively, the data size can be reduced by arithmetic encoding the latent code. In this disclosure, a method for more effectively compressing this data is described.

[0544] Furthermore, based on the above configuration, the information processing device can generate any viewpoint image at any time within the period t0-t5. For example, when acquiring the extended 3D data generation model NNt0-2 and the latent codes Zt0-Zt2, the information processing device generates the extended 3D data generation model NNt0-2 and the latent codes Zt0-Zt2 by using multi-viewpoint images captured at times t0, t1, and t2 as learning data, and by machine learning based on the camera poses corresponding to the multi-viewpoints and the latent codes Zt0-Zt2. Moreover, when generating a motion image of viewpoint A, the information processing device can generate not only viewpoint images A at times t0, t1, and t2, but also images of any viewpoint at times t0.5 and t1.5 between times t0, t1, and t2. Time t0.5 is the time between time t0 and time t1, and time t1.5 is the time between time t1 and time t2.

[0545] Therefore, not only the time corresponding to the image during learning, the information processing device can also generate an image of any viewpoint corresponding to a time offset from the time corresponding to the image during learning, thus enabling the generation of motion images of viewpoint A at a high frame rate.

[0546] Furthermore, in this case, the information processing device can also use the learned latent codes to calculate the latent codes Zt0.5, Zt1.5, and Zt2.5 corresponding to times t0.5, t1.5, and t2.5. For example, the information processing device can also use latent codes Zt0 and Zt1 to calculate the latent code Zt0.5. Specifically, the information processing device can calculate Zt0.5 based on the average of Zt0 and Zt1. In this way, the information processing device calculates the latent code corresponding to a certain time based on the latent codes that have already been calculated, thereby reducing the data size by not needing to maintain the latent codes for all times. In addition, as the learning data for the extended 3D data generation model NNt0-2 and the latent codes Zt0~Zt2, the information processing device can also add learning data for time t3, for example, in addition to the learning data for times t0, t1, and t2. As a result, the information processing device can generate viewpoint images after time t2 with high accuracy, such as the viewpoint image at time 2.5.

[0547] Furthermore, the learning data for the extended 3D data generation model NNt3-5 and the latent codes Zt3~Zt5 are not only the learning data corresponding to times t3, t4, and t5, but the information processing device can also add learning data corresponding to times t2 and t6 for learning. Thus, the information processing device can generate images from any viewpoint before time t3 or from any viewpoint after time t5 with high precision.

[0548] Furthermore, as a switching point between the extended 3D data generation models, for example, in the case described above, when generating a viewpoint image at time 2.5 between the time t2 and t3 of the switching between the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5, the information processing device can also generate viewpoint images at time t2.5 using both the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5, and generate the average of the two generated viewpoint images at time t2.5 as the viewpoint image at time t2.5. This allows for the generation of a high-precision viewpoint image at time t2.5.

[0549] Furthermore, the information processing device can also use latent codes Zt2 and Zt3 to calculate latent code Zt2.5. Specifically, the information processing device can calculate latent code Zt2.5 based on the average of latent codes Zt2 and Zt3. In this way, the information processing device calculates the latent code corresponding to a certain time based on the latent codes that have already been calculated, thereby eliminating the need to maintain the latent codes for all times and reducing the data size.

[0550] In this way, by specifying the latent code and viewpoint information corresponding to the time within the period corresponding to the extended three-dimensional data generation model, the information processing device can generate an image of the object observed from the specified viewpoint at the time corresponding to the specified latent code.

[0551] Figure 58 This is a diagram illustrating the structure of the encoding device of Embodiment 1 in Embodiment 3, as shown in the first example.

[0552] The encoding device 1510 includes an extended three-dimensional data generation model acquisition unit 1511, a buffer unit 1512, a network model encoding unit 1513, and a latent code encoding unit 1514.

[0553] The extended 3D data generation model acquisition unit 1511 acquires learning data for each period t0~t2 and t3~t5, from time t0 to t5. Using the acquired learning data for each period, it generates an extended 3D data generation model NNt0-2 and latent codes Zt0~Zt2 for period t0~t2, and an extended 3D data generation model NNt3-5 and latent codes Zt3~Zt5 for period t3~t5 through learning. The learning data includes multiple viewpoint images of an object captured from one or more viewpoint positions along one or more viewing directions at each time t0~t5, more than one viewpoint information representing one or more viewpoint positions and one or more viewing directions corresponding to the multiple viewpoint images, and latent codes Zt0~Zt5 corresponding to each time t0~t5. The more than one viewpoint information may be the position and pose of the camera when capturing each image of the multiple viewpoint images. In addition, the learning data is not limited to this and may also include information obtained from other sensors. For example, the learning data may also include point cluster data and depth images acquired at each time using LiDAR or TOF sensors. This can improve the accuracy of the extended 3D data generation model obtained through learning.

[0554] The buffer unit 1512 stores the extended three-dimensional data generation model and latent codes Ztm to Ztn for the period tm-n from time tm (m is an integer) to time tn (n is an integer greater than m), generated by the extended three-dimensional data generation model acquisition unit 1511. The buffer unit 1512 is implemented by a storage device such as a memory. The extended three-dimensional data generation model and latent codes Ztm to Ztn for the period tm-n stored in the buffer unit 1512 can also be used as the initial model or latent code when the extended three-dimensional data generation model acquisition unit 1511 acquires (generates) the extended three-dimensional data generation model for the period after tm-n through learning. As a result, the learning time can be shortened and the accuracy of the extended three-dimensional data generation model for the period after tm-n can be improved.

[0555] Furthermore, the buffer unit 1512 can also store multiple extended 3D data generation models and multiple latent codes corresponding to multiple periods. Therefore, for example, an initial model and an initial latent code can be generated based on the multiple extended 3D data generation models and multiple latent codes stored in the buffer unit 1512, for example, through averaging or other processing. The extended 3D data generation model acquisition unit 1511 can obtain a high-precision extended 3D data generation model by using this initial model and the initial latent code to learn the extended 3D data generation model for the period after period tm-n.

[0556] Furthermore, if the extended 3D data generation model acquisition unit 1511 does not refer to the extended 3D data generation model and latent code from the past period during learning, the encoding device 1510 may not need to include the buffer unit 1512. This reduces the amount of storage used as the buffer unit 1512.

[0557] The network model encoding unit 1513 uses the method described in the above embodiment to encode the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5 obtained by the extended three-dimensional data generation model acquisition unit 1511 and outputs a bit stream.

[0558] Alternatively, as a network model encoding method, NNC from the MPEG standard can be used to encode the input network model to reduce the data size. Specifically, the network model encoding unit 1513 uses NNC to encode the extended 3D data generation models NNt0-2 and NNt3-5, and appends the encoding result to the bitstream. In other words, the network model encoding unit 1513 generates encoded data as the encoding result and generates a bitstream containing the encoded data.

[0559] Alternatively, as a network model encoding method, NNC (Neural Network Coding) in the MPEG standard can be used to encode the input network model to reduce the data size.

[0560] Furthermore, the network model encoding method is not limited to extended 3D data generation models; it can also correspond to the encoding of other network models. Therefore, it is not necessary to prepare a network model encoding unit 1513 according to the network model of the encoding object, thus reducing storage requirements and circuit size. Additionally, the network model is sometimes simply referred to as the network.

[0561] The latent code encoding unit 1514 encodes the latent codes Zt0 to Zt5 output from the extended 3D data generation model acquisition unit 1511 and appends them to the bitstream. Furthermore, as an encoding method for the latent codes, for example, when the latent codes are Zt = (value0t, value1t, ..., value255t), the latent code encoding unit 1514 can binarize the values ​​of each of value0t to value255t and perform arithmetic encoding on each of the binarized bits by assigning context. Therefore, when the values ​​of value0t to value255t are close, encoding efficiency can be improved.

[0562] Furthermore, when there is no correlation between the values ​​of value 0t to value 255t, the latent code encoding unit 1514 may not assign context to each bit after binarization, but instead use bypass mode for arithmetic encoding. This reduces the number of context bits.

[0563] Furthermore, the latent code encoding unit 1514 can also allocate context to a portion of each bit after binarization, and perform arithmetic encoding on the other portion using a bypass mode. This allows for a reduction in the number of context bits while improving encoding efficiency.

[0564] Furthermore, the encoding method for the latent code is not limited to this; any encoding method can be used. For example, the latent code encoding unit 1514 can also map the latent code to pixels of a two-dimensional image and encode it using image coding. As a more specific example, if the image codec is a codec corresponding to YUV 420, the latent code encoding unit 1514 can assign the values ​​0t to 255t of the latent code Zt = (value 0t, value 1t, ..., value 255t) to the 256 pixels of the Y component, and for the pixel values ​​of the other YUV components, set a certain value, such as 0 or half of the maximum possible value, to perform image coding on the YUV 420 image. Thus, by applying predictive coding or arithmetic coding inherent in image coding, coding efficiency can be improved.

[0565] Furthermore, the latent code encoding unit 1514 can also append the dimension of the latent code to the bitstream. This allows the decoding device to identify where the latent code value ends in the Y component, enabling correct decoding of the latent code. Similarly, when using image codecs corresponding to YUV 444 or YUV 400, the latent code encoding unit 1514 can map the latent code to a portion of the Y, U, or V components for image encoding. Therefore, regardless of the image codec used, the latent code can be encoded as an image.

[0566] Furthermore, the latent code encoding unit 1514 can also use lossless encoding when encoding the latent code. Therefore, the decoding device can recover the latent code having the same value as the encoding device.

[0567] Furthermore, the latent code encoding unit 1514 may also append information indicating whether the latent code has been encoded using arithmetic encoding or whether it has been encoded using a picture codec to the bitstream. Thus, the decoding device can switch between arithmetic decoding and picture decoding of the latent code based on the value appended to the bitstream, and can appropriately decode the bitstream.

[0568] Furthermore, the latent code encoding unit 1514 can also apply predictive coding to the encoding of the latent code. For example, when the latent code is a multi-dimensional vector such as Zt = (value 0t, value 1t, ..., value 255t), predictive coding can be applied between the components of the vector. More specifically, for example, the latent code encoding unit 1514 can encode the difference value obtained by subtracting value 0t from value 1t instead of directly encoding the value of value 1t. Similarly, the latent code encoding unit 1514 can encode the difference value obtained by subtracting value 1t from value 2t instead of directly encoding the value of value 2t. That is, when encoding the value of a component A, the latent code encoding unit 1514 can use the values ​​of one or more components that have been encoded or decoded before component A to calculate the predicted value P of component A, and encode the value obtained by subtracting the predicted value P from the value of component A. Thus, encoding efficiency can be improved when the correlation between the components of the latent code is high.

[0569] Furthermore, the predicted value P can be calculated based on the average, minimum, or maximum value of one or more components encoded or decoded before component A. This improves the accuracy of the predicted value P. In this way, the latent code encoding unit 1514 can sequentially encode the latent codes for each time period and append each encoding result to the bitstream obtained by encoding the network model, thereby reducing the amount of coding.

[0570] Additionally, at this time, the encoding device 1510 can also append time information, representing the time period corresponding to the encoded extended 3D data generation model or latent code, as metadata to the bitstream. Thus, by decoding and referencing the metadata, the decoding device can determine which time period corresponds to the decoded extended 3D data generation model or latent code, and can appropriately generate motion images of the object from any viewpoint.

[0571] Figure 59 This is a diagram illustrating the first example of the configuration of the decoding device in Embodiment 1 of Embodiment 3.

[0572] The decoding device 1515 includes a network model decoding unit 1516, a latent code decoding unit 1517, and a rendering unit 1518.

[0573] The network model decoding unit 1516 acquires the bit stream and, based on the acquired bit stream, decodes metadata such as the extended three-dimensional data generation model NNt0-2 during period t0~t2 and the extended three-dimensional data generation model NNt3-5 during period t3~t5, as well as the time information corresponding to these extended three-dimensional data generation models NNt0-2 and NNt3-5.

[0574] The latent code decoding unit 1517 decodes information related to the latent code from the input bitstream and decodes the latent codes Zt0 to Zt5 for periods t0 to t2 and t3 to t5. Furthermore, the decoding method for the latent code can be switched according to the encoding method of the encoding device 1510. For example, if the encoding device 1510 performs arithmetic encoding on the latent code, the decoding device 1515 can decode the latent code using arithmetic decoding; if the encoding device 1510 performs image encoding on the latent code, the decoding device 1515 can decode the latent code using image decoding.

[0575] Furthermore, the latent code decoding unit 1517 can also use information appended to the bitstream to switch the decoding method of the latent code. Thus, the decoding device 1515 is able to properly decode the bitstream.

[0576] Furthermore, if predictive coding is applied to the encoding of the latent code in the encoding device 1510, the decoding device 1515 can apply predictive decoding. For example, if the latent code is a multidimensional vector such as Zt = (value 0t, value 1t, ..., value 255t), the decoding device 1515 can apply predictive decoding between the vector components. More specifically, for example, if the encoding device 1510 performs arithmetic coding on the difference value obtained by subtracting value 0t from value 1t, the decoding device 1515 can perform arithmetic decoding on the difference value and decode value 1t by adding the decoded value 0t.

[0577] Similarly, when the encoding device 1510 performs arithmetic encoding on the difference value obtained by subtracting value 1t from value 2t, the decoding device 1515 can perform arithmetic decoding on the difference value and decode value 2t by adding the decoded value 1t. That is, when the encoding device 1510 encodes the value of a component A, if it calculates the prediction value P of component A using the values ​​of one or more components encoded or decoded before component A, and performs arithmetic encoding on the difference value obtained by subtracting the prediction value P from the value of component A, the decoding device 1515 can perform arithmetic decoding on the difference value, calculate the prediction value P using the same method as the encoding device 1510, and add the decoded difference value to the prediction value P to decode the value of component A. Thus, when the correlation between components in the latent code is high, the decoding device 1515 can appropriately decode the bitstream with improved coding efficiency. Alternatively, the prediction value P can be calculated based on the average, minimum, or maximum value of one or more components decoded before component A. This can improve the accuracy of the predicted value P.

[0578] The rendering unit 1518 uses metadata such as the extended 3D data generation models NNt0-2 and NNt3-5 decoded by the network model decoding unit 1516 and time information, and generates a motion image of viewpoint A at time t based on the latent code Zt for time t specified by the user or system and the viewpoint information of viewpoint A. Specifically, the rendering unit 1518 inputs the viewpoint information of viewpoint A and the latent codes Zt0, Zt1, and Zt2 corresponding to the times within the period t0-t2 into the extended 3D data generation model NNt0-2 for the period t0-t2, and generates the image IMGt0 of viewpoint A at time t0, the image IMGt1 of viewpoint A at time t1, and the image IMGt2 of viewpoint A at time t2. The rendering unit 1518 applies the image generation processing of the period t0-t2 to the extended 3D data generation model NNt3-5 for the period t3-t5 to generate the images IMGt3-IMGt5 of viewpoint A at times t3-t5. Furthermore, the rendering unit 1518 uses metadata such as images IMGt0~IMGt5 and time information to generate motion images of the object observed from viewpoint A at times t0~t5. The motion image may, for example, include images IMGt0~IMGt5 and prompt time information for calculating prompt times based on the images IMGt0~IMGt5 at times t0~t5.

[0579] Furthermore, the viewpoint information can change according to time. For example, viewpoint information of viewpoint A and latent codes Zt0~Zt2 can be input into the extended 3D data generation model NNt0-2 during time t0~t2, and viewpoint information of viewpoint B and latent codes Zt3~Zt5 can be input into the extended 3D data generation model NNt3-5 during time t3~t5. As a result, the rendering unit 1518 generates multiple images of the object observed from viewpoint A during time t0~t2, and generates multiple images of the object observed from viewpoint B during time t3~t5. That is, the rendering unit 1518 can generate motion images of the object observed, with the viewpoint switching from viewpoint A to viewpoint B at time t3.

[0580] Furthermore, the rendering unit 1518 does not necessarily need to generate moving images; it can also generate still images with specified viewpoint information at a specified time. Thus, the user can switch between generating moving images and generating still images depending on the application.

[0581] Furthermore, the rendering unit 1518 is not limited to generating moving or still images based on the extended 3D data generation model. For example, the rendering unit 1518 can also generate point group data or mesh data that the extended 3D data generation model can represent, and output the generated point group data or mesh data as dynamic point group data or dynamic mesh data. Thus, users can use dynamic 3D data of audiovisual objects such as HMDs (Head Mount Displays), and can also use the dynamic 3D data to measure the amount of motion of objects.

[0582] Figure 60 This is a second example of the configuration of the encoding device in Embodiment 1 of Embodiment 3.

[0583] The encoding device 1520 includes an extended three-dimensional data generation model acquisition unit 1521, a buffer unit 1522, a difference calculation unit 1523, a network model encoding unit 1524, and a latent code encoding unit 1525.

[0584] The extended three-dimensional data generation model acquisition unit 1521 is the same as the extended three-dimensional data generation model acquisition unit 1511 of the encoding device 1510.

[0585] The buffer unit 1522 is the same as the buffer unit 1512 of the encoding device 1510, but it differs from the buffer unit 1512 in that it inputs the extended three-dimensional data generation model stored in the memory or the like as a reference extended three-dimensional data generation model to the differential calculation unit 1523 and in that it inputs the latent code stored in the memory or the like as a reference latent code to the differential calculation unit 1523.

[0586] The difference calculation unit 1523 calculates difference information, which represents the difference between the extended 3D data generation model NNt0-2 generated by the extended 3D data generation model acquisition unit 1521 for periods t0 to t2 and the extended 3D data generation model NNt3-5 for periods t3 to t5, and the extended 3D data generation model generated by the extended 3D data generation model acquisition unit 1521 before each period (hereinafter referred to as the reference extended 3D data generation model). Here, the difference information may include the difference of the weight parameters of the nodes of each network model, etc. For example, the difference calculation unit 1523 obtains the extended 3D data generation model NNt3-5 for periods t3 to t5 from the extended 3D data generation model acquisition unit 1521, and obtains the extended 3D data generation model NNt0-2 for periods t0 to t2 from the buffer unit 1522 as the reference extended 3D data generation model.

[0587] Alternatively, the difference calculation unit 1523 can use the extended 3D data generation model NNt3-5 and the extended 3D data generation model NNt0-2, for example, to calculate the difference (change) between the weight parameters of the nodes in the network model of the extended 3D data generation model NNt3-5 and the weight parameters of the nodes in the network model of the extended 3D data generation model NNt0-2, and input the difference information representing the difference to the network model encoding unit 1524. Thus, the difference information is encoded by the network model encoding unit 1524. That is, the encoding device 1520 can also reduce the amount of data by predictive encoding, which encodes the difference between the predicted value and the information related to the network model in the extended 3D data generation model NNt3-5 based on the extended 3D data generation model NNt0-2. Through such predictive encoding, for example, in cases where the changes in the extended 3D data generation model over time are small, such as when the object hardly moves, the value of the encoded difference becomes smaller, thus improving encoding efficiency. For example, the encoding device 1520 can also be set to RNNt0-2 = 0 and RNNt3-5 = NNt0-2, and the extended three-dimensional data generation model of the previous time period can be used as a reference extended three-dimensional data generation model to reduce the number of bits through predictive coding.

[0588] In addition, the differential calculation unit 1523 calculates the differential information (hereinafter referred to as differential latent code) between the latent codes Zt0~Zt5 generated during periods t0~t2 and t3-5 generated by the extended three-dimensional data generation model acquisition unit 1521 and the latent codes generated before each period (referred to as reference latent codes).

[0589] Here, the differential latent code is, for example, the vector difference value (change) Zt1-Zt0 of the latent code calculated using the latent code Zt1 at time t1 output from the extended 3D data generation model acquisition unit 1521 and the latent code Zt0 at time t0 output from the buffer unit 1512 as a reference latent code. The differential calculation unit 1523 can output this differential latent code to the latent code encoding unit 1525. Thus, the differential latent code is encoded by the latent code encoding unit 1525. In this way, the encoding device 1520 can also predict the latent code Ztn at time n based on the latent code Ztm at time m and calculate the predicted value, thereby reducing the amount of data by encoding the latent code Ztn at time n, the predicted value, and the differential latent code (differential vector) that is their difference. Through such predictive encoding, for example, in cases where the change of the latent code between time intervals is small, such as when the object hardly moves, the value of the encoded difference becomes smaller, thus improving the encoding efficiency.

[0590] Furthermore, the encoding device 1520 in the first example performs predictive encoding on the potential code Ztn at time n based on the potential code Ztm at time m, but it is not necessarily limited to this. For example, the encoding device 1520 can select a reference potential code for prediction from one or more reference potential codes stored in the buffer 1522, and perform predictive encoding using the selected reference potential code. In this case, in order to pass the selected reference potential code to the decoding device, the encoding device 1520 can append information representing the selected reference potential code (reference potential code information) to the bitstream. Thus, the encoding device 1520 can select the optimal reference potential code from the viewpoint of encoding efficiency, thereby improving encoding efficiency. Furthermore, by decoding the reference potential code, the decoding device can appropriately decode the bitstream with improved encoding efficiency.

[0591] Furthermore, when the encoding device 1520 performs predictive coding with reference to two or more potential codes stored in the buffer 1522, it can also append information representing the two or more reference potential codes to the bitstream. Thus, the encoding device 1520 can use two or more reference potential codes to improve the coding efficiency of predictive coding. Moreover, the decoding device can appropriately decode the bitstream with improved coding efficiency.

[0592] Furthermore, when no reference latent code is stored in the buffer 1522, for example, when encoding the initial latent code (initial frame) in data order, the encoding device 1520 can encode the latent code to be processed without calculating the difference from the predicted value (hereinafter referred to as intra-frame prediction), or it can encode by calculating the difference from the predicted value set to 0. Additionally, when a certain period tm-n is set as a random access point, the encoding device 1520 can encode the latent code corresponding to the period tm-n through intra-frame prediction, or it can encode by calculating the difference from the predicted value set to 0. Therefore, the decoding device can start decoding the latent code from the initial latent code (initial frame) or the random access point in data order, improving functionality during playback.

[0593] Furthermore, a set of multiple latent codes (multiple frames) can be defined (hereinafter referred to as GOF (Group of Frame)). The first frame of the GOF can also be encoded through intra-frame prediction. Thus, the decoding device can randomly access the first frame of the GOF. In addition, by decoding the first frame of the GOF, functionality such as fast-forward playback can be improved.

[0594] Furthermore, the encoding device 1520 can also append permission information indicating whether inter-GOF prediction referencing is permitted to the bitstream. For example, if the bitstream contains permission information indicating that inter-GOF prediction referencing is prohibited, the decoding device can determine that multiple GOFs can be decoded in parallel. Additionally, for example, by permitting inter-GOF prediction referencing, encoding efficiency can be improved.

[0595] The network model encoding unit 1524 is the same as the network model encoding unit 1513 of the encoding device 1510, but it differs in that it encodes the differential information d0-2 and d3-5 of the extended three-dimensional data generation models NNt0-2 and NNt3-5 input from the differential calculation unit 1523 and outputs a bit stream.

[0596] Furthermore, the latent code encoding unit 1525 has the same function as the latent code encoding unit 1514, but it differs in that it encodes the differential latent code input from the differential calculation unit 1523 and outputs a bit stream.

[0597] Furthermore, the encoding apparatus 1520 separately describes a difference calculation unit 1523, a network model encoding unit 1524, and a latent code encoding unit 1525, but it is not necessarily limited to these. For example, the difference calculation unit 1523 may be included within the network model encoding unit 1524 or the latent code encoding unit 1525. That is, the network model encoding unit 1524 or the latent code encoding unit 1525 may also perform the processing of the difference calculation unit 1523.

[0598] Furthermore, the difference calculation unit 1523 calculates the difference values ​​of the extended three-dimensional data generation model and the difference values ​​of the latent code, but is not limited to this. For example, a network model difference calculation unit that calculates the difference values ​​of the network model and a latent code difference calculation unit that calculates the difference values ​​of the latent code can also be provided in the encoding device. As a result, the processing speed can be increased by processing the difference calculations in parallel.

[0599] Alternatively, the encoding device 1520 may append prediction coding information, indicating whether the latent code was encoded using intra-frame prediction or using a reference latent code for prediction coding (hereinafter referred to as inter-frame prediction), to the bitstream. Thus, by decoding the prediction coding information, the decoding device can appropriately determine whether intra-frame prediction or inter-frame prediction should be used to decode the latent code.

[0600] Figure 61 This is a second example of the configuration of the decoding device in Embodiment 1 of Embodiment 3.

[0601] The decoding device 1530 includes a network model decoding unit 1531, a latent code decoding unit 1532, an addition unit 1533, a buffer unit 1534, and a rendering unit 1535.

[0602] The network model decoding unit 1531 acquires the bit stream, and based on the acquired bit stream, decodes the information related to the network model, and decodes the metadata such as the difference information d0-2, d3-5 and time information of the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5 during the period t0~t2.

[0603] The latent code decoding unit 1532 decodes information related to the latent code from the input bitstream, and decodes the differential latent codes Ztd0 to Ztd5 corresponding to the periods t0 to t2 and t3 to t5. Furthermore, the decoding method of the differential latent code can be switched according to the encoding method of the encoding device 1520. For example, if the encoding device 1520 performs arithmetic encoding on the differential latent code, the latent code decoding unit 1532 decodes the differential latent code using arithmetic decoding; if the encoding device 1520 performs image encoding on the differential latent code, the latent code decoding unit 1532 decodes the differential latent code using image decoding. Additionally, the latent code decoding unit 1532 can also use information appended to the bitstream to switch the decoding method of the differential latent code. Therefore, the bitstream can be decoded appropriately.

[0604] The addition unit 1533 adds the difference information d0-2, d3-5 of the extended three-dimensional data generation models NNt0-2 and NNt3-5 corresponding to periods t0~t2 and t3~t5, decoded by the network model decoding unit 1531, and the reference extended three-dimensional data generation models RNNt0-2 and RNNt3-5 obtained from the buffer unit 1534 to the corresponding periods, and calculates the extended three-dimensional data generation models NNt0-2 and NNt3-5. In this way, the decoding device 1530 can also be set to RNNt0-2 = 0 and RNNt3-5 = NNt0-2, and use the extended three-dimensional data generation model of the previous time period as the reference extended three-dimensional data generation model for prediction decoding.

[0605] Furthermore, the addition unit 1533 adds the differential latent codes Ztd0~Ztd5 corresponding to times t0~t5, decoded by the latent code decoding unit 1532, and the reference latent codes RZt0~RZt5 input from the buffer unit 1534 at each corresponding time to calculate the latent codes Zt0~Zt5. Thus, by performing the addition in parallel, the processing speed can be increased.

[0606] Furthermore, the decoding apparatus 1530 in the second example separately describes an addition unit 1533, a network model decoding unit 1531, and a latent code decoding unit 1532, but it is not necessarily limited to this. For example, the addition unit 1533 may be included within the network model decoding unit 1531 or the latent code decoding unit 1532. That is, the network model decoding unit 1531 may perform the processing of the addition unit 1533, or the latent code decoding unit 1532 may perform the processing of the addition unit 1533.

[0607] Furthermore, if the buffer 1534 does not store the reference latent code, for example, when decoding the initial latent code (initial frame) in data order, the decoding device 1530 may perform decoding without adding the differential information to the latent code by the addition unit 1533 and without prediction (hereinafter referred to as intra-frame prediction), or it may add the prediction value set to 0 to the differential information for decoding. Additionally, if the decoding device 1530 sets a certain period tm-n as a random access point, it may decode the latent code corresponding to the period tm-n through intra-frame prediction, or it may add the prediction value set to 0 to the differential information for decoding. Moreover, if the bitstream contains prediction encoding information indicating that the latent code to be decoded has been encoded through intra-frame prediction, the latent code may be decoded through intra-frame prediction, or it may add the prediction value set to 0 to the differential information for decoding. Therefore, the decoding device 1530 can start decoding the latent code from the latent code (starting frame) that is the beginning of the data sequence, the random access point, or the latent code that has been encoded by intra-frame prediction, thereby improving the functionality during playback.

[0608] Furthermore, in the second example, the decoding device 1530 performs predictive decoding on the latent code corresponding to time n based on the latent code at time m, but it is not necessarily limited to this. For example, the decoding device 1530 may select a reference latent code for prediction from one or more latent codes stored in the buffer 1534, and perform predictive decoding using the selected latent code. In this case, the decoding device 1530 can decode the information representing the selected latent code (reference latent code) from the bitstream. Thus, the decoding device 1530 decodes the reference latent code from the bitstream generated by the encoding device 1460, which selects the reference latent code that is optimal from the viewpoint of coding efficiency, thereby enabling appropriate decoding of the bitstream with improved coding efficiency.

[0609] Furthermore, when performing predictive decoding with reference to two or more latent codes stored in the buffer 1534, the decoding device 1530 can also decode information representing two or more reference latent codes from the bitstream. Thus, the decoding device 1530 can appropriately decode a bitstream that improves the coding efficiency of predictive coding using two or more reference latent codes.

[0610] The rendering unit 1535 is the same as the rendering unit 1518 of the decoding device 1515. The rendering unit 1535 does not necessarily need to generate moving images; it can also generate still images with specified viewpoint information at a specified time.

[0611] [Modification of Example 1] Furthermore, when the encoding device transmits the bitstream of the encoded extended 3D data generation model or the latent code using the network, it can also transmit the bitstream of the extended 3D data generation model and the bitstream of the latent code separately. For example, if the device at the destination generates a rendered image at times t0 to t9, the encoding device can first transmit the extended 3D data generation model NNt0-9 and the latent code Zt0 associated with time t0 to ensure that the rendered image is generated in time t0. Then, the encoding device can sequentially transmit the latent codes Zt1 to Zt9 to ensure that the rendered image is generated in time t1 to t9 at the receiving device (the device at the destination, such as the decoding device). This reduces the load on the network's transmission capacity, allowing the receiving device to generate the rendered image before the desired time. Alternatively, the encoding device can transmit the extended 3D data generation model and the latent code associated with each random access point, and transmit the latent code outside of random access points. This reduces the overall encoding load while appropriately inserting random access points.

[0612] Alternatively, the information processing device can learn the extended 3D data generation model and latent code by setting a shorter sampling interval (time interval), while the encoding device sends a bitstream containing latent codes with a sampling interval longer than the learning time. For example, during learning, the sampling interval of the latent code can be set to 0.1 seconds, and the sampling interval of the latent codes in the encoded bitstream can be set to 1 second. In this case, the receiving device can use the decoded latent codes for each 1 second to calculate the latent codes for each 0.1 second using interpolation or filtering. This allows for the generation of a high-precision extended 3D data generation model and latent code through learning, while suppressing the amount of encoding in the generated bitstream.

[0613] Furthermore, the information processing device can also learn by assigning one or more common latent codes to two or more extended 3D data generation models. For example, the information processing device can simultaneously learn using a common latent code shared with the extended 3D data generation models of both space A and space B, generating extended 3D data generation models for space A, space B, and a latent code. In this case, regarding the receiving device, as an example of the rendering operation, for instance, if rendered images A and B of space A and space B are simultaneously generated from viewpoint V at time t, respectively, it is considered to obtain image A based on the extended 3D data generation model of space A, viewpoint V information, and the latent code related to time t, and to obtain image B based on the extended 3D data generation model of space B, viewpoint V information, and the latent code related to time t. In this way, by assigning one or more common latent codes to two or more extended 3D data generation models, it is possible to generate rendered images of multiple spaces while suppressing the amount of data.

[0614] Furthermore, the extended 3D data generation model is not limited to information related to the shape in 3D space. The information processing device can also learn model parameters to generate multiple attribute information such as color or reflectivity from a single model. This reduces the amount of data in the generated extended 3D data generation model. Additionally, the extended 3D data generation model can maintain a different 3D data generation model for each attribute. Specifically, model parameters can be learned and maintained separately for each attribute. This allows for obtaining an extended 3D data generation model that best suits the attribute information.

[0615] Furthermore, the metadata sent to the receiving side may include information such as which viewpoint information among more than one viewpoint information is more likely to improve the quality of the rendered image. Thus, the receiving device can generate high-quality motion pictures by selecting the viewpoint information with high quality for the rendered image at each moment. Additionally, the metadata may also include information such as which latent code among more than one latent code is more likely to improve the quality of the rendered image. Thus, the receiving device can generate high-quality motion pictures by selecting the moment associated with the latent code that has high quality for the rendered image. Furthermore, the receiving device can also generate high-quality motion pictures by using latent codes encoded and transmitted as bitstreams to generate the rendered image. Moreover, information indicating which attributes such as RGB, reflectivity, and transparency the extended 3D data generation model can output can be added to the metadata. Thus, the receiving device can switch the extended 3D data generation model used according to the application.

[0616] [Replenish] A latent code is, for example, a code assigned to an image (or object or scene) at a given moment, representing the features of that image. Latent codes can represent various characteristics such as appearance, shape, and lighting. As mentioned above, a latent code can be a value with n dimensions (n ​​is an integer greater than 2), represented by a multidimensional vector. Multidimensional vectors include low-dimensional vectors (two to three dimensions). Latent codes are a concept in machine learning or deep learning, used to provide information for models to learn features or patterns from input data to achieve a compact representation. Latent codes are used to compress and represent changes in the motion and appearance of a scene. Instead of moments, by using automatically learned latent information as a code, it is possible to record geometric or textural motion with richer representation. Using latent codes allows for smooth representation of changes over time, contributing to the generation of visual effects. Furthermore, latent codes can also be set as one-dimensional values ​​representing moments.

[0617] Time intervals t0, t1, t2, t3, t4, and t5 are consecutive moments arranged at equal intervals. For example, if the interval between each moment is set to 1 second, and t0 is set to 0 seconds, then t1 is 1 second later, t2 is 2 seconds later, and so on. This is just one example; the interval between each moment can be any value. The interval between each moment can be specified by an integer or a decimal. In other words, time interval tn can also be a decimal value.

[0618] The encoding or decoding apparatus in one embodiment of this application illustrates examples of intra-frame encoding or intra-frame decoding, and inter-frame encoding or inter-frame decoding, respectively. However, the encoding or decoding apparatus can also be configured to perform any one of intra-frame encoding or intra-frame decoding, and inter-frame encoding or inter-frame decoding, and switch between them. This switching can be applied to the encoding or decoding of extended 3D data generation models, and can also be applied to the encoding or decoding of latent codes. Thus, combinations that can further reduce the amount of coding are possible.

[0619] The encoding device in this embodiment can be a device comprising a circuit and a memory connected to the circuit, wherein the circuit uses the memory to (1) encode three-dimensional data generation model information capable of generating images corresponding to times within a first range from any viewpoint into a bitstream, and (2) encode a latent code into the bitstream. Thus, by performing arithmetic encoding on the latent code, the data size can be reduced. Furthermore, the decoding device receiving the bitstream can generate a motion image based on the three-dimensional data generation model.

[0620] The decoding device in this embodiment can be a device comprising a circuit and a memory connected to the circuit. The circuit uses the memory to obtain (1) three-dimensional data generation model information capable of generating an image corresponding to a time within a first range from a bit stream and (2) a latent code. Using the three-dimensional data generation model and the latent code, the image within the first range is generated. Thus, a motion image can be generated based on the three-dimensional data generation model.

[0621] The three-dimensional data generation model can be (1) the first network corresponding to the time in the first range, or (2) the difference information between the first network and the second network corresponding to the time in the second range before the first range.

[0622] The extended 3D data generation model can be generated by inputting first viewpoint information, the latent code at the first moment, and the image of the first viewpoint at the first moment into the neural network.

[0623] Alternatively, the extended 3D data generation model may include a first model and a second model, wherein the first model (NNt0-2) is capable of generating images corresponding to times within the range from time t0 to time t2, and the second model (NNt3-5) is capable of generating images corresponding to times within the range from time t3 to time t5.

[0624] [Example 2] A 3D model generated using basic NeRF can also consist of multiple networks. Furthermore, the networks referred to here are learned models obtained through learning using neural networks. Multiple networks can, for example, include networks learned using sparse sampling points and networks learned using dense sampling points. Thus, multiple networks are networks with different numbers of input sampling points or different sampling point densities. Sampling points can, for example, be 3D points representing 3D locations.

[0625] Alternatively, multiple networks may include, for example, a network for outputting geometric information such as the object's density, probability of existence, and geometric coordinates, and a network for outputting information (attribute information) associated with geometry, such as color information, reflectivity, normal vector, color coordinates, timestamp, and object ID, based on the geometric information. Multiple networks may include two or more networks. Multiple networks may also be three or more networks with different sampling points, or have two or more networks for outputting geometric information or two or more networks for outputting attribute information.

[0626] Multiple networks can be encoded using multiple network coding units. Multiple network coding units can encode multiple networks using existing network coding methods, such as NNC (Neural Network Coding) in the MPEG standard.

[0627] The information processing device can also apply the encoding or decoding methods described in this embodiment to an extended 3D data generation model. This extended 3D data generation model learns by adding latent codes to a 3D data generation model consisting of two or more networks, thereby generating still images observed from any viewpoint at any time, and generating moving images by arranging them in chronological order. This reduces the amount of encoding required for the extended 3D data generation model consisting of two or more networks and the latent codes.

[0628] Next, taking NeRF, one of the methods for 3D modeling, as an example, we will use... Figure 62 This paper describes a method for generating an extended 3D data generation model and a latent code from multiple 2D images, and encoding the generated extended 3D data generation model (network model) and the latent code. Furthermore, the method described here is an example and is not limited to the NeRF method described here; it can also be applied to other NeRF methods or 3D modeling methods.

[0629] Figure 62 This is a block diagram illustrating an example of the configuration of an encoding apparatus for encoding multiple networks in Embodiment 2 of Implementation 3.

[0630] The encoding device 1540 includes an extended 3D data generation model learning unit 1541, a 3D data generation model encoding unit 1545, and a latent code encoding unit 1549. The encoding device 1540 may also include a bitstream data constructing unit 1552.

[0631] First, the specific configuration of the extended 3D data generation model learning unit 1541 will be explained. Specifically, the extended 3D data generation model learning unit 1541 includes a first network learning unit 1542, a sampling point determination unit 1543, and a second network learning unit 1544.

[0632] The first network learning unit 1542 uses multiple input two-dimensional images, viewpoint information (camera pose) of each input two-dimensional image, and input latent code to learn a three-dimensional data generation model for each viewpoint and each time moment. That is, the first network learning unit 1542 learns two-dimensional images corresponding to each viewpoint information based on multiple two-dimensional images and the viewpoint information of each two-dimensional image, thereby generating a three-dimensional data generation model (first network) and a learned latent code (first latent code). Furthermore, the viewpoint information includes the viewpoint at which the two-dimensional image was captured and the line-of-sight vector (line-of-sight direction) from that viewpoint. The first network learning unit 1542 can also be input with sampling points (first sampling points). The first sampling points can be, for example, a set of points with large (coarse) intervals between them. The coordinates of each point included in the first sampling points can be predetermined coordinates or coordinates calculated using a prescribed method. The first network learning unit 1542 outputs the learned three-dimensional data generation model (first network) and latent code (first latent code). The three-dimensional data generation model (first network) generated by the first network learning unit 1542 is a network that outputs density information for sampling points at a time (or time period) represented by a latent code. Alternatively, the first network learning unit 1542 can also output density information for sampling points obtained during learning at a specific time (or time period).

[0633] Here, density information represents the density of objects at a sampling point at a given moment (or time period). For example, density information is set to a higher (i.e., above a predetermined value) value when the object is a person or a table, a lower (i.e., below a predetermined value) value when the object is a translucent object like glass, and a value close to 0 when there are no objects. Therefore, density information can also be called information indicating the presence or probability of an object's existence. Additionally, density information can also be called geometric information.

[0634] The sampling point determination unit 1543 determines the density of an object at a certain moment (or time period) in the coordinates represented by the coarsely sampled first sampling point based on the density information for a certain moment (or time period) output from the first network learning unit 1542 for the first sampling point, and determines a second sampling point for use in learning in the second network learning unit 1544. In determining the second sampling point, the sampling point determination unit 1543 may, for example, determine that an object exists if the density of the sampling point is greater than a predetermined density, and decide to sample more finely the surrounding area of ​​the determined object (the space where the object is determined to exist and the space around it). Furthermore, in determining the second sampling point, the sampling point determination unit 1543 may, for example, determine that there is no object in the space where the sampling point is located if the density of the sampling point is less than a predetermined density, and decide to sample more coarsely in that space, or decide not to sample. For example, a PDF sampler can be used for the sampling point determination unit 1543.

[0635] The meaning of the sampling points output by the sampling point determination unit 1543 varies depending on the density determination method. For example, sampling points output when an object is determined to exist can also be referred to as geometric information representing the coordinates of the object. Furthermore, based on the density of the sampling points, it is possible to distinguish between objects with high transmittance, such as glass, and hard materials, and to perform processing such as removing sampling points from objects (space) that are determined to be objects with high transmittance or hard materials from the sampling points of the extracted object. In this way, sampling points that become the objects to be extracted can be determined based on the density of the sampling points. Therefore, by extracting sampling points of objects or materials with specific densities that meet specific conditions, the extracted sampling points can be determined as geometric information.

[0636] The sampling point determination unit 1543 can use a predetermined method or parameters, or a method or parameters selected from multiple methods or parameters, in determining the sampling points. In this case, information representing the predetermined method or parameters can be encoded as learning metadata and stored in the bitstream. Thus, information representing the predetermined method or parameters can be notified to the decoding device as learning metadata contained in the bitstream.

[0637] The second network learning unit 1544 has the same configuration as the first network learning unit 1542. The second network learning unit 1544 uses multiple input two-dimensional images and their respective viewpoint information (camera pose), second sampling points (detailed sampling points) output from the sampling point determination unit 1543, and an initial latent code to learn a 3D model for each viewpoint at a given time (or time period). That is, based on multiple two-dimensional images and the viewpoint information of each two-dimensional image, the second sampling points, and the initial latent code, the second network learning unit 1544 learns two-dimensional images corresponding to each viewpoint information, thereby generating a 3D model (the second network). The second network learning unit 1544 outputs the 3D data generated through learning (the second network) and a latent code (the second latent code).

[0638] Here, the second network generated by the second network learning unit 1544 is a network capable of outputting color information and density information. Alternatively, the second network learning unit 1544 may output the density information and color information obtained during learning for the sampling points, and use this density information and color information for other processing.

[0639] Next, the specific structure of the three-dimensional data generation model encoding unit 1545 will be explained.

[0640] The 3D data generation model encoding unit 1545 includes a first network encoding unit 1546, a second network encoding unit 1547, and a metadata encoding unit 1548.

[0641] The first network encoding unit 1546 encodes the learned first network generated by the first network learning unit 1542. The first network encoding unit 1546 outputs the encoded data obtained by encoding the first network.

[0642] The second network encoding unit 1547 encodes the learned second network generated by the second network learning unit 1544. The second network encoding unit 1547 outputs the encoded data obtained by encoding the second network.

[0643] The metadata encoding unit 1548 encodes the metadata generated by the sampling point determination unit 1543. The metadata encoding unit 1548 outputs the encoded data obtained by encoding the metadata.

[0644] Thus, in the three-dimensional data generation model encoding unit 1545, encoded data obtained by encoding the first network, the second network, and the metadata is generated, and the generated encoded data is output.

[0645] Furthermore, the 3D data generation model encoding unit 1545 can use existing network encoding methods such as NNC (Neural Network Coding) in the MPEG standard for encoding. The learned first and second networks include: multiple layers including an input layer, intermediate layers, and an output layer, nodes in each layer, weight coefficients for each node, and transformation functions for each node. The learned first and second networks can each have: a density network for outputting density information, a color network for outputting color information, a reflectance network for outputting reflectance information, etc. Alternatively, the learned first and second networks can each have attribute networks for outputting attribute information such as color information or reflectance information.

[0646] Figure 63 This is a diagram showing an example of the encoded data of the learned first network in Implementation 3.

[0647] If the first network that has been learned is a network used to generate sampling points, it may include at least a density network for outputting density information. The first network that has been learned may also include a color network for outputting color information, or an attribute network (reflectance network) for outputting other attribute information (such as reflectance information).

[0648] Figure 64 This is a diagram representing an example of the encoded data of the learned second network in Implementation 3.

[0649] The learned second network can be a network used to output color information or other attribute information for the sampled points. The learned second network can include a density network for outputting density information and a color network for outputting color information. Additionally, the learned second network can also include an attribute network (reflectance network) for outputting other attribute information (such as reflectance information).

[0650] Furthermore, if no attribute information needs to be output, the learned first or second network may not include a color network for outputting color information, or an attribute network (reflectance network) for outputting other attribute information (such as reflectance information).

[0651] Furthermore, the 3D data generation model encoding unit 1545 can also generate the predicted value of the second network from the first network and encode the value (prediction residual) obtained by subtracting the predicted value from the second network. This reduces the encoding amount of the second network. For example, the first network can be used as the predicted value. In this case, the value obtained by subtracting the value of the first network from the second network is encoded as the prediction residual of the second network. Therefore, when the correlation between the first and second networks is high, the encoding amount can be reduced.

[0652] Furthermore, the 3D data generation model encoding unit 1545 can also generate the predicted value of the first network from the second network, and encode the value (prediction residual) obtained by subtracting the predicted value from the first network. This reduces the encoding amount of the first network. For example, the second network can be used as the predicted value. In this case, the value obtained by subtracting the value of the second network from the first network is encoded as the prediction residual of the first network. Therefore, when the correlation between the first and second networks is high, the encoding amount can be reduced.

[0653] The latent code encoding unit 1549 includes a first latent code encoding unit 1550 and a second latent code encoding unit 1551. The first latent code encoding unit 1550 encodes the first latent code input from the extended 3D data generation model learning unit 1541 and outputs the encoded first latent code to the bitstream data construction unit 1552. Similarly, the second latent code encoding unit 1551 encodes the second latent code input from the extended 3D data generation model learning unit 1541 and outputs the encoded second latent code to the bitstream data construction unit 1552.

[0654] In addition, as an encoding method for latent codes, it can be applied to... Figure 58 and Figure 60 The encoding method described herein can reduce the amount of latent code required.

[0655] Furthermore, the encoding results of latent codes can be appended to the encoding data of a learned network. For example, the encoding result of the first latent code can be appended to the encoding data of a learned first network, and the encoding result of the second latent code can be appended to the encoding data of a learned second network. This simplifies the association between learned networks and their corresponding latent codes, making management easier.

[0656] Furthermore, the first latent code and the second latent code can also be encoded sequentially by a common latent code encoder. Therefore, it is unnecessary to set up multiple latent code encoders, which reduces storage space and circuit size.

[0657] Furthermore, the learned first latent code can be set as the initial latent code for the second latent code. This shortens the learning time for the second latent code.

[0658] Furthermore, the encoding device 1540 can also learn and generate a common latent code by simultaneously learning the first network, the second network, and a latent code. This eliminates the need to encode multiple latent codes, thus reducing the amount of encoding required.

[0659] Furthermore, the initial latent codes for the first and second latent codes can each be set with different initial latent codes. Therefore, by setting appropriate initial latent codes, each latent code can be learned efficiently.

[0660] Furthermore, the latent code encoding unit 1549 can also generate a predicted value for the second latent code based on the first latent code, and encode the value obtained by subtracting the predicted value from the second latent code (prediction residual). This reduces the coding complexity of the second latent code. For example, the first latent code can be used as the predicted value. In this case, the value obtained by subtracting the value of the first latent code from the second latent code is encoded as the prediction residual of the second latent code. Therefore, when the correlation between the first and second latent codes is high, the coding complexity can be reduced.

[0661] Furthermore, the latent code encoding unit 1549 can also generate a predicted value for the first latent code based on the second latent code, and encode the value obtained by subtracting the predicted value from the first latent code (prediction residual). This reduces the coding complexity of the first latent code. For example, the second latent code can be used as the predicted value. In this case, the value obtained by subtracting the second latent code from the first latent code is encoded as the prediction residual of the first latent code. Therefore, when the correlation between the first and second latent codes is high, the coding complexity can be reduced.

[0662] Next, the decoding device 1560 for decoding multiple networks will be described. Figure 65 This is a block diagram illustrating an example of the configuration of a decoding apparatus for decoding multiple networks in Embodiment 3.

[0663] The decoding device 1560 includes a bitstream data segmentation unit 1561, a three-dimensional data generation model decoding unit 1562, a reconstruction unit 1566, and a latent code decoding unit 1571.

[0664] The bitstream data segmentation unit 1561 segments the input bitstream into encoded data for the first network, the second network, and metadata.

[0665] Next, the specific configuration of the 3D data generation model decoding unit 1562 will be described. The 3D data generation model decoding unit 1562 includes a first network decoding unit 1563, a second network decoding unit 1564, and a metadata decoding unit 1565.

[0666] The first network decoding unit 1563 decodes the learned first network based on the encoded data of the first network. The first network decoding unit 1563 outputs the decoded learned first network.

[0667] The second network decoding unit 1564 decodes the learned second network based on the encoded data of the second network. The second network decoding unit 1564 outputs the decoded learned second network.

[0668] The metadata decoding unit 1565 decodes the metadata based on the encoded metadata. The metadata decoding unit 1565 outputs the decoded metadata.

[0669] Next, the specific configuration of the latent code decoding unit 1571 will be described. The latent code decoding unit 1571 has a first latent code decoding unit 1572 and a second latent code decoding unit 1573.

[0670] The first latent code decoding unit 1572 decodes and outputs the first latent code.

[0671] The second latent code decoding unit 1573 decodes and outputs the second latent code.

[0672] In addition, as a decoding method for latent codes, it can be applied to use... Figure 59 and Figure 61 The decoding method described herein enables the appropriate decoding of bitstreams with reduced latent code size.

[0673] Next, the specific structure of the reconstruction unit 1566 will be explained. The reconstruction unit 1566 includes a density estimation unit 1567, a sampling point determination unit 1568, an attribute information estimation unit 1569, and a rendering unit 1570.

[0674] The density estimation unit 1567 uses the learned first network, the first sampling point, and the first latent code associated with time t to estimate the density information for the first sampling point at time t, and outputs the estimated density information.

[0675] The sampling point determination unit 1568 determines the second sampling point based on the density information. The sampling point determination unit 1568 determines the second sampling point using parameters contained in the metadata, in the same method as the encoding device 1520. The sampling point determination unit 1568 outputs the determined second sampling point.

[0676] The attribute information estimation unit 1569 uses the learned second network, the second sampling point, and the second latent code related to time t to estimate the density information and color information corresponding to the second sampling point at time t. The attribute information estimation unit 1569 outputs the estimated density information and color information. The attribute information estimation unit 1569 can also estimate the density information and color information corresponding to the sampling point of any viewpoint at the input time t, given that the sampling point of any viewpoint and the second latent code related to time t are input, and output the estimated density information and color information.

[0677] The rendering unit 1570 performs rendering processing based on the density and color information of each of the second sampling points, generates a two-dimensional image of each viewpoint information, and outputs the generated two-dimensional image of each viewpoint information.

[0678] In addition, the reconstruction unit 1566 can also directly output the second sampling point and the attribute information (density information and color information) corresponding to the second sampling point inferred by the attribute information inference unit 1569.

[0679] As shown in this embodiment, when the 3D data generation model consists of two or more networks, the latent code can also be maintained for each network. Therefore, during the learning of each network, by using each latent code, an appropriate latent code can be generated for each network. During generation, by using this latent code, the accuracy of the output results of each network can be improved.

[0680] [Encoding device] Figure 66 This is a diagram illustrating an example of the configuration of the encoding device in Embodiment 3. Figure 67 This is a flowchart illustrating the first example of the encoding method of the encoding device in Embodiment 3.

[0681] The encoding device 1580 includes a circuit 1581 and a memory 1582 connected to the circuit 1581. The encoding device 1580 is a device that implements the encoding devices 1510, 1520, and 1540.

[0682] Circuit 1581 performs the following actions.

[0683] Circuit 1581 acquires multiple three-dimensional data generation models (S1501). Circuit 1581 encodes the multiple three-dimensional data generation models to generate a bit stream (S1502). The multiple three-dimensional data generation models include a first three-dimensional data generation model corresponding to the first time step and a second three-dimensional data generation model corresponding to the second time step.

[0684] Here, multiple 3D data generation models can output 2D images of the subject as viewed from the specified viewpoint and viewing direction at times when the latent codes are set, provided they are input with viewpoint information including viewpoint and viewing direction. The bitstream contains the latent codes. Latent codes are set for the times corresponding to each of the multiple 3D data generation models.

[0685] Therefore, by encoding the latent code, it is possible to reduce the data size of the bitstream generated by encoding multiple 3D data generation models.

[0686] For example, a latent code is a value set for a two-dimensional image at a specific moment, representing the characteristics of the two-dimensional image at that specific moment.

[0687] Latent codes are used, for example, to compress and represent changes in the motion and appearance of a scene. Instead of time intervals, by using automatically learned latent information as codes, it is possible to record the motion of geometry or texture with richer representation. By using latent codes, changes over time can be smoothly represented, which can contribute to the generation of visual effects.

[0688] For example, a latent code is a multidimensional vector with n values ​​(n is an integer greater than 2).

[0689] In this way, by representing the latent code with multidimensional vectors, the features at each time step can be represented in more detail.

[0690] For example, in obtaining multiple three-dimensional data generation models (S1501), circuit 1581 learns using latent codes set for the time corresponding to each of the multiple three-dimensional data generation models.

[0691] Therefore, since codes are used instead of time points to learn the potential information for automatic learning, it is possible to record geometric or textural motion with richer representations, for example. Furthermore, since learning can be performed in a manner similar to images that, although at different times, represent the state of the subject, it is possible to reduce the amount of data in the generated 3D data generation model.

[0692] For example, the initial value of the potential code used for learning is set to 0 or a random value. This reduces the processing required to calculate the initial value.

[0693] For example, the initial value of the latent code used for learning is set to the value of the latent code at a time different from the time corresponding to that latent code. This reduces the processing required to calculate the initial value.

[0694] For example, the initial value of the latent code used for learning is set to the value of the already learned latent code. This enables the generation of extended 3D data generation models or latent codes with high accuracy in scenarios with little motion.

[0695] For example, circuit 1581 also performs predictive coding on the latent code. The latent code contained in the bitstream is the predictively coded latent code. This improves the coding efficiency of the latent code.

[0696] For example, in predictive coding of a latent code, circuit 1581 encodes the residual between the latent code and the predicted value. The predicted-coded latent code is the encoded residual. Thus, by encoding the residual, the amount of coding required for the latent code can be reduced.

[0697] For example, in predictive coding of latent codes, circuit 1581 switches between intra-frame predictive coding and inter-frame predictive coding. This allows the coding method to be changed according to the purpose.

[0698] For example, multiple 3D data generation models are each composed of multiple networks, including a first network and a second network different from the first network. When obtaining multiple 3D data generation models (S1501), circuit 1581(i) inputs a first initial value of the latent code into the first network and outputs the first latent code, and inputs a second initial value of the latent code into the second network and outputs the second latent code. When generating a bit stream (S1502), circuit 1581(ii) generates a bit stream containing the first latent code and the second latent code.

[0699] Therefore, it is possible to output a latent code for each network, and thus it is possible to set a latent code corresponding to each network. This makes it possible to further reduce the data size of the bitstream generated by encoding multiple 3D data generation models.

[0700] For example, the first initial value and the second initial value are different values ​​from each other.

[0701] Therefore, by setting appropriate initial latent codes, each latent code can be learned efficiently.

[0702] For example, circuit 1581 also encodes the second latent code and the first residual based on the first prediction value of the first latent code. The latent code contained in the bitstream is the encoded first residual.

[0703] This allows for a reduction in the coding complexity of the second latent code. Alternatively, the first latent code can be used as the first prediction value. In this case, the value obtained by subtracting the value of the first latent code from the second latent code is encoded as the prediction residual of the second latent code. Thus, when the correlation between the first and second latent codes is high, the coding complexity can be reduced.

[0704] For example, circuit 1581 also encodes the first latent code and the second residual based on the second prediction value of the second latent code. The latent code contained in the bitstream is the encoded second residual.

[0705] This allows for a reduction in the coding complexity of the first latent code. For example, a second latent code can also be used as the second prediction value. In this case, the value obtained by subtracting the second latent code from the first latent code is encoded as the prediction residual of the first latent code. Thus, when the correlation between the first and second latent codes is high, the coding complexity can be reduced.

[0706] Figure 68 This is a flowchart illustrating a second example of the encoding method of the encoding device in Embodiment 3.

[0707] Circuit 1581 can also perform the following operations.

[0708] Circuit 1581 obtains a three-dimensional data generation model including a first network and a second network different from the first network (S1511). Circuit 1581 encodes the three-dimensional data generation model to generate a bit stream (S1512). In the encoding of the three-dimensional data generation model, circuit 1581 encodes the first predicted value based on the first network and the first residual based on the second network.

[0709] Therefore, for the second network of the 3D data generation model, the first residual obtained using the first prediction value based on the first network is encoded, thus reducing the encoding amount of the second network.

[0710] For example, circuit 1581 also encodes the second predicted value based on the second network and the second residual of the first network in the encoding of the three-dimensional data generation model.

[0711] Therefore, for the first network of the 3D data generation model, the second residual obtained by using the second prediction value based on the second network is encoded, thus reducing the encoding amount of the first network.

[0712] [Decoding device] Figure 69 This is a diagram illustrating an example of the configuration of the decoding device in Embodiment 3. Figure 70 This is a flowchart illustrating the first example of the decoding method of the decoding device in Embodiment 3.

[0713] The decoding device 1590 includes a circuit 1591 and a memory 1592 connected to the circuit 1591. The decoding device 1590 is a device that implements the decoding devices 1515, 1530, and 1560.

[0714] Circuit 1591 performs the following actions.

[0715] Circuit 1591 acquires the bit stream (S1521). Circuit 1591 decodes multiple three-dimensional data generation models and latent codes from the bit stream (S1522). The multiple three-dimensional data generation models include a first three-dimensional data generation model corresponding to the first time step and a second three-dimensional data generation model corresponding to the second time step.

[0716] Here, multiple 3D data generation models can output 2D images of the subject as viewed from the viewpoint and viewing direction at times when the latent codes are set, provided that the viewpoint and viewing direction are input as viewpoint information and latent codes. Latent codes are set for the times corresponding to each of the multiple 3D data generation models.

[0717] Therefore, it is possible to appropriately decode multiple 3D data generation models from a bitstream with reduced data size.

[0718] For example, a latent code is a value set for a two-dimensional image at a specific moment, representing the characteristics of the two-dimensional image at that specific moment.

[0719] Thus, it is possible to use latent codes to appropriately decode multiple 3D data generation models from a bitstream with reduced data size.

[0720] For example, a latent code is a multidimensional vector with n values ​​(n is an integer greater than 2).

[0721] In this way, by representing the latent code with multidimensional vectors, it is possible to appropriately decode multiple three-dimensional data generation models that more comprehensively represent the features at each time step.

[0722] For example, circuit 1591 learns by using latent codes set for the time corresponding to each of the multiple three-dimensional data generation models in the acquisition of multiple three-dimensional data generation models.

[0723] Thus, it is possible to use latent codes to appropriately decode multiple 3D data generation models from a bitstream with reduced data size.

[0724] For example, the initial value of the potential code used for learning is set to 0 or a random value. This reduces the processing required to calculate the initial value.

[0725] For example, the initial value of the latent code used for learning is set to the value of the latent code at a time different from the time corresponding to that latent code. This reduces the processing required to calculate the initial value.

[0726] For example, the initial value of the latent code used for learning is set to the value of the already learned latent code. This enables the generation of extended 3D data generation models or latent codes with high accuracy in scenarios with little motion.

[0727] For example, circuit 1591 also performs predictive decoding on the latent code. This allows for appropriate decoding of the latent code, thus improving coding efficiency.

[0728] For example, in the prediction decoding of a latent code, circuit 1591 decodes the residual between the latent code and the predicted value by adding the predicted value to the residual. This allows for the appropriate decoding of latent codes with relatively small coding amounts.

[0729] For example, in the predictive decoding of the latent code, circuit 1591 switches between intra-frame predictive decoding and inter-frame predictive decoding. This allows the decoding method to be changed depending on the purpose.

[0730] For example, multiple 3D data generation models are each composed of multiple networks, including a first network and a second network different from the first network. The bitstream contains a first latent code and a second latent code. The first latent code is the latent code output by inputting the first initial value of the latent code into the first network. The second latent code is the latent code output by inputting the second initial value of the latent code into the second network.

[0731] Therefore, it is possible to appropriately decode the latent code corresponding to the network. Consequently, it is possible to appropriately decode multiple 3D data generation models from a bitstream with further reduced data size.

[0732] For example, the first initial value and the second initial value are different from each other. Therefore, it is possible to appropriately decode each latent code after efficient learning.

[0733] For example, circuit 1591 also decodes the second latent code and the first residual based on the first prediction value of the first latent code. Circuit 1591 decodes the second latent code by adding the first prediction value to the first residual.

[0734] Therefore, it is possible to properly decode the second latent code with reduced coding size.

[0735] For example, circuit 1591 also decodes the first latent code and the second residual based on the second prediction value of the second latent code. Circuit 1591 decodes the first latent code by adding the second prediction value to the second residual.

[0736] Therefore, it is possible to properly decode the first latent code with reduced coding size.

[0737] Figure 71This is a flowchart illustrating the second example of the encoding method of the decoding device in Embodiment 3.

[0738] Circuit 1591 can also perform the following actions.

[0739] Circuit 1591 acquires the bitstream (S1531). Circuit 1591 decodes the three-dimensional data generation model from the bitstream, which includes a first network and a second network different from the first network (S1532). In the decoding of the three-dimensional data generation model, circuit 1591 decodes the first predicted value based on the first network and the first residual based on the second network.

[0740] Therefore, the second network, with its reduced coding volume, can be properly decoded.

[0741] For example, circuit 1591 also decodes the second predicted value based on the second network and the second residual of the first network in the decoding of the three-dimensional data generation model.

[0742] Therefore, it is possible to properly decode the first network with reduced coding volume.

[0743] (Implementation Method 4) Implementation method 4 describes a method for encoding and transmitting a three-dimensional model (a learning model for generating three-dimensional data).

[0744] For example, point group data, such as point clusters and meshes, contains information about 3D points or lines connecting those points, surface information, attribute information corresponding to the points, and attribute information corresponding to the surfaces. Therefore, if the resolution of the points or meshes increases, or the area of ​​the points or meshes increases, the amount of point group data increases proportionally to the increase in resolution or area.

[0745] When the area of ​​points or grids is large, even if the constituent elements of point group data such as points or grids are encoded, the amount of data after encoding will also increase due to the large amount of data in the data.

[0746] In contrast, 3D models, as learning models used to generate 3D data, show minimal increase in data volume even when the area of ​​points or grids increases. A 3D model is a network model that learns from 2D data (2D images) or 3D data (point groups or grids) using neural networks or similar methods to obtain 3D shapes and corresponding attribute information. Furthermore, since a 3D model is a network model used to generate 3D data, it can also be called a 3D generative model.

[0747] Therefore, in order to reduce the storage capacity or transmission volume of 3D data, a method for encoding and transmitting 3D models is needed.

[0748] In this embodiment, the three-dimensional model learning unit (three-dimensional model acquisition unit), three-dimensional model encoding unit (network model encoding unit), and three-dimensional model decoding unit (network model decoding unit) described in the previous embodiments will be specifically described as a method for modeling a three-dimensional model. The method for modeling using NeRF (Neural Radiance Fields), the encoding method of the three-dimensional model, the decoding method of the encoded three-dimensional model, and the method for decoding two-dimensional images and three-dimensional data from the three-dimensional model will be described.

[0749] A 3D model generated using basic NeRF can also consist of multiple networks. Furthermore, the networks referred to here are learned models obtained through learning using neural networks. These multiple networks can include, for example, networks learned using sparse sampling points and networks learned using dense sampling points. Thus, the multiple networks are networks with different numbers of input sampling points or different sampling point densities. Sampling points can, for example, be 3D points representing 3D positions.

[0750] Alternatively, multiple networks may include, for example, a network for outputting geometric information such as the object's density, probability of existence, and geometric coordinates, and a network for outputting information (attribute information) associated with geometry, such as color information, reflectivity, normal vector, color coordinates, timestamp, and object ID, based on the geometric information. Multiple networks may also include two or more networks. Multiple networks can be three or more networks with different sampling points, or they can have two or more networks for outputting geometric information, or two or more networks for outputting attribute information.

[0751] For multiple networks, multiple network coding units can be used for encoding. Multiple network coding units can also encode multiple networks using existing network coding methods such as NNC (Neural Network Coding) in the MPEG standard.

[0752] Next, the data structure for storing the encoded data will be explained.

[0753] Based on this structure, the decoding device can identify each category of data from the NeRF encoded bitstream, thus enabling data segmentation and decoding by category. Furthermore, the decoding device readily performs data-by-data processing, enabling parallel decoding, random access, partial decoding, and scalable decoding.

[0754] Furthermore, in a 3D model composed of multiple networks, by assigning the same identification ID to the data that constitute the same 3D model, the decoder can identify the networks of the same 3D model.

[0755] Figure 72 This is a diagram illustrating an example of the syntax of metadata for sequence units in implementation 4. Figure 73 This is a diagram of an example of the syntax for representing metadata of frame units.

[0756] The learned network using NeRF includes: multiple layers such as the input layer, intermediate layers, and output layer; nodes in each layer; weight coefficients for each node; transformation functions for each node; and the latent code. Parameters that are common throughout the sequence are stored in metadata shared within the sequence, such as the Sequence Parameter Set (SPS). Furthermore, the dimension of the latent code may also be the same throughout the sequence, and therefore can also be stored in shared metadata such as the SPS. For example, if the latent code is n-dimensional (n is an integer greater than 1), the value n representing the dimension of the latent code can also be stored in shared metadata such as the SPS. Thus, by setting the dimension of the latent code to a common value throughout the sequence when it is the same throughout the sequence, the amount of encoding in the header can be reduced.

[0757] In addition, parameters shared per frame, per access unit, or across multiple frames are stored in metadata shared within frames or across multiple frames. Furthermore, metadata shared across a sequence can also be set as a VPS (Video Parameter Set), specifying parameters common to the entire video that includes more than one sequence.

[0758] For example, if the network's structural information, such as information about multiple layers, is constant in the sequence, it can be stored in the SPS or in the frame unit's metadata. The network's structural information can also be stored in both the SPS and the frame unit's metadata, in which case the frame unit's metadata can be used preferentially. Furthermore, the network's structural information can be stored in either the SPS or the frame unit's metadata. In this case, a flag indicating which of the SPS and metadata the network's structural information is stored in can be stored in the higher-level SPS, and the decoding device 1530 determines which of the SPS and metadata the network's structural information is stored in based on this flag.

[0759] Furthermore, for example, when the dimension n of the latent code is constant throughout the sequence, the dimension n of the latent code can also be stored in the SPS. This reduces the amount of encoding in the header when the dimension n of the latent code is constant throughout the sequence. Additionally, when the dimension n of the latent code is changed on a frame-by-frame or multi-frame-by-multiple ... Furthermore, a flag indicating whether the dimension n is stored in the metadata of the lower-level unit or the metadata of the higher-level unit can also be stored in the higher-level metadata (SPS). The decoding device can also use this flag to determine whether the dimension n is stored in the metadata of the lower-level unit or the metadata of the higher-level unit. Thus, the dimension n of the latent code can be flexibly appended to the bitstream according to the specified units.

[0760] Thus, the bitstream contains multiple networks and multiple latent codes. Each network corresponds to multiple time points. Each network corresponds to a different time point. Multiple networks belong to a defined unit. That is, there are multiple defined units, and the networks are classified according to each defined unit. Multiple latent codes correspond to multiple time points. The dimension information representing the dimension of the latent codes is set according to each defined unit. That is, the dimension can also be set to be common to each defined unit. The dimension information represents the common dimension among the multiple latent codes corresponding to multiple networks belonging to the same defined unit.

[0761] Next, the data structure of the network's coding layer will be explained.

[0762] Figure 74 This is a diagram illustrating an example of the syntax of data units in a high-density network in Implementation 4. Figure 75 This diagram illustrates an example of the syntax of data units in a low-density network in Implementation Method 4. Furthermore, high density indicates that the interval between sampling points is greater than a predetermined value, resulting in coarse sampling. Low density indicates that the interval between sampling points is less than a predetermined value, resulting in fine sampling.

[0763] Figure 76 This is a diagram illustrating an example of the configuration of the data units of the first network in Implementation Method 4. Figure 77This diagram illustrates an example of the configuration of data units in the second network of Embodiment 4. Furthermore, the first network is a high-density network, and the second network is a lower-density network than the first network.

[0764] The parameters of the network, such as node weights and latent codes, obtained from learning, can also be stored as network data in the network's data units. Network data can also contain information about the network's structure. This structure information includes, for example, information used to determine the multiple layers containing the network (input, intermediate, output, etc.), the nodes in each layer, the weights for each node, and the transformation functions for each node.

[0765] In addition, the data related to the output of density information (density network for outputting density information), the data related to the output of color information (color network for outputting color information), the data related to other attribute information (attribute network for outputting other attribute information (e.g., reflectance information)) and the latent code in the network data can be grouped and saved.

[0766] For example, if the learned first network has a density network for outputting the density of the sampling points, the network data, the density network, and the first latent code are encoded and stored in the payload of the network's data unit.

[0767] Additionally, for example, if the learned second network has a density network for outputting the density of the sampling points and a color network for outputting color information, the network data, density network, color network, and second latent code are encoded and stored in the payload of the network's data unit.

[0768] In network encoding, existing network encoding methods such as NNC (Neural Network Coding) of the MPEG standard can also be used. In this case, NNC data units can also be used as the data units of the network. Encoded data defined by NNC can also be stored in data structures defined by NNC. Furthermore, network data units can also be divided into multiple data formats according to each component of the network.

[0769] Alternatively, the dimension of the latent code can be appended to the network data unit, setting the dimension of the latent code for each network. This allows for setting an appropriate dimension for each network. For example, setting a larger dimension of the latent code enables the generation of networks with higher time resolution, while setting a smaller dimension enables the generation of networks with smaller coding size.

[0770] Next, the data structure of the encoded data of the NeRF 3D model will be explained. Figure 78 This is a diagram illustrating an example of the syntax of the encoded data of the three-dimensional model of NeRF in Implementation 4. Figure 79 This is a diagram illustrating an example of the cell type of NeRF in Implementation 4. Figure 80A This is a diagram illustrating another example of the syntax of the encoded data of the three-dimensional model of NeRF in Implementation 4. Figure 80B This is a diagram illustrating an example of the syntax of the Latent Data Unit in Implementation 4. Figure 81 This is a diagram illustrating an example of the data structure of the encoded data of the three-dimensional model of NeRF in Implementation 4.

[0771] For example, the encoded data of the NeRF 3D model may be stored in the `model_codec_unit` within the `codec_unit()` function, which is used to synthesize and process the encoding methods, as described in the data structure of the encoded data in the above-described embodiment, and then transmitted. Alternatively, the encoded data of the NeRF 3D model may be transmitted directly as a `model_codec_unit` without being stored in the `codec_unit`. Here, the encoded data of the NeRF 3D model is referred to as a NeRF 3D model Unit.

[0772] The encoded data of a NeRF 3D model consists of a header and NeRF 3D model data, and can also store Fine NW Data Units, Coarse NW Data Units, NeRF Metadata, SPS, FPS, NPS, SEI, etc. The Nerf_unit_type in the header indicates the type of data stored in the NeRF 3D model data. Therefore, the encoding device can generate data that the decoding device 1530 can recognize as the constituent elements of the encoded data of the NeRF model.

[0773] Here, when the data is a Fine NW Data Unit (first network) or a Coarse NW Data Unit (second network), a 3D sub-model ID that identifies the 3D model can also be assigned. For example, when they are Fine NW Data Units or Coarse NW Data Units in the same 3D model, an ID representing a data unit that is in the same 3D model can also be assigned.

[0774] Additionally, the Network Data Unit of the first network may also contain the first latent code. The Network Data Unit of the second network may also contain the second latent code.

[0775] Furthermore, the encoded data of the NeRF 3D model may also include a 3D model frame ID representing the frame number of the 3D model. Additionally, data units of frames within the same time frame may be assigned the same frame ID in the encoded data of the NeRF 3D model. Furthermore, when the 3D model data is region-segmented data, it may also include a space ID indicating which region it belongs to. That is, data within the same region are assigned the same space ID. Moreover, this space ID is the same as the space ID described in the above embodiment.

[0776] Furthermore, the 3D model sub-model ID and the data unit ID described in the above embodiment can also be the same. These identifiers enable the establishment of correspondences between related data, allowing the decoding device 1530 to identify the established corresponding data.

[0777] Alternatively, a different Nerf_unit_type can be defined to store information about the latent code. For example, consider preparing a Latent Data Unit with Nerf_unit_type=6. That is, the bitstream can also contain Nerf_unit_type as type information representing the type of data, and the data in the bitstream that contains latent code metadata (data with Nerf_unit_type=6) is latent code metadata (Latent DataUnit).

[0778] Additionally, the Latent Data Unit can also contain information about the dimension of the latent code. Therefore, it is possible to appropriately switch the dimension of the latent code for each Latent Data Unit.

[0779] Furthermore, a Latent Data Unit can also contain information indicating which network the latent code is associated with. That is, the metadata can also contain network identification information indicating the network corresponding to the latent code. For example, a Latent Data Unit can also contain information for referencing a Fine NW Data Unit or a Coarse NW Data Unit. Thus, the latent code contained within a Latent Data Unit can be associated with its associated network, and appropriate information can be output using the information from both sides (i.e., the latent code and the network).

[0780] Additionally, a Latent Data Unit can also contain a latent data id (metadata identification information) used to identify the Latent Data Unit (or establish a correspondence with the latent data id), and an NW data unit (network) can also contain a latent data id of a Latent Data Unit that has data corresponding to the NW data unit. Thus, a correspondence can also be established between NW data units and Latent data units.

[0781] For example, if a first network and a first latent code corresponding to the first network exist, the first latent code can be stored in a Latent Data unit with latent data id=0 appended. The NW data unit storing the first network can also have the information ref_latent_data_id, indicating the latent data id to be referenced, appended and set to ref_latent_data_id=0, thus indicating that the first latent code is used when processing the first network. In this way, the latent code contained in the Latent Data Unit can be associated with its associated network, and appropriate information can be output using the information from both.

[0782] Thus, the bitstream contains a network and metadata, which includes a latent code set for the time corresponding to that network. The network contains metadata identification information indicating that it contains metadata with a latent code corresponding to that network.

[0783] In addition, such as Figure 80B As shown, a Latent Data Unit can also contain multiple latent data (latent codes) corresponding to multiple time points. That is, a single metadata unit can also contain multiple latent codes corresponding to multiple networks.

[0784] Next, the structural information of the three-dimensional model of NeRF (the first network or the second network) will be explained. Figure 82 This is a diagram illustrating an example of the SPS syntax of the three-dimensional model of NeRF in Implementation 4. Figure 83 This is a diagram illustrating an example of the syntax of the structural information of the three-dimensional model of NeRF in Implementation 4. Figure 84 This is a diagram representing an example of component_type in implementation method 4. Figure 85 This is a diagram illustrating an example of the component coding type in implementation method 4.

[0785] The SPS (Sequence Parameter Set) stores the structural information of the NeRF that constitutes the sequence corresponding to the SPS. Therefore, the decoding device 1530 can obtain information about the constituent elements or components of the bitstream containing the SPS, and can begin decoding based on this information.

[0786] The SPS contains a number_of_component indicating the number of components that make up the bitstream, and a component_type that serves as an identifier for each component, according to the number of components. The component_type, for example, indicates whether the encoded data is geometric or the density of the geometry. In addition, the component_type can also represent... Figure 84 The types shown.

[0787] In addition, SPS includes a component coding type indicating the encoding method used to encode the component. Encoding methods represented by the component coding type include, for example, MPEG G-PCC for point group compression, VVC for video codecs, and NNC for network compression. Furthermore, the component coding type can also represent... Figure 85 The encoding method is illustrated.

[0788] In this embodiment, in the example of a bitstream consisting of two networks, the number of components is 2, which can also be represented as follows.

[0789] component0: component_type = 1 or 5, component coding type = 4 component1: component_type = 2 or 6, component coding type = 4 In addition, the above description indicates whether it refers to the case where component_type=1 for component0 and component_type=2 for component1, or the case where component_type=5 for component0 and component_type=6 for component1.

[0790] Thus, the structural information of the three-dimensional model of NeRF is communicated to the decoding device 1530, which can then begin decoding based on the structural information of the three-dimensional model of NeRF.

[0791] in addition, Figure 84 The component_types shown in 0~6 are just one example; they may not all be shown, or only a portion may be shown.

[0792] Furthermore, not limited to the case where component coding type == NNC, when the component is composed of a network, the structural information of the three-dimensional model of NeRF can also include information such as the number of network layers, the number of nodes in each network layer, or the dimension of the latent code. Thus, even without using NNC as the codec, the transmitting device (encoding device) can appropriately transmit network-related information to the receiving device (decoding device).

[0793] Next, the reference relationships of the encoded data of the NeRF three-dimensional model will be explained. Figure 86 This is a diagram illustrating the reference relationship of the encoded data of the three-dimensional model of NeRF in Implementation 4.

[0794] For example, to indicate that they are networks corresponding to the same frame 0, the first network (CoarseNW) and the second network (FineNW) in the 3D generative model of frame 0 are assigned the same 3D model frame id. Additionally, to indicate that they are networks corresponding to the same 3D generative model, the first network (CoarseNW) and the second network (FineNW) are assigned the same model identifier 3D model sub-model id.

[0795] Therefore, the decoding device 1530 can identify the first network (CoarseNW) and the second network (FineNW) used in the generation of the 3D model. In addition, the first network (CoarseNW) and the second network (FineNW) are each assigned a ref_nerf_metadata_id. During decoding, the decoding device 1530 can perform decoding using the referenced NeRF metadata by referring to NeRFmetadata with the same nerf_metadata_id.

[0796] Furthermore, when the latent codes of the first network (Coarse NW) and the second network (Fine NW) have the same dimension, the dimension of the common latent code can also be included in the NeRF Metadata. Therefore, when the latent codes of the first network (Coarse NW) and the second network (Fine NW) have the same dimension, the dimension of the latent code does not need to be appended to the data units of the first network (Coarse NW) and the second network (Fine NW), thus reducing the encoding complexity of the data units of the first network (Coarse NW) and the second network (Fine NW).

[0797] The first network (Coarse NW) may also contain the ID of the latent code corresponding to the first network (Coarse NW) or the latent code to be referenced. The latent code ID is used to identify the latent code contained in the NeRF metadata and corresponding to the first network. Similarly, the second network (Fine NW) may also contain the ID of the latent code corresponding to the second network (Fine NW) or the latent code to be referenced. The latent code ID is used to identify the latent code contained in the NeRF metadata and corresponding to the second network.

[0798] Thus, the bitstream may contain multiple networks, such as a first network and a second network. The first network corresponds to the first latent code, and the second network corresponds to the second latent code. The metadata (NeRF metadata) contains the first latent code and the second latent code. The first network contains first identification information (id) for referencing the first latent code, and the second network contains second identification information (id) for referencing the second latent code.

[0799] The first network (Coarse NW) and the second network (Fine NW) are 3D models used to reconstruct 3D data of the same scene, therefore, as Figure 86 As shown, these correspond to the moments that are shared by both networks. Compared to the first network (CoarseNW), the second network (Fine NW) uses a higher resolution sampling point for learning.

[0800] Next, the bit stream segmentation process in the decoding device will be explained.

[0801] The bitstream input to the decoding device contains data from various NeRF 3D model Units. The decoding device first parses the header of the NeRF 3D model Unit. If the Nerf_unit_type is Fine NW Data Unit, the decoding device 1530 recognizes the subsequent data as encoded data of the second network (FineNW) and decodes the data of the second network (FineNW). Additionally, if the latent code is contained within an NW Data Unit, the decoding device can also decode that latent code.

[0802] Similarly, the decoding device uses Nerf_unit_type to determine which of the following is represented: coarse NW Data Unit, NeRFMetadata, Sequence Parameter Set, Frame Parameter Set, or Network Parameter Set, and decodes the data accordingly. Alternatively, if Nerf_unit_type indicates that LatentData Unit is included in the bitstream (e.g., Nerf_unit_type=6), the decoding device can identify that subsequent data is related to the latent code and decode the latent code data.

[0803] Next, the decoding process of the first and second networks will be explained.

[0804] The decoding device parses the 3D model frame ID, 3D model sub-model ID, 3D model space ID, and ref_nerf_metadata_id to determine which frame, model, and space the encoded data from the first or second network belongs to, and pairs encoded data with the same identifier. The decoding device can perform decoding at the point when all encoded data with the same identifier are available.

[0805] For example, when the decoding device receives a fine NW Data Unit with 3D model frame id=0 and 3D model sub-model id=1, it searches for a Coarse NW Data Unit with the same ID. When a Coarse NW Data Unit is found, it determines that it can be decoded and begins decoding.

[0806] Furthermore, if the NW data unit contains a latent code ID (ref_latent_data_id) to be referenced, the decoding device can also pair the NW data unit with the Latent Data Unit represented by ref_latent_data_id. Additionally, the decoding device can be set to be capable of decoding at the point when all Latent Data Units represented by ref_latent_data_id are available. For example, when receiving a fine NW data unit containing ref_latent_data_id=0, the decoding device can search for a Latent Data Unit with the corresponding ID, determine that it can be decoded upon finding one, and begin decoding. Thus, the decoding device can appropriately decode the latent code contained in the Latent Data Unit and its corresponding network, and output appropriate information using the information from both.

[0807] Next, the constraints on the order of data arrangement will be explained.

[0808] The encoding device can also, in cases where the second network (Fine Network) must reference the first network (Coarse Network), impose a constraint that the encoded data of the first network (Coarse Network) is transmitted before the encoded data of the second network (Fine Network). The first network (Coarse Network) is received and decoded by the decoding device before the second network (Fine Network). The decoding device receives the second network (Fine Network) after the first network (Coarse Network) and decodes it, searching for the corresponding 3D model frame ID and 3D model sub-model ID.

[0809] Furthermore, when the second latent code must reference the first latent code, a constraint can be set such that the first latent code is transmitted before the second latent code. For example, as described above, consider the case where the encoding device uses the first latent code to predictively encode the second latent code. In this case, the first latent code is received and decoded by the decoding device before the second latent code. The second latent code is received by the decoding device after the first latent code and is decoded using the decoding result of the first latent code. In this case, if the information indicating that the second latent code should reference the first latent code in the Latent Data Unit of the second latent code, such as the value of latent_data_id containing the first latent code's Latent data unit being 0, can be appended to the Latent Data Unit of the second latent code, indicating that the second latent code should reference the first latent code, with ref_latent_data_id=0. Thus, the receiving device (decoding device) can know the reference relationship between the latent codes and can appropriately decode each data.

[0810] Furthermore, if the received data is network encoded data, the decoding device parses the header of the network data unit and decodes the network using a prescribed method. For example, if it is an NNC data unit, the decoding device decodes it using a method specified by the NNC standard.

[0811] Next, an example of data segmentation will be described based on the process of segmenting three-dimensional data into more than one three-dimensional data as described in the above embodiments.

[0812] Figure...

Claims

1. An encoding device, wherein, have: Circuits; and The memory is connected to the circuit. The circuit is in operation. Obtain the network constituting the three-dimensional data generation model, and the latent code set for the time corresponding to the network. Generate a bitstream containing the network and metadata, wherein the metadata contains the latent code. The network contains metadata identification information, which represents metadata containing the latent code corresponding to the network.

2. The encoding device according to claim 1, wherein, The bitstream also includes category information, which indicates the type of data contained in the bitstream. The data contained in the bitstream that is represented by the category information as metadata containing the latent code is metadata containing the latent code.

3. The encoding device according to claim 1 or 2, wherein, The metadata also includes dimension information representing the dimension of the latent code.

4. The encoding device according to claim 3, wherein, The bitstream includes: Includes the network and multiple networks corresponding to multiple times; and Including the potential codes and multiple potential codes set for the multiple times, The dimension information represents the common dimension of the multiple potential inter-code elements.

5. The encoding device according to claim 1 or 2, wherein, The bitstream includes multiple networks, including the network in question. The plurality of networks includes a first network corresponding to the first latent code and a second network corresponding to the second latent code. The metadata includes the first latent code and the second latent code. The first network contains first identification information for referencing the first latent code. The second network contains second identification information for referencing the second potential code.

6. The encoding device according to claim 5, wherein, The first latent code and the second latent code correspond to a common moment. Compared to the first network, the second network has a higher resolution for the sampling points used for learning.

7. The encoding device according to claim 1 or 2, wherein, The metadata includes network identification information representing the network corresponding to the latent code.

8. A decoding device, wherein, have: Circuits; and The memory is connected to the circuit. The circuit is in operation. A bitstream is obtained, which contains a network constituting a 3D data generation model, and metadata containing latent codes set for times corresponding to the network. The network contains metadata identification information, which represents metadata containing the latent code corresponding to the network. The network is decoded based on the bit stream.

9. The decoding apparatus according to claim 8, wherein, The bitstream also includes category information, which indicates the type of data contained in the bitstream. The data contained in the bitstream that is represented by the category information as metadata containing the latent code is metadata containing the latent code.

10. The decoding apparatus according to claim 8 or 9, wherein, The metadata also includes dimension information representing the dimension of the latent code.

11. The decoding apparatus according to claim 10, wherein, The bitstream includes: Includes the network and multiple networks corresponding to multiple times; and Including the potential codes and multiple potential codes set for the multiple times, The dimension information represents the common dimension of the multiple potential inter-code elements.

12. The decoding apparatus according to claim 8 or 9, wherein, The bitstream includes multiple networks, including the network in question. The plurality of networks includes a first network corresponding to the first latent code and a second network corresponding to the second latent code. The metadata includes the first latent code and the second latent code. The first network contains first identification information representing the first potential code. The second network contains second identification information representing the second potential code.

13. The decoding apparatus according to claim 12, wherein, The first latent code and the second latent code correspond to a common moment. Compared to the first network, the second network has a higher resolution for the sampling points used for learning.

14. The decoding apparatus according to claim 8 or 9, wherein, The metadata includes network identification information representing the network corresponding to the latent code.

15. An encoding method, executed by an encoding device, wherein, Obtain the network constituting the three-dimensional data generation model, and the latent code set for the time corresponding to the network. Generate a bitstream containing the network and metadata, wherein the metadata contains the latent code. The network contains metadata identification information, which represents metadata containing the latent code corresponding to the network.

16. A decoding method, executed by a decoding device, wherein, A bitstream is obtained, which contains a network constituting a 3D data generation model, and metadata containing latent codes set for times corresponding to the network. The network contains metadata identification information, which represents metadata containing the latent code corresponding to the network. The network is decoded based on the bit stream.

Images