Encoding device, decoding device, encoding method, and decoding method

Abstract

In the present invention, an encoding device acquires a first three-dimensional data generation model corresponding to a first timepoint and a second three-dimensional data generation model corresponding to a second timepoint, and generates a bit stream by encoding the acquired first three-dimensional data generation model and second three-dimensional data generation model. A decoding device acquires a bit stream, and decodes, from the bit stream, a first three-dimensional data generation model corresponding to a first timepoint and a second three-dimensional data generation model corresponding to a second timepoint. When viewpoint information including a viewpoint and a line-of-sight direction is inputted, the first three-dimensional data generation model and the second three-dimensional data generation model each output a two-dimensional image of a subject as viewed from the viewpoint and the line-of-sight direction.

Description

Encoding device, decoding device, encoding method, and decoding method

[0001] The present disclosure relates to an encoding device, a decoding device, an encoding method, and a decoding method.

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a representation called a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. In a point cloud, the position and color of the points are stored. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just as with two-dimensional moving images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] Furthermore, compression of point clouds is partially supported by a public library (Point Cloud Library) that performs processing related to point clouds.

[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1).

[0006] International Publication No. 2014 / 020663

[0007] ISO / IEC 15938-17:2022 (Information technology - Multimedia content description interface - Part 17: Compression of neural networks for multimedia content description and analysis (https / / www.iso.org / standard / 78480.html))

[0008] An object of the present disclosure is to provide an encoding device and the like that can reduce the amount of data from which moving images from any viewpoint can be obtained.

[0009] An encoding device according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time, and generates a bitstream by encoding the acquired first three-dimensional data generation model and second three-dimensional data generation model, and when viewpoint information including a viewpoint and a line of sight is input, each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as seen from the viewpoint and the line of sight.

[0010] A decoding device according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a bitstream and decodes from the bitstream a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time, and when viewpoint information including a viewpoint and a line of sight is input, each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as viewed from the viewpoint and the line of sight.

[0011] These comprehensive or specific aspects may be realized as a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0012] The decoding device and the like of the present disclosure can output three-dimensional data at different resolutions.

[0013] FIG. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to the first embodiment. FIG. 2 is a diagram showing the configuration of point cloud data according to the first embodiment. FIG. 3 is a diagram showing an example of the configuration of a data file in which information about point cloud data is described according to the first embodiment. FIG. 4 is a diagram showing the configuration of three-dimensional mesh data according to the first embodiment. FIG. 5 is a diagram showing an example of the configuration of a data file in which information about three-dimensional mesh data is described according to the first embodiment. FIG. 6 is a diagram for explaining a three-dimensional model according to the first embodiment. FIG. 7 is a diagram showing types of three-dimensional data according to the first embodiment. FIG. 8 is a diagram for explaining the encoding process of three-dimensional data according to the first embodiment. FIG. 9 is a diagram for explaining the decoding process of three-dimensional data according to the first embodiment. FIG. 10 is a diagram showing a two-dimensional schematic diagram of tiles and slices of three-dimensional data according to the first embodiment. FIG. 11 is a block diagram showing an example of the functional configuration of a server and a terminal according to the first embodiment. FIG. 12 is a block diagram showing another example of the data generation unit of the server according to the first embodiment. FIG. 13 is a diagram for explaining the relationship between three-dimensional space and encoded data according to the first embodiment. Fig. 14 is a diagram showing an example of the syntax of a coding scheme unit in Embodiment 1. Fig. 15 is a diagram showing an example of the syntax of a coded point group in Embodiment 1. Fig. 16 is a diagram showing an example of the syntax of a coded mesh in Embodiment 1. Fig. 17 is a diagram showing an example of the syntax of a coded 3D model in Embodiment 1. Fig. 18 is a diagram showing an example of the syntax of 3D data information in Embodiment 1. Fig. 19 is a diagram for explaining the data structure of a coded point group in Embodiment 1. Fig. 20 is a diagram for explaining the data structure of a coded mesh in Embodiment 1. Fig. 21 is a diagram for explaining the data structure of a coded 3D model in Embodiment 1. Fig. 22 is a diagram showing an example of a plurality of 3D spaces in two dimensions in Embodiment 1. Fig. 23 is a diagram showing an example of a bounding box in Embodiment 1.FIG. 24 is a diagram showing an example of the syntax of 3D space information in Embodiment 1. FIG. 25 is a flowchart showing an example of partial decoding in Embodiment 1. FIG. 26 is a diagram showing an example of a 3D space region to be partially decoded in Embodiment 1. FIG. 27 is a diagram showing an example of the data structure of a partially decoded encoded point group in Embodiment 1. FIG. 28 is a diagram showing an example of the data structure of a partially decoded encoded mesh in Embodiment 1. FIG. 29 is a diagram showing an example of the data structure of a partially decoded encoded 3D model in Embodiment 1. FIG. 30 is a diagram showing an example of the configuration of a decoding device in Embodiment 1. FIG. 31 is a flowchart showing an example of a decoding method used by the decoding device in Embodiment 1. FIG. 32 is a flowchart showing another example of a decoding method used by the decoding device. FIG. 33 is a diagram showing an example of the configuration of an encoding device. FIG. 34 is a flowchart showing an example of an encoding method used by the encoding device. FIG. 35 is a diagram for explaining processing during training of a 3D generation model in Embodiment 2. FIG. 36 is a diagram for explaining processing for generating a still image of a subject viewed from an arbitrary viewpoint using a 3D generation model in Embodiment 2. FIG. 37 is a diagram for explaining a moving image generation method using a 3D data generation model of Example 1 in Embodiment 2. FIG. 38 is a diagram for explaining a first example of the configuration of an encoding device of Example 1 in Embodiment 2. FIG. 39 is a diagram for explaining a first example of the configuration of a decoding device of Example 1 in Embodiment 2. FIG. 40 is a diagram for explaining a second example of the configuration of an encoding device of Example 1 in Embodiment 2. FIG. 41 is a diagram for explaining a second example of the configuration of a decoding device of Example 1 in Embodiment 2. FIG. 42 is a diagram for explaining a moving image generation method using an extended 3D data generation model of Example 2 in Embodiment 2. FIG. 43 is a diagram for explaining a first example of the configuration of an encoding device of Example 2 in Embodiment 2. FIG. 44 is a diagram for explaining a first example of the configuration of a decoding device of Example 2 in Embodiment 2. FIG. 45 is a diagram for explaining a second example of the configuration of an encoding device of Example 2 in Embodiment 2. FIG. 46 is a diagram for explaining a second example of the configuration of a decoding device of Example 2 in Embodiment 2.Fig. 47 is a diagram illustrating a moving image generation method using an extended 3D data generation model according to a modification of Embodiment 2. Fig. 48 is a diagram illustrating a moving image generation method using a 3D data generation model according to a modification of Embodiment 2. Fig. 49 is a diagram illustrating an example of the configuration of an encoding device according to Embodiment 2. Fig. 50 is a flowchart illustrating an example of an encoding method by the encoding device according to Embodiment 2. Fig. 51 is a diagram illustrating an example of the configuration of a decoding device according to Embodiment 2. Fig. 52 is a flowchart illustrating an example of a decoding method by the decoding device according to Embodiment 2. Fig. 53 is a diagram illustrating an example of the configuration of an encoding device. Fig. 54 is a diagram illustrating an example of the configuration of a decoding device.

[0014] An encoding device according to a first aspect of the present disclosure includes a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time, and generates a bitstream by encoding the acquired first three-dimensional data generation model and second three-dimensional data generation model, and when viewpoint information including a viewpoint and a line of sight is input, each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as viewed from the viewpoint and the line of sight.

[0015] This makes it possible to generate a bitstream that includes a first three-dimensional data generation model that can obtain a two-dimensional image corresponding to a first time according to arbitrary viewpoint information, and a second three-dimensional data generation model that can obtain a two-dimensional image corresponding to a second time, thereby generating a bitstream that compresses data that can obtain moving images from arbitrary viewpoints. This makes it possible to reduce the memory capacity required to store data that can obtain moving images from arbitrary viewpoints, or the network bandwidth required to transmit that data.

[0016] An encoding device according to a second aspect of the present disclosure is an encoding device according to the first aspect, wherein each of the first three-dimensional data generation model and the second three-dimensional data generation model is a learning model using a neural network.

[0017] A coding device according to a third aspect of the present disclosure is a coding device according to the first or second aspect, wherein the bitstream includes first time information indicating the first time and second time information indicating the second time.

[0018] A coding device according to a fourth aspect of the present disclosure is a coding device according to the third aspect, wherein the bitstream includes a first frame number corresponding to the first time and a second frame number corresponding to the second time.

[0019] An encoding device according to a fifth aspect of the present disclosure is an encoding device according to any one of the first to fourth aspects, wherein the bit stream includes frame rate information regarding frame rates of a plurality of training images used to generate the first three-dimensional data generation model and the second three-dimensional data generation model, and the plurality of training images are two-dimensional images obtained by photographing at a plurality of different times.

[0020] An encoding device according to a sixth aspect of the present disclosure is an encoding device according to any one of the first to fourth aspects, wherein the bitstream includes viewpoint information including viewpoints and line-of-sight directions of a plurality of training images used in generating the first three-dimensional data generation model and the second three-dimensional data generation model.

[0021] A coding device according to a seventh aspect of the present disclosure is a coding device according to the sixth aspect, wherein the plurality of training images are two-dimensional images obtained by photographing the subject from mutually different viewpoints and viewing directions, and the viewpoint information includes the mutually different viewpoints and viewing directions.

[0022] An encoding device according to an eighth aspect of the present disclosure is an encoding device according to any one of the first to seventh aspects, wherein the circuit, in encoding the second three-dimensional data generation model, calculates difference information indicating the difference between the first three-dimensional data generation model and the second three-dimensional data generation model, and the bit stream includes the difference information.

[0023] A coding device according to a ninth aspect of the present disclosure is a coding device according to the eighth aspect, wherein the difference includes a difference between weight parameters associated with nodes included in the first three-dimensional data generation model and the second three-dimensional data generation model.

[0024] A coding device according to a tenth aspect of the present disclosure is a coding device according to the eighth or ninth aspect, wherein the bitstream includes reference information indicating that the differential information was calculated by reference to the first three-dimensional data generation model.

[0025] An encoding device according to an eleventh aspect of the present disclosure is an encoding device according to any one of the first to tenth aspects, wherein the first time corresponds to a random access point, and the first 3D data generation model is encoded using intra prediction, or is encoded using inter prediction with a prediction value of 0.

[0026] A coding device according to a twelfth aspect of the present disclosure is a coding device according to the eleventh aspect, wherein the first three-dimensional data generation model and the second three-dimensional data generation model are included in one set of a plurality of sets, and the first three-dimensional data generation model is the first in data order among the plurality of three-dimensional data generation models included in the one set.

[0027] A coding device according to a thirteenth aspect of the present disclosure is a coding device according to the twelfth aspect, wherein the bitstream includes permission information indicating whether or not, in encoding each of the plurality of three-dimensional data generation models, the three-dimensional data generation model is allowed to reference three-dimensional data generation models included in other sets.

[0028] A coding device according to a fourteenth aspect of the present disclosure is a coding device according to any one of the first to thirteenth aspects, wherein the first three-dimensional data generation model corresponds to a first period including the first time, and the second three-dimensional data generation model corresponds to a second period including the second time.

[0029] A coding device according to a fifteenth aspect of the present disclosure is a coding device according to the fourteenth aspect, wherein the plurality of first learning images used to generate the first three-dimensional data generation model are two-dimensional images obtained by photographing at a plurality of different times during the first period.

[0030] A coding device according to a sixteenth aspect of the present disclosure is a coding device according to the fourteenth or fifteenth aspect, wherein the first three-dimensional data generation model outputs a two-dimensional image of the subject at the input time when a time included in the first period is input.

[0031] A coding device according to a seventeenth aspect of the present disclosure is a coding device according to any one of the fourteenth to sixteenth aspects, wherein the bitstream includes numerical information indicating the maximum number of images that can be generated by the first three-dimensional data generation model.

[0032] An encoding device according to an 18th aspect of the present disclosure is an encoding device according to the 15th aspect, wherein the bitstream includes first information regarding the plurality of first learning images, and the first information includes a plurality of viewpoints, a plurality of line-of-sight directions, and a plurality of different timings corresponding to the plurality of first learning images.

[0033] A coding device according to a 19th aspect of the present disclosure is a coding device according to any one of the 14th to 18th aspects, wherein the first period or the second period is dynamically determined depending on the subject.

[0034] An encoding device according to a twentieth aspect of the present disclosure is an encoding device according to any one of the first to nineteenth aspects, wherein the circuit stores the generated first three-dimensional data generation model in the memory, and generates the second three-dimensional data generation model based on the first three-dimensional data generation model stored in the memory.

[0035] An encoding device according to a 21st aspect of the present disclosure is an encoding device according to any one of the 1st to 19th aspects, wherein the circuit stores the generated first three-dimensional data generation model and the second three-dimensional data generation model in the memory, generates an initial model based on the first three-dimensional data generation model and the second three-dimensional data generation model stored in the memory, and generates a third three-dimensional data generation model corresponding to a third time based on the initial model.

[0036] A decoding device according to a 22nd aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a bitstream and decodes from the bitstream a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time, and when viewpoint information including a viewpoint and a line of sight is input, each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as viewed from the viewpoint and the line of sight.

[0037] This makes it possible to generate a bitstream that includes a first three-dimensional data generation model that can obtain a two-dimensional image corresponding to a first time according to arbitrary viewpoint information, and a second three-dimensional data generation model that can obtain a two-dimensional image corresponding to a second time, thereby generating a bitstream that compresses data that can obtain moving images from arbitrary viewpoints. This makes it possible to reduce the memory capacity required to store data that can obtain moving images from arbitrary viewpoints, or the network bandwidth required to transmit that data.

[0038] A decoding device according to a 23rd aspect of the present disclosure is a decoding device according to the 22nd aspect, wherein each of the first three-dimensional data generation model and the second three-dimensional data generation model is a learning model using a neural network.

[0039] A decoding device according to a 24th aspect of the present disclosure is a decoding device according to the 22nd or 23rd aspect, wherein the bitstream includes first time information indicating the first time and second time information indicating the second time.

[0040] A decoding device according to a 25th aspect of the present disclosure is a decoding device according to the 24th aspect, wherein the bitstream includes a first frame number corresponding to the first time and a second frame number corresponding to the second time.

[0041] A decoding device according to a 26th aspect of the present disclosure is a decoding device according to any one of the 22nd to 25th aspects, wherein the bitstream includes frame rate information regarding the frame rates of a plurality of training images used to generate the first three-dimensional data generation model and the second three-dimensional data generation model, and the plurality of training images are two-dimensional images obtained by photographing at a plurality of different times.

[0042] A decoding device according to a 27th aspect of the present disclosure is a decoding device according to any one of the 22nd to 25th aspects, wherein the bitstream includes viewpoint information including the viewpoint and line of sight of a plurality of training images used in generating the first three-dimensional data generation model and the second three-dimensional data generation model.

[0043] A decoding device according to a 28th aspect of the present disclosure is a decoding device according to the 27th aspect, wherein the plurality of learning images are two-dimensional images obtained by photographing the subject from mutually different viewpoints and viewing directions, and the viewpoint information includes the mutually different viewpoints and viewing directions.

[0044] A decoding device according to a 29th aspect of the present disclosure is a decoding device according to any one of the 22nd to 28th aspects, wherein the bitstream includes difference information indicating the difference between the first three-dimensional data generation model and the second three-dimensional data generation model.

[0045] A decoding device according to a 30th aspect of the present disclosure is a decoding device according to the 29th aspect, wherein the difference includes a difference between weight parameters associated with nodes included in the first three-dimensional data generation model and the second three-dimensional data generation model.

[0046] A decoding device according to a 31st aspect of the present disclosure is a decoding device according to the 29th or 30th aspect, wherein the bitstream includes reference information indicating that the differential information was calculated with reference to the first three-dimensional data generation model.

[0047] A decoding device according to a 32nd aspect of the present disclosure is a decoding device according to any one of the 22nd to 31st aspects, wherein the first time corresponds to a random access point, and the first 3D data generation model is decoded by intra prediction or by inter prediction with a prediction value of 0.

[0048] A decoding device according to a 33rd aspect of the present disclosure is a decoding device according to the 32nd aspect, wherein the first three-dimensional data generation model and the second three-dimensional data generation model are included in one set of a plurality of sets, and the first three-dimensional data generation model is the first in data order among the plurality of three-dimensional data generation models included in the one set.

[0049] A decoding device according to a 34th aspect of the present disclosure is a decoding device according to the 33rd aspect, wherein the bitstream includes permission information indicating whether or not, in decoding each of the plurality of three-dimensional data generation models, the three-dimensional data generation model is allowed to refer to three-dimensional data generation models included in other sets.

[0050] A decoding device according to the 35th aspect of the present disclosure is a decoding device according to any one of the 22nd to 34th aspects, wherein the first three-dimensional data generation model corresponds to a first period including the first time, and the second three-dimensional data generation model corresponds to a second period including the second time.

[0051] A decoding device according to a 36th aspect of the present disclosure is a decoding device according to the 35th aspect, wherein the plurality of first learning images used to generate the first three-dimensional data generation model are two-dimensional images obtained by photographing at a plurality of different times during the first period.

[0052] A decoding device according to a 37th aspect of the present disclosure is a decoding device according to the 35th or 36th aspect, wherein when a time included in the first period is input, the first three-dimensional data generation model outputs a two-dimensional image of the subject at the input time.

[0053] A decoding device according to a 38th aspect of the present disclosure is a decoding device according to any one of the 35th to 37th aspects, wherein the bitstream includes numerical information indicating the maximum number of images that can be generated by the first three-dimensional data generation model.

[0054] A decoding device according to a 39th aspect of the present disclosure is a decoding device according to a 36th aspect, wherein the bitstream includes first information regarding the plurality of first learning images, and the first information includes a plurality of viewpoints, a plurality of line-of-sight directions, and a plurality of different timings corresponding to the plurality of first learning images.

[0055] A decoding device according to a 40th aspect of the present disclosure is a decoding device according to any one of the 35th to 39th aspects, wherein the first period or the second period is dynamically determined depending on the subject.

[0056] A decoding device according to a 41st aspect of the present disclosure is a decoding device according to any one of the 22nd to 40th aspects, wherein the circuit stores the generated first three-dimensional data generation model in the memory, and generates the second three-dimensional data generation model based on the first three-dimensional data generation model stored in the memory.

[0057] A decoding device according to a 42nd aspect of the present disclosure is a decoding device according to any one of the 22nd to 40th aspects, wherein the circuit stores the generated first three-dimensional data generation model and the second three-dimensional data generation model in the memory, generates an initial model based on the first three-dimensional data generation model and the second three-dimensional data generation model stored in the memory, and generates a third three-dimensional data generation model corresponding to a third time based on the initial model.

[0058] These comprehensive or specific aspects may be realized as a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0059] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that represents a superordinate concept will be described as optional components.

[0060] (Embodiment 1) The configuration of a three-dimensional data encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 1001, a three-dimensional data decoding system 1002, a sensor terminal 1003, and an external connection unit 1004.

[0061] The three-dimensional data encoding system 1001 generates encoded data or multiplexed data by encoding three-dimensional data. Note that the three-dimensional data encoding system 1001 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 1001.

[0062] The three-dimensional data encoding system 1001 includes a three-dimensional data generation system 1011, a presentation unit 1012, an encoding unit 1013, a multiplexing unit 1014, an input / output unit 1015, and a control unit 1016. The three-dimensional data generation system 1011 includes a sensor information acquisition unit 1017 and a three-dimensional data generation unit 1018.

[0063] The sensor information acquisition unit 1017 acquires a sensor signal from the sensor terminal 1003 and outputs the sensor signal to the three-dimensional data generation unit 1018. The three-dimensional data generation unit 1018 generates three-dimensional data from the sensor signal and outputs the three-dimensional data to the encoding unit 1013.

[0064] The presentation unit 1012 presents the sensor signal or the three-dimensional data to the user. For example, the presentation unit 1012 displays information or an image based on the sensor signal or the three-dimensional data.

[0065] The encoding unit 1013 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 1014. The additional information includes, for example, a sensor signal.

[0066] The multiplexing unit 1014 generates multiplexed data by multiplexing the coded data input from the coding unit 1013, the control information, and the additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0067] The input / output unit 1015 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 1016 (or the application execution unit) controls each processing unit. That is, the control unit 1016 controls encoding, multiplexing, etc. The control unit 1016 may also control demultiplexing, decoding, or presentation.

[0068] The sensor signal may be input to the encoding unit 1013 or the multiplexing unit 1014. The input / output unit 1015 may output the three-dimensional data or the encoded data directly to the outside.

[0069] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 1001 is input to the three-dimensional data decoding system 1002 via the external connection unit 1004 .

[0070] The three-dimensional data decoding system 1002 generates three-dimensional data by decoding encoded data or multiplexed data. Note that the three-dimensional data decoding system 1002 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 1002.

[0071] The three-dimensional data decoding system 1002 includes a sensor information acquisition unit 1021 , an input / output unit 1022 , a demultiplexing unit 1023 , a decoding unit 1024 , a presentation unit 1025 , a user interface 1026 , and a control unit 1027 .

[0072] The sensor information acquisition unit 1021 acquires a sensor signal from the sensor terminal 1003 .

[0073] The input / output unit 1022 acquires a transmission signal, decodes multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 1023 .

[0074] The demultiplexing unit 1023 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoding unit 1024 .

[0075] The decoding unit 1024 reconstructs the point cloud data by decoding the encoded data.

[0076] The presentation unit 1025 presents the point cloud data to the user. For example, the presentation unit 1025 displays information or an image based on the point cloud data. The user interface 1026 acquires instructions based on user operations. The control unit 1027 (or the application execution unit) controls each processing unit. That is, the control unit 1027 controls demultiplexing, decoding, presentation, etc.

[0077] The input / output unit 1022 may acquire point cloud data or encoded data directly from the outside. The presentation unit 1025 may acquire additional information such as a sensor signal and present information based on the additional information. The presentation unit 1025 may also present information based on a user instruction acquired by the user interface 1026.

[0078] The sensor terminal 1003 generates a sensor signal, which is information obtained by a sensor. The sensor terminal 1003 is a terminal equipped with a sensor or a camera, and may be, for example, a mobile object such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

[0079] The sensor signals obtainable by the sensor terminal 1003 include, for example, (1) a signal indicating the distance between the sensor terminal 1003 and an object or the reflectance of the object obtained from a LIDAR, millimeter-wave radar, or infrared sensor, and (2) a signal indicating the distance between a camera and an object or the reflectance of the object obtained from multiple monocular camera images or stereo camera images. The sensor signals may also include the attitude, orientation, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor signals may also include temperature, air pressure, humidity, magnetism, etc.

[0080] The external connection unit 1004 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0081] Next, point cloud data will be described. Fig. 2 is a diagram showing the structure of point cloud data. Fig. 3 is a diagram showing an example of the structure of a data file in which information about point cloud data is written.

[0082] Point cloud data includes data on multiple points. Each point includes location information (three-dimensional coordinates) and attribute information for that location. A collection of multiple points is called a point cloud. For example, a point cloud can represent the three-dimensional shape of an object.

[0083] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.

[0084] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0085] The data file configuration example shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.

[0086] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0087] Next, the three-dimensional mesh data will be described. Fig. 4 is a diagram showing the structure of the three-dimensional mesh data. Fig. 5 is a diagram showing an example of the structure of a data file in which information about the three-dimensional mesh data is written.

[0088] Three-dimensional mesh data is a data format used in computer graphics (CG) and represents the three-dimensional shape of an object using a collection of multiple pieces of surface information. Each piece of surface information represents a polygon such as a triangle or a quadrangle. Three-dimensional mesh data is also called a polygon or polygon mesh.

[0089] The components are a set of three-dimensional point clouds, vertices that are multiple three-dimensional points of the three-dimensional point cloud, edges connecting two vertices at multiple three-dimensional points, and faces surrounded by multiple edges. A three-dimensional point cloud is a set of points that include position information in three-dimensional space and attribute information corresponding to the position information. Note that three-dimensional points may simply be referred to as points.

[0090] A vertex may have attribute information such as color information, reflectance, and normal vector for a three-dimensional point. The relationship between the vertices that make up an edge or a face may be indicated by information called connectivity. Note that a vertex may be expressed as a position. The front and back of a face may be expressed by the direction of the normal vector for the three-dimensional point. Also, a vertex may have attribute information for the face.

[0091] An example of a mesh data file format is an object file. In a mesh data file such as that shown in Fig. 5, the position information G(1) to G(N) of the N vertices that make up the mesh and the attribute information A(1) to A(N) of the vertices are indicated as vertex information. In a mesh data file, the vertex information does not need to include attribute information.

[0092] Furthermore, the attribute information does not have to correspond one-to-one to the vertices. The mesh data file in Fig. 5 shows an example in which the three-dimensional mesh data has M pieces of attribute information A2.

[0093] Face information is represented by a combination of vertex indices, where n[1,3,4] indicates a triangular face made up of three vertices: vertex n=1, vertex n=3, and vertex n=4.

[0094] Furthermore, m[2,4,6] indicates that the attribute information of m=2, the attribute information of m=4, and the attribute information of m=6 in the attribute information A2 correspond to three vertices, respectively. Note that although an example in which a face is formed by three vertices has been shown here, the number of vertices of a face is not limited to three, as long as it is an integer equal to or greater than three. For example, if the face is a quadrangle, the number of vertices is four, and if the face is a polygon, the number of vertices is equal to the number of vertices that form the polygon.

[0095] The attribute information A2 may be represented in a file separate from the mesh data file, or may include pointer information thereto. For example, the attribute information may be stored in a two-dimensional attribute map file, and the attribute map file name and two-dimensional coordinates in the attribute map may be represented by the attribute information A2 of the mesh data file. In this way, the attribute information A2 may be included in the mesh data file or represented in a file separate from the mesh data file. Either method can be used to specify attribute information for a three-dimensional point.

[0096] Next, the three-dimensional model will be described with reference to Fig. 6.

[0097] A three-dimensional model is a model generated based on two-dimensional or three-dimensional data.

[0098] The three-dimensional model learning unit 1031 learns, for example, two-dimensional data (two-dimensional images) or three-dimensional data (point clouds or meshes) to generate a three-dimensional model, which is a network model in which three-dimensional shapes and attribute information corresponding to the three-dimensional shapes are learned using a neural network or the like.

[0099] The three-dimensional model learning unit 1031 may generate a three-dimensional model by learning using Neural Radiance Fields (NeRF) based on two-dimensional images. The three-dimensional model learning unit 1031 may generate a three-dimensional model after converting the two-dimensional images into three-dimensional data by performing photogrammetry using the two-dimensional images. The three-dimensional model may be generated using three-dimensional data acquired by a sensor (distance sensor).

[0100] The three-dimensional model data is the elements that make up the three-dimensional model, and includes information indicating the structure of the network model, feature quantities, etc. The three-dimensional model data includes, for example, information about the components of the neural network. The information about the components includes, for example, multiple layers such as an input layer, an intermediate layer, and an output layer, nodes in each layer, weight coefficients for the nodes, and transformation functions for the nodes.

[0101] The three-dimensional model encoding unit 1032 may encode the three-dimensional model data and transmit the encoded three-dimensional model data.

[0102] The three-dimensional model decoding unit 1033 receives the transmitted encoded three-dimensional model data and decodes the three-dimensional model based on the encoded three-dimensional model data.

[0103] The rendering reconstruction unit 1034 reconstructs (generates) two-dimensional data (two-dimensional image) or three-dimensional data (point cloud or mesh) based on the decoded three-dimensional model. For example, when using a three-dimensional model modeled using NeRF, the rendering reconstruction unit 1034 acquires viewpoint position or line-of-sight vector information, generates rendered two-dimensional data (two-dimensional image) based on the three-dimensional model and the viewpoint position or line-of-sight vector, and outputs the two-dimensional data. The generated two-dimensional data represents a three-dimensional object as seen from the viewpoint position, or a two-dimensional image of the three-dimensional object as seen from the line of sight indicated by the line-of-sight vector. The three-dimensional object is the three-dimensional object of the subject that was the source of the two-dimensional data or three-dimensional data input to the three-dimensional model learning unit 1031.

[0104] Next, the types of three-dimensional data will be explained. Fig. 7 shows the types of three-dimensional data. As shown in Fig. 7, three-dimensional data includes static objects and dynamic objects.

[0105] A static object is three-dimensional data at any time (a certain time). A dynamic object is three-dimensional data that changes over time. Hereinafter, point cloud data at a certain time will be referred to as a PCC frame or frame. Mesh data at a certain time will be referred to as a mesh frame or frame.

[0106] The object may be three-dimensional data with a certain area restriction, such as ordinary video data, or may be three-dimensional data with no area restriction, such as map information.

[0107] There may also be sparse point cloud data (sparse mesh data) and dense point cloud data (dense mesh data) with points of various densities.

[0108] Each processing unit will be described in detail below. Sensor information is acquired by various methods, such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The three-dimensional data generation unit 1018 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 1017. The three-dimensional data generation unit 1018 generates position information (geometry information) as the point cloud data, and adds attribute information for the position information to the position information.

[0109] The three-dimensional data generation unit 1018 may process the point cloud data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 1018 may reduce the amount of data by deleting point clouds with overlapping positions. Furthermore, the three-dimensional data generation unit 1018 may convert the position information (such as by shifting, rotating, or normalizing) or process the point cloud data to generate mesh data. Furthermore, the three-dimensional data generation unit 1018 may render the attribute information.

[0110] In FIG. 1, the three-dimensional data generation system 1011 is included in the three-dimensional data encoding system 1001, but it may be provided independently outside the three-dimensional data encoding system 1001.

[0111] The encoding unit 1013 generates encoded data by encoding the three-dimensional data based on a predetermined encoding method. The encoding methods include G-PCC (an encoding method using position information), V-PCC (an encoding method using a video codec), Draco (a mesh encoding method), and V-DMC (a mesh encoding method). The encoding method is not limited to these methods, and may be, for example, a method of encoding a dynamic mesh, or another method that combines these methods.

[0112] The decoding unit 1024 decodes the coded data based on a predetermined coding method to decode the three-dimensional data.

[0113] The multiplexing unit 1014 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the encoded data of the three-dimensional data, the multiplexing unit 1014 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 1014 may also multiplex attribute information related to sensor information or point cloud data.

[0114] Multiplexing methods or file formats include ISOBMFF, MPEG-DASH, which is an ISOBMFF-based transmission method, MMT, MPEG-2 TS Systems, and RTP.

[0115] The demultiplexing unit 1023 extracts the coded data of the three-dimensional data, other media, time information, and the like from the multiplexed data.

[0116] The input / output unit 1015 transmits the multiplexed data using a method suited to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 1015 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0117] The communication protocol used may be http, ftp, TCP, UDP, etc. A PULL type communication method or a PUSH type communication method may be used.

[0118] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter waves, etc. are used.

[0119] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0120] Next, a process of dividing three-dimensional data into one or more three-dimensional data will be described. Fig. 8 is a diagram for explaining a process of encoding three-dimensional data. Fig. 9 is a diagram for explaining a process of decoding three-dimensional data.

[0121] 8 , the data dividing unit 1041 divides the three-dimensional data into one or more three-dimensional spaces and generates one or more divided three-dimensional data (i.e., one or more divided three-dimensional data). The encoding unit 1042 may encode one or more divided three-dimensional data to generate encoded data. The data dividing unit 1041 and the encoding unit 1042 may be included in a single encoding device as components of the single encoding device, or may be included in separate devices.

[0122] Each of the one or more three-dimensional spaces may be referred to as a tile or a space. The three-dimensional space may be, for example, a bounding box. Furthermore, the three-dimensional data included in each of the divided three-dimensional spaces may be referred to as a slice. A slice is divided three-dimensional data and includes any of a point cloud, a mesh, and a three-dimensional model having position information (geometry) or attribute information (attribute). Each of the multiple slices is coded by the coding unit 1042 for each component element and output as coded data. The coded data includes the coded multiple slices.

[0123] As shown in FIG. 9 , in the decoding process, the decoding unit 1051 decodes one or more pieces of divided three-dimensional data (one or more slices) based on the encoded data. The data combining unit 1052 combines the one or more pieces of divided three-dimensional data to restore (generate) three-dimensional data. The decoding unit 1051 and the data combining unit 1052 may be included in a single decoding device as components of the single decoding device, or may be included in separate devices. The one or more pieces of divided three-dimensional data decoded by the decoding unit 1051 do not need to be combined. The decoding unit 1051 may decode some of the one or more pieces of divided three-dimensional data based on part of the encoded data and output the decoded some of the divided three-dimensional data. In this case, the decoding device does not need to have the data combining unit 1052.

[0124] FIG. 10 is a two-dimensional schematic diagram showing tiles and slices of three-dimensional data.

[0125] When encoding multiple slices, the encoding device may encode them using a dependency relationship between the multiple slices, or may encode them without using the dependency relationship. When encoding without using the dependency relationship, the encoding device can encode each slice independently, and can reduce processing time by encoding multiple slices using parallel processing. Furthermore, when multiple slices are encoded without using the dependency relationship, the decoding device can decode each slice independently, and can reduce processing time by decoding multiple slices using parallel processing. Furthermore, the decoding device can reduce the amount of processing by performing partial decoding, which decodes some of the multiple slices.

[0126] When encoding using a dependency relationship, the encoding device signals an identifier indicating the dependency relationship and encodes the data in order starting with the dependent data. When multiple slices are encoded using a dependency relationship, the decoding device decodes the data in order starting with the dependent data based on the identifier.

[0127] The division of the three-dimensional data may be performed using any number of divisions and any division method. The division of the three-dimensional data may involve determining the shape of an object and dividing each object into multiple three-dimensional points. The division of the three-dimensional data may also be performed based on the number of three-dimensional points included in a slice. In other words, an upper limit on the number of three-dimensional points included in one slice may be set. The three-dimensional data may also be divided based on whether or not it is included in the three-dimensional space (tile information) using map information or location information. Multiple tile shapes may overlap.

[0128] By dividing the three-dimensional data into a plurality of divided three-dimensional data in this way, adaptive encoding according to the content or object and parallel processing in decoding become possible.

[0129] Next, a method for selecting three-dimensional data to be presented or transmitted from among a plurality of three-dimensional data will be described.

[0130] The server stores multiple pieces of three-dimensional data for the same space. The server stores, for example, point cloud data and mesh data for the same space. The server is an example of an encoding device. The terminal switches the three-dimensional data acquired from the server based on the purpose of the terminal and presents the switched three-dimensional data. The terminal may be, for example, a terminal that analyzes three-dimensional data. In this case, the terminal may switch the three-dimensional data to be presented based on the purpose of analysis or presentation, or user operation. The terminal is an example of a decoding device.

[0131] The switching of three-dimensional data may involve switching between presenting a point cloud or a mesh as the three-dimensional data. Furthermore, the switching of three-dimensional data may involve switching between transmitting a point cloud or a mesh as the three-dimensional data. For example, the terminal may transmit a user's selection result to a server, receive (download) three-dimensional data based on the selection result from the server, and present the received three-dimensional data. The three-dimensional data (point cloud or mesh) may or may not be encoded by the server. If the three-dimensional data is encoded, the terminal may receive the encoded three-dimensional data from the server, decode the three-dimensional data based on the received encoded three-dimensional data, and present the decoded three-dimensional data.

[0132] Next, a description will be given of the configuration of the server 1070 and the terminal 1090. Fig. 11 is a block diagram showing an example of the functional configuration of the server and the terminal.

[0133] The server 1070 includes a data generation unit 1071 , a synchronization unit 1075 , a point cloud encoding unit 1076 , a mesh encoding unit 1077 , a model encoding unit 1078 , a multiplexing unit 1079 , and a data extraction unit 1080 .

[0134] The data generation unit 1071 generates three-dimensional data based on at least one of two-dimensional data and three-dimensional data. The generated three-dimensional data includes at least two of point cloud data, mesh data, and three-dimensional model data. The data generation unit 1071 includes a point cloud generation unit 1072, a mesh generation unit 1073, and a model generation unit 1074. The data generation unit 1071 may include at least two of the point cloud generation unit 1072, the mesh generation unit 1073, and the model generation unit 1074. The point cloud generation unit 1072 generates point cloud data based on at least one of the two-dimensional data and the three-dimensional data. The mesh generation unit 1073 generates mesh data based on at least one of the two-dimensional data and the three-dimensional data. The model generation unit 1074 generates three-dimensional model data by performing machine learning based on at least one of the two-dimensional data and the three-dimensional data.

[0135] The two-dimensional data input to the data generation unit 1071 may be a two-dimensional image acquired by a camera. The three-dimensional data input to the data generation unit 1071 may be point cloud data acquired by a sensor such as LiDAR of a space such as a construction site, factory, or office. The data generation unit 1071 may generate color information corresponding to each point included in the point cloud data of the three-dimensional data as attribute information using a two-dimensional image of the two-dimensional data. The three-dimensional data generated by the data generation unit 1071 may be divided into any space. The point cloud data, mesh data, and three-dimensional model data may each be divided into any space.

[0136] The synchronization unit 1075 synchronizes the spatial positions of the point cloud data, mesh data, and 3D model data generated by the data generation unit 1071 or the time of each data (playback time, decoding time, acquisition time, etc.). The time of each data is the playback time, decoding time, acquisition time, etc. The synchronization unit 1075 may generate synchronization information for synchronization without synchronizing the point cloud data, mesh data, and 3D model data. The synchronization unit 1075 may synchronize at least two types of 3D data among the point cloud data, mesh data, and 3D model data generated by the data generation unit 1071, or may perform processing to generate synchronization information (synchronization signal) for synchronization, and may not necessarily perform processing to synchronize three types of 3D data (synchronization processing).

[0137] The point cloud encoding unit 1076 encodes the point cloud data after synchronization processing has been performed by the synchronization unit 1075. Note that the point cloud encoding unit 1076 does not necessarily have to encode the point cloud data. The point cloud data may be encoded in advance, or may be encoded in response to a request from the terminal 1090.

[0138] The mesh encoding unit 1077 encodes the mesh data after the synchronization process has been performed by the synchronization unit 1075 .

[0139] The model encoding unit 1078 encodes the three-dimensional model data after the synchronization process has been performed by the synchronization unit 1075 .

[0140] The multiplexing unit 1079 multiplexes the encoded point cloud data (encoded point cloud), the encoded mesh data (encoded mesh data), the encoded 3D model data, and the synchronization information using a predetermined format or a predetermined multiplexing method. Note that multiplexing by the multiplexing unit 1079 does not have to be performed. In this case, the server 1070 does not have to be equipped with the multiplexing unit 1079.

[0141] The data extraction unit 1080 extracts a portion of the multiplexed three-dimensional data in response to a request from the terminal 1090, and transmits the extracted portion of the three-dimensional data to the terminal 1090. Note that data extraction by the data extraction unit 1080 does not have to be performed. In this case, the server 1070 does not have to be equipped with the data extraction unit 1080. If data extraction by the data extraction unit 1080 is not performed, the server 1070 may transmit to the terminal 1090 the three-dimensional data multiplexed by the multiplexing unit 1079. Furthermore, if multiplexing by the multiplexing unit 1079 is not performed, the server 1070 may transmit to the terminal 1090 the coded point cloud data (coded point cloud), coded mesh data (coded mesh), coded three-dimensional model data (coded three-dimensional model), and synchronization information, or may transmit to the terminal 1090 a bitstream including the coded point cloud data (coded point cloud), coded mesh data (coded mesh), coded three-dimensional model data (coded three-dimensional model), and synchronization information.

[0142] The terminal 1090 includes a control unit 1091 , a decoding unit 1092 , and a presentation unit 1093 .

[0143] The control unit 1091 transmits a request for the part of the three-dimensional data to be presented to the server 1070. The control unit 1091 may accept an operation by the user to specify the part of the three-dimensional data.

[0144] The decoding unit 1092 decodes a portion of the three-dimensional data based on the bit stream (encoded data) acquired from the server 1070 .

[0145] The presentation unit 1093 renders and presents a portion of the decoded three-dimensional data.

[0146] The data generation unit 1071 in Fig. 11 may be realized by a data generation unit 1110 shown in Fig. 12. Fig. 12 is a block diagram showing another example of the data generation unit of the server.

[0147] The data generation unit 1110 includes a point cloud generation unit 1111 , a mesh generation unit 1112 , and a model generation unit 1113 .

[0148] The point cloud generation unit 1111 has the same function as the point cloud generation unit 1072. The point cloud generation unit 1111 acquires point cloud data obtained from the point cloud sensor 1101 and a two-dimensional image obtained from the camera 1102, and generates point cloud data based on the point cloud data and the two-dimensional image. The point cloud data generated by the point cloud generation unit 1111 includes position information of each point and attribute information corresponding to each point indicated by the position information, and includes attribute information (such as color information) extracted from the two-dimensional image.

[0149] The mesh generation unit 1112 generates mesh data based on the point cloud data generated by the point cloud generation unit 1111 .

[0150] The model generation unit 1113 has the same functions as the model generation unit 1074. The model generation unit 1113 acquires point cloud data obtained from the point cloud sensor 1101 and two-dimensional images obtained from the camera 1102, and performs machine learning based on the point cloud data and the two-dimensional images to generate three-dimensional model data.

[0151] The point cloud data, mesh data, and three-dimensional model data may be data generated independently of each other, as described in Fig. 11. The mesh data may be generated from the point cloud data, as described in Fig. 12. Note that the point cloud data may be generated from the mesh data.

[0152] A mesh may be generated from a point cloud, or a point cloud may be generated from a mesh.

[0153] The point cloud data, mesh data, and three-dimensional model data may be generated by the server 1070, or may be generated by a sensor or a terminal 1090 equipped with a sensor. The sensor is, for example, a point cloud sensor 1101 and a camera 1102.

[0154] Next, the relationship between the three-dimensional space and the coded data will be described with reference to Fig. 13.

[0155] As described above, the three-dimensional data includes, for example, any one of point cloud data, mesh data, and a three-dimensional model.

[0156] As shown in Fig. 13, when three-dimensional data is divided into three pieces of three-dimensional data in three three-dimensional spaces (tiles or spaces), the encoding device encodes each of the three divided pieces of three-dimensional data, attaches a header to each piece, and creates a data unit. The header signals (assigns) an identifier (Space_ID) of the space to which the encoded data of the data unit belongs, and an identifier (DataUnit_ID) of the data unit.

[0157] The data units are further given a header including an identifier of the data unit or length information of the data unit, and are then unitized to generate coding scheme units.

[0158] Next, the syntax of the coding method unit will be described. Fig. 14 is a diagram showing an example of the syntax of the coding method unit. Fig. 15 is a diagram showing an example of the syntax of the coding point group. Fig. 16 is a diagram showing an example of the syntax of the coding mesh. Fig. 17 is a diagram showing an example of the syntax of the coding 3D model.

[0159] The unit_type indicates the type of data unit stored in the encoding method unit, thereby specifying the type of data unit stored in the encoding method unit.

[0160] The length indicates the length of the data unit.

[0161] data() indicates the body of the data unit.

[0162] In Fig. 15, when unit_type indicates 0, it indicates that the data unit is position information (geometry) of the coded point group. When unit_type indicates 1, it indicates that the data unit is attribute information of the coded point group. When unit_type indicates 2, it indicates that the data unit is metadata of the coded point group.

[0163] In Fig. 16, when unit_type indicates 0, it indicates that the data unit is position information (geometry) of the coded mesh. When unit_type indicates 1, it indicates that the data unit is attribute information of the coded mesh. When unit_type indicates 2, it indicates that the data unit is metadata of the coded mesh.

[0164] In Fig. 17, when unit_type indicates 0, it indicates that the data unit is element 1 of the encoded 3D model. When unit_type indicates 1, it indicates that the data unit is element 2 of the encoded 3D model. When unit_type indicates 2, it indicates that the data unit is metadata of the encoded 3D model.

[0165] Note that the syntaxes shown in Figures 15 to 17 are merely examples and are not limited to the above configurations. These syntaxes may use a partial configuration of the syntax, or a type (classification) not described above may be used, or the order of the syntax components may be changed. For example, the syntax of the encoding method unit may have a configuration of the encoding method unit common to multiple encoding methods as shown in Figure 14, and may include the unit_type, length, and data() shown in Figures 15 to 17.

[0166] The coding method unit may further include a header indicating the type of the coding method unit. The coding unit type may include, for example, point_cloud_codec_unit indicating point cloud data, mesh_codec_unit indicating mesh data, and model_codec_unit indicating 3D model data. This allows multiple coding methods to be handled in an integrated manner.

[0167] FIG. 18 is a diagram showing an example of the syntax of three-dimensional data information.

[0168] In the syntax, when multiple encoding methods are stored in one format, the number of 3D data included in the format (number_of_3Dformat) and the type of 3D data (format_type) are indicated, and data of each format may be stored. This makes it possible to handle multiple encoding methods or 3D data in an integrated manner, and also to identify multiple encoding methods or 3D data.

[0169] 3Ddata_info indicates format structure information for storing multiple three-dimensional data.

[0170] number_of_3Dformat indicates the number of 3D formats used.

[0171] format_type indicates the type of format of the three-dimensional data to be stored. For example, the number of format_type and the format corresponding to that number may be defined as follows: When format_type indicates 0, it indicates that the format of the three-dimensional data to be stored is point cloud data. When format_type indicates 1, it indicates that the format of the three-dimensional data to be stored is mesh data. When format_type indicates 2, it indicates that the format of the three-dimensional data to be stored is G-PCC data (g-pcc). When format_type indicates 3, it indicates that the format of the three-dimensional data to be stored is V-DMC data (v-dmc). When format_type indicates 4, it indicates that the format of the three-dimensional data to be stored is three-dimensional model data (3Dmodel).

[0172] Next, the data structure of encoded data of multiple pieces of three-dimensional data will be described for each type of three-dimensional data. Fig. 19 is a diagram for explaining the data structure of an encoded point group. Fig. 20 is a diagram for explaining the data structure of an encoded mesh. Fig. 21 is a diagram for explaining the data structure of an encoded three-dimensional model.

[0173] The encoding device divides each of the multiple types of three-dimensional data into multiple pieces of three-dimensional data for each of multiple spatial regions, encodes each of the multiple pieces of divided three-dimensional data (i.e., multiple divided three-dimensional data), and generates encoded data.

[0174] A header is attached to each piece of encoded data, and at least one of a data_unit_id and a space_id is stored.

[0175] Here, data_unit_id is an identifier that identifies a data unit in the coded data and is unique within the coded data. Furthermore, space_id indicates identification information for a spatial region. If data_unit_id or space_id is common to multiple pieces of three-dimensional data, the same value is indicated in the multiple pieces of three-dimensional data.

[0176] 19 to 21, the data unit with data_unit_id=0 in the encoded point cloud, the data unit with data_unit_id=3 in the encoded mesh, and the data unit with data_unit_id=0 in the encoded 3D model are all assigned space_id=1, which means that they are 3D data contained in a common 3D space indicated by Space_ID #1.

[0177] The data and headers may be included in a bitstream structure such as a data unit or encoding method, or may be stored in a predetermined file format such as each box of ISOBMFF.

[0178] Next, three-dimensional space information will be described. Fig. 22 is a diagram showing an example of a plurality of three-dimensional spaces in two dimensions. Fig. 23 is a diagram showing an example of a bounding box. Fig. 24 is a diagram showing an example of the syntax of three-dimensional space information.

[0179] In the syntax of the three-dimensional space information, 3Dspace_info is information indicating the divided three-dimensional space. 3Dspace_info can be used for partial decoding.

[0180] The number_of_space indicates the number of divided three-dimensional spaces.

[0181] The space_id indicates the identifier of the divided three-dimensional space.

[0182] The three-dimensional space information includes bounding box information as information for defining the bounding box shown in FIG.

[0183] The bounding box information includes bounding_box_xyz and bounding_box_whd.

[0184] bounding_box_xyz indicates the coordinates of the reference point of the bounding box, and in the example of Fig. 23, is expressed as x, y, and z coordinate values ​​(x0, y0, z0).

[0185] The bounding_box_whd field indicates the size of the bounding box, which in the example of Fig. 23 is expressed as width w, height h, and depth d (w0, h0, d0).

[0186] The three-dimensional spatial information may also include an identifier of a data unit for each piece of coded data. However, the three-dimensional spatial information does not necessarily have to include the identifier. In other words, the identifier does not necessarily have to be signaled.

[0187] The pointcloud_id indicates the identifier of the data unit of the encoded point cloud of the space corresponding to the space_id.

[0188] mesh_id indicates the identifier of the data unit of the encoding mesh of the space corresponding to space_id.

[0189] model_id indicates the identifier of the data unit of the coded three-dimensional model of the space corresponding to space_id.

[0190] When the data unit does not indicate a space_id but indicates a data_unit_id, an identifier of the data unit for each piece of coded data may be stored in the information indicating each space of the three-dimensional space information, thereby allowing the three-dimensional space information to be associated with the divided three-dimensional coded data.

[0191] Furthermore, when a space_id is indicated in a data unit, the three-dimensional spatial information may be associated with an identifier of the data unit for each encoded data by the space_id. In this case, the identifier of the data unit for each encoded data does not need to be stored.

[0192] The division method, the origin of each divided space, and the bounding box size may be the same for mesh data and point cloud data, thereby sharing the three-dimensional spatial information of the point cloud data and the mesh data. Furthermore, the same three-dimensional spatial information may be used for both point cloud data and mesh data. In this way, the three-dimensional spatial information may be shared or the same three-dimensional spatial information may be used between multiple different types of three-dimensional data. Sharing the three-dimensional spatial information facilitates switching between different types of three-dimensional data (e.g., switching between presentations or transmissions). Furthermore, in a format that handles multiple three-dimensional data in an integrated manner, three-dimensional spatial information does not need to be provided for each piece of three-dimensional data; instead, one piece of three-dimensional spatial information can be used for each piece of three-dimensional data, thereby reducing the amount of three-dimensional spatial information.

[0193] In addition to the point cloud data and mesh data, the three-dimensional spatial information of the three-dimensional model may be synchronized with other types of three-dimensional data, or may be made common with the three-dimensional spatial information of other types of three-dimensional data.

[0194] Next, the relationship between the data structure of three-dimensional data and partial decoding will be described. Fig. 25 is a flowchart showing an example of partial decoding. Fig. 26 is a diagram showing an example of a three-dimensional spatial region that is the target of partial decoding. Fig. 27 is a diagram showing an example of the data structure of an encoded point group that is partially decoded. Fig. 28 is a diagram showing an example of the data structure of an encoded mesh that is partially decoded. Fig. 29 is a diagram showing an example of the data structure of an encoded three-dimensional model that is partially decoded.

[0195] In partial decoding, the decoding device first determines a three-dimensional spatial region to be partially decoded (S1001).

[0196] Next, the decoding device uses the three-dimensional spatial information (3Dspace_info) to identify an area that overlaps with the target three-dimensional spatial area from the bounding box information of multiple three-dimensional spatial areas, and obtains the space_id corresponding to the identified area (S1002).

[0197] Next, the decoding device obtains and decodes the data unit having the obtained space_id from the encoded data (S1003). As a result, the decoding device performs partial decoding, which decodes a part of the three-dimensional data. In partial decoding, the decoding device decodes only a part of the three-dimensional data without decoding the entire three-dimensional data.

[0198] For example, as shown in FIG. 26, when the three-dimensional space area to be partially decoded is the area indicated by the thick line, the space_id of the three-dimensional space to be acquired is determined to be #2 from the three-dimensional space information.

[0199] Then, as shown in FIGS. 27 to 29, the data unit associated with Space_id=#2 from the coded data of the multiple types of three-dimensional data is obtained and decoded.

[0200] The decoding device may also obtain a data unit ID instead of space_id from the three-dimensional spatial information, and obtain and partially decode a data unit having the obtained data unit ID.

[0201] In the above-described embodiment, point cloud data, mesh data, and three-dimensional model data are exemplified as three-dimensional data representing a three-dimensional object, but the present invention is not limited to these. For example, a three-dimensional object may be represented by a plurality of sets, each of which includes line-of-sight information indicating a line of sight and a two-dimensional image of the three-dimensional object as viewed from the line of sight. In other words, data including the plurality of sets may be treated as a type of three-dimensional data. Furthermore, the three-dimensional data may be data in another format, such as Gaussian splatting data.

[0202] Fig. 30 is a diagram showing an example of the configuration of a decoding device, and Fig. 31 is a flowchart showing an example of a decoding method performed by the decoding device.

[0203] The decoding device 1130 includes a circuit 1131 and a memory 1132 connected to the circuit 1131 .

[0204] The circuit 1131 performs the following operations.

[0205] The circuit 1131 acquires encoded data including first data representing a three-dimensional object, encoding format information (format) indicating one of encoding formats including second data representing the three-dimensional object, and identification information indicating a three-dimensional space including the three-dimensional object (S1021). Next, the circuit 1131 decodes the first data and the second data corresponding to the three-dimensional space based on the encoded data (S1022). Next, the circuit 1131 renders the first data to generate first presentation data for presentation (S1023). Next, the circuit 1131 renders the second data to generate second presentation data for presentation (S1024). Next, the circuit 1131 switches from the generated second presentation data to the first presentation data and presents it (S1025). Note that the first presentation data and the second presentation data are, for example, two-dimensional data or three-dimensional data generated by the rendering reconstruction unit 1034.

[0206] According to this, the first presentation data and the second presentation data are generated based on the first data and the second data corresponding to the three-dimensional space, and the first presentation data is switched to be presented, so that the two pieces of data representing the three-dimensional object can be presented without causing a spatial misalignment when they are switched, thereby making it possible to appropriately present the first presentation data and the second presentation data.

[0207] For example, the first data is point cloud data representing the three-dimensional object.

[0208] Therefore, the second presentation data is switched to the first presentation data based on the point cloud data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0209] For example, the second data is mesh data representing the three-dimensional object.

[0210] Therefore, the second presentation data based on mesh data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0211] For example, the second data is three-dimensional model data representing the three-dimensional object, and the three-dimensional model data represents a machine learning model obtained by machine learning a plurality of sets of gazes and two-dimensional images.

[0212] Therefore, the second presentation data based on the three-dimensional model data is switched to the first presentation data, so that the two pieces of data representing the three-dimensional object can be switched and presented without causing any spatial misalignment.

[0213] For example, the second data is a two-dimensional image of the three-dimensional object when viewed from a predetermined line of sight.

[0214] Therefore, the second presentation data based on a two-dimensional image is switched to the first presentation data, so that the two pieces of data representing a three-dimensional object can be switched and presented without causing any spatial misalignment.

[0215] For example, the circuit may further receive a request to switch presentation data from a user, and in the presentation, the circuit may switch from the second presentation data to the first presentation data in response to the request to switch presentation data.

[0216] Therefore, the switching can be performed at a timing designated by the user.

[0217] For example, the circuit may further receive an operation from a user to change the presentation mode, and the circuit may change the presentation mode in response to the operation, and switch from the second presentation data to the first presentation data in response to the change.

[0218] Therefore, switching can be performed at a timing according to the user's operation.

[0219] For example, the circuit may acquire the encoded data from an encoding device via a communication network during the acquiring step, and may switch from the second presentation data to the first presentation data during the presenting step in accordance with a bandwidth of the communication network.

[0220] Therefore, switching can be performed according to the bandwidth of the communication network; for example, when the bandwidth of the communication network changes from less than a specified bandwidth to greater than or equal to the specified bandwidth, the second presentation data can be switched to the first presentation data and presented.

[0221] For example, the circuit switches from the second presentation data to the first presentation data in accordance with the available capabilities of the circuit during the presentation.

[0222] Therefore, switching can be performed according to the capacity of the available circuit; for example, when the capacity of the available circuit changes from less than a predetermined capacity to greater than or equal to the predetermined capacity, the second presentation data can be switched to the first presentation data and presented.

[0223] For example, the encoded data includes synchronization information for synchronizing a coordinate system of the first data and a coordinate system of the second data, and the circuit, during presentation, presents the first presentation data and the second presentation data based on the synchronization information.

[0224] Therefore, the first presentation data and the second presentation data can be switched to the first presentation data after the coordinate systems of the first presentation data and the second presentation data are aligned, so that the two pieces of data representing the three-dimensional object can be switched and presented with less spatial misalignment.

[0225] For example, the circuit further determines whether to synchronize a coordinate system of the first data with a coordinate system of the second data. If the circuit determines that the coordinate system of the first data is synchronized with the coordinate system of the second data, the circuit presents the first presentation data and the second presentation data based on the synchronization information during the presentation.

[0226] This allows synchronization processing to be performed when necessary and can be skipped when not necessary, potentially reducing the processing load.

[0227] For example, the first data and the second data each have a common structure.

[0228] This reduces the amount of coded data, thereby reducing communication capacity.

[0229] For example, the encoded data includes spatial information for identifying the three-dimensional space in which the three-dimensional object is located. The circuit further acquires a target area indicating a portion of the three-dimensional space. The circuit identifies first duplicate data that is a portion of the first data and overlaps with the target area based on the spatial information. In the decoding process, the circuit decodes the identified first duplicate data.

[0230] Therefore, for example, by acquiring only the first duplicate data, the volume of data to be acquired can be reduced, thereby reducing communication capacity. Also, for example, it is possible to decrypt only the first duplicate data, thereby reducing processing load.

[0231] Alternatively, the circuit 1131 may operate according to the decoding method shown in the flowchart of Fig. 32. Fig. 32 is a flowchart showing another example of the decoding method performed by the decoding device.

[0232] The circuit 1131 decodes encoding information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data (S1031). The circuit 1131 decodes second data in the second encoding method indicated by the encoding information (S1032). The second data is used to generate second presentation data for presentation.

[0233] This allows the second data of the second encoding method indicated by the encoding method information obtained by decoding to be decoded, thereby obtaining second data for generating appropriate second presentation data for presentation.

[0234] Fig. 33 is a diagram showing an example of the configuration of an encoding device, and Fig. 34 is a flowchart showing an example of an encoding method performed by the encoding device.

[0235] The encoding device 1140 includes a circuit 1141 and a memory 1142 connected to the circuit 1141 .

[0236] The circuit 1141 performs the following operations.

[0237] The circuit 1141 generates encoding information representing the three-dimensional object and indicating a second encoding method different from the first encoding method of the first data (S1041). The circuit 1141 generates second data in the second encoding method indicated by the encoding information (S1042). The circuit 1141 generates a bitstream including the encoding information and the second data (S1043). The second data is used to generate second presentation data for presentation.

[0238] This allows a bitstream including encoding method information and second data to be generated, and a decoding device that acquires the bitstream can obtain the second data for generating appropriate second presentation data for presentation.

[0239] Second Embodiment A method for generating a still image of a subject (three-dimensional object) viewed from an arbitrary viewpoint in a still space using a three-dimensional data generation model, which is a learning model obtained based on learning, will be described.

[0240] Fig. 35 is a diagram for explaining the processing during learning of a 3D generation model in embodiment 2. Fig. 36 is a diagram for explaining the processing for generating a still image of a subject viewed from an arbitrary viewpoint using a 3D generation model in embodiment 2.

[0241] The information processing device can generate a still image viewed from an arbitrary viewpoint in a still space by acquiring a three-dimensional data generation model through learning. For example, there is a three-dimensional data generation model generated by a method such as NeRF (Neural Radiance Fields).

[0242] During learning, the information processing device acquires learning data including, for example, a viewpoint A image (ground truth value) acquired from an arbitrary viewpoint A and viewpoint information (such as the camera posture) of viewpoint A when the image was acquired. The viewpoint information may include viewpoint A and the line of sight direction from viewpoint A. The information processing device, for example, uses an evaluation function 1402 to input the viewpoint information from the learning data into a three-dimensional data generation model 1401, thereby optimizing parameters of a network included in the three-dimensional data generation model so as to minimize the difference between the generated image of viewpoint A output from the three-dimensional data generation model 1401 and the viewpoint A image, which is an input image corresponding to viewpoint A. The information processing device can acquire a more accurate three-dimensional data generation model by performing this learning process using multiple learning data corresponding to multiple different viewpoints. The learning process is performed for each of the multiple viewpoints. In other words, the same process as the learning process for viewpoint A is performed for each viewpoint.

[0243] During generation, when viewpoint information of viewpoint B, for example, is input to the trained three-dimensional data generation model 1403, the information processing device outputs a generated image of viewpoint B, and when viewpoint information of viewpoint Z different from viewpoint B is input, the information processing device outputs a generated image of viewpoint Z. The viewpoint information of viewpoint B may include viewpoint B and the line of sight direction from viewpoint B. The viewpoint information of viewpoint Z may include viewpoint Z and the line of sight direction from viewpoint Z.

[0244] In this way, a still image viewed from an arbitrary viewpoint in a still space can be generated by acquiring the three-dimensional data generation model 1403 through learning. However, a moving image cannot be generated in this state.

[0245] While Figure 36 illustrates an example of a 3D data generation model that generates an image from a viewpoint when viewpoint information is input, the present invention is not limited to this, and the data output from the 3D data generation model may be in any format. For example, the 3D data generation model may be a network model that outputs 3D data of a target space obtained by learning in the form of point cloud data or mesh data. This allows a user to view the target space in a stereoscopic manner using 3D data such as point cloud data or mesh data, and also allows the user to measure the dimensions of an object in the target space output as 3D data using the point cloud data or mesh data.

[0246] 37 is a diagram for explaining a moving image generation method using a three-dimensional data generation model of Example 1 in Embodiment 2. Note that in this Example, an example configuration of a device and a method for encoding or decoding three-dimensional data generation models NNt0 to NNt5 generated corresponding to times t0 to t5 will be described, but the present invention is not necessarily limited to this, and may also be applied to a device and a method for encoding or decoding a three-dimensional data generation model at each time during an arbitrary period.

[0247] This example illustrates a method for generating a moving image of a target object (subject) viewed from an arbitrary viewpoint using a three-dimensional data generation model. With this method, as shown in FIG. 37 , for example, by acquiring a three-dimensional data generation model corresponding to each time, still images of the target object viewed from an arbitrary viewpoint at each time can be generated, and a moving image can be generated by arranging the generated still images in chronological order. More specifically, when generating a moving image from times t0 to t5, multiple three-dimensional data generation models NNt0 to NNt5 corresponding to times t0 to t5 are generated by learning, and viewpoint information (such as camera posture) of viewpoint A from which a moving image is to be generated is input to the generated three-dimensional data generation models NNt0 to NNt5 corresponding to times t0 to t5. As a result, generated images of viewpoint A from times t0 to t5 are output by the three-dimensional data generation models NNt0 to NNt5, and by temporally connecting these images, a moving image of the target object viewed from viewpoint A from times t0 to t5 can be generated.

[0248] However, in this case, it is necessary to store multiple 3D data generation models corresponding to multiple times, which requires a huge storage capacity for storing the data of the multiple 3D data generation models in storage or a huge network bandwidth for transmitting the data of the multiple 3D data generation models over a network. Therefore, the data size may be reduced by data encoding the multiple 3D data generation models corresponding to multiple times using, for example, Neural Network Coding (NNC) of the Moving Picture Experts Group (MPEG) standard. This disclosure describes a method for more efficiently compressing this data.

[0249] NNC is described in Non-Patent Document 1.

[0250] FIG. 38 is a diagram illustrating a first example of the configuration of an encoding device according to a first example of the second embodiment.

[0251] The encoding device 1420 includes a three-dimensional data generation model acquisition unit 1421 , a buffer unit 1422 , and a network model encoding unit 1423 .

[0252] The 3D data generation model acquisition unit 1421 acquires learning data from times t0 to t5 and uses the acquired learning data from times t0 to t5 to generate 3D data generation models NNt0 to NNt5 for times t0 to t5 through learning. The learning data includes multiple viewpoint images obtained by photographing the target object from one or more viewpoint positions and one or more line-of-sight directions for each of the times t0 to t5, and one or more pieces of viewpoint information indicating one or more viewpoint positions and one or more line-of-sight directions corresponding to the multiple viewpoint images. The one or more pieces of viewpoint information may be the position and orientation of the camera when each of the multiple viewpoint images was captured. Note that the learning data is not limited to this and may further include information obtained from other sensors. For example, the learning data may include point cloud data, depth images, etc. acquired at each time using a LiDAR or TOF sensor. This improves the accuracy of the 3D data generation model obtained through learning.

[0253] The buffer unit 1422 stores the three-dimensional data generation model at time t generated by the three-dimensional data generation model acquisition unit 1421. The buffer unit 1422 is realized by a storage device such as a memory. The three-dimensional data generation model at time t stored in the buffer unit 1422 may be used, for example, as an initial model when the three-dimensional data generation model acquisition unit 1421 acquires (generates) a three-dimensional data generation model after time t by learning. This makes it possible to shorten the learning time while improving the accuracy of the three-dimensional data generation model after time t.

[0254] The buffer unit 1422 may store multiple 3D data generation models corresponding to multiple times. This allows, for example, a single initial model to be generated by processing such as averaging based on the multiple 3D data generation models stored in the buffer unit 1422. The 3D data generation model acquisition unit 1421 can acquire a highly accurate 3D data generation model by learning a 3D data generation model after time t using this initial model. If the 3D data generation model acquisition unit 1421 does not refer to past 3D data generation models during learning, the encoding device 1420 may not need to include the buffer unit 1422. This allows the amount of memory used by the buffer unit 1422 to be reduced.

[0255] The network model encoding unit 1423 encodes the three-dimensional data generation models NNt0 to NNt5 acquired by the three-dimensional data generation model acquisition unit 1421 and outputs a bit stream.

[0256] As a network model encoding method, for example, data size may be reduced by data encoding using the NNC of the MPEG standard. That is, the network model encoding unit 1423 encodes the three-dimensional data generation models NNt0 to NNt5 using the NNC and adds the encoding results to a bitstream. In other words, the network model encoding unit 1423 generates encoded data as the encoding result and generates a bitstream including the encoded data.

[0257] Specifically, the network model encoding unit 1423 first encodes the three-dimensional data generation model NNt0 at time t0 using an NNC and adds the encoding result to the bit stream. Next, the network model encoding unit 1423 encodes the three-dimensional data generation model NNt1 at time t1 using an NNC and adds the encoding result to the bit stream. In this way, the network model encoding unit 1423 may reduce the amount of code by encoding the three-dimensional data generation model at each time in sequence using an NNC and adding each encoding result to the bit stream.

[0258] At this time, the network model encoding unit 1423 may add time information indicating the time to which the encoded three-dimensional data generation model corresponds to the bitstream as metadata. This allows the decoding device to know the time to which the decoded three-dimensional data generation model corresponds by decoding and referring to the metadata included in the bitstream, and to appropriately generate a moving image from an arbitrary viewpoint of the target object.

[0259] The metadata is not limited to time information, and may include information related to the acquisition (generation) of learning data, or information required for the decoding device to generate a moving image.

[0260] For example, the network model encoding unit 1423 may add, as metadata, information about the frame rate of the camera when acquiring (generating) the training data, which allows the decoding device to decode the frame rate of the generated video from the bitstream and appropriately set the frame rate.

[0261] Alternatively, the network model encoding unit 1423 may add a frame number corresponding to each time to the bitstream as metadata instead of the time information, and link each frame number to the time information using another parameter. For example, the network model encoding unit 1423 may add the time information and frame rate of the first frame as metadata, and the decoding device may calculate the time information of each frame from the metadata, thereby reducing the amount of code by the amount of time information for each frame.

[0262] Furthermore, the network model encoding unit 1423 may add viewpoint information of the viewpoint image used in training to the bitstream. This allows the decoding device to generate high-quality video by, for example, preferentially selecting a viewpoint that is close to the viewpoint position corresponding to the image used in training. This is because the closer the viewpoint position or time is to the time of training, the more likely it is that the 3D data generation model will be able to generate viewpoint images with higher quality.

[0263] FIG. 39 is a diagram illustrating a first example of the configuration of a decoding device according to a first example of the second embodiment.

[0264] The decoding device 1425 includes a network model decoding unit 1426 and a rendering unit 1427 .

[0265] The network model decoding unit 1426 acquires the bit stream and decodes the three-dimensional data generation models NNt0 to NNt5 for times t0 to t5 and metadata such as time information based on the acquired bit stream.

[0266] The rendering unit 1427 generates a moving image of viewpoint A based on viewpoint information of viewpoint A specified by a user or a system, using the three-dimensional data generation models NNt0 to NNt5 decoded by the network model decoding unit 1426 and metadata such as time information. Specifically, the rendering unit 1427 inputs the viewpoint information of viewpoint A into the three-dimensional data generation model NNt0 at time t0 to generate an image IMGt0 of viewpoint A at time t0, and then inputs the viewpoint information of viewpoint A into the three-dimensional data generation model NNt1 at time t1 to generate an image IMGt1 of viewpoint A at time t1. The rendering unit 1427 applies the image generation process for each of these times to each of times t2 to t5, thereby generating images IMGt2 to IMGt5 of viewpoint A at times t2 to t5. Then, the rendering unit 1427 uses the images IMGt0 to IMGt5 and metadata such as time information to generate a moving image of the target object from time t0 to t5 as viewed from viewpoint A. The moving image may include, for example, the images IMGt0 to IMGt5 and presentation time information for calculating the presentation times of the images IMGt0 to IMGt5 based on the times t0 to t5.

[0267] Note that the viewpoint information may change depending on the time. For example, viewpoint information of viewpoint A may be input to the three-dimensional data generation models NNt0 to NNt3 from time t0 to t3, and viewpoint information of viewpoint B may be input to the three-dimensional data generation models NNt4 to NNt5 from time t4 to t5. This allows the rendering unit 1427 to generate multiple images of the target object viewed from viewpoint A from time t0 to t3, and multiple images of the target object viewed from viewpoint B from time t4 to t5. In other words, the rendering unit 1427 can generate a moving image of the target object in which the viewpoint switches from viewpoint A to viewpoint B at time t4.

[0268] The rendering unit 1427 does not necessarily have to generate a moving image, but may generate a still image of a specified viewpoint at a specified time, thereby allowing the user to switch between generating a moving image or a still image depending on the application.

[0269] Note that the rendering unit 1427 is not limited to generating moving images or still images from the three-dimensional data generation model. For example, the rendering unit 1427 may generate point cloud data or mesh data from the three-dimensional data generation model and output the generated point cloud data or mesh data as dynamic point cloud data or dynamic mesh data. This allows the user to view dynamic three-dimensional data of a dynamic target object on a head-mounted display (HMD) or the like, and also allows the user to measure the amount of movement of the target object using the dynamic three-dimensional data.

[0270] FIG. 40 is a diagram illustrating a second example of the configuration of the encoding device according to the first example of the second embodiment.

[0271] The encoding device 1430 includes a three-dimensional data generation model acquisition unit 1431 , a buffer unit 1432 , a difference calculation unit 1433 , and a network model encoding unit 1434 .

[0272] The three-dimensional data generation model acquisition unit 1431 is the same as the three-dimensional data generation model acquisition unit 1421 of the encoding device 1420 .

[0273] The buffer unit 1432 is similar to the buffer unit 1422 of the encoding device 1420, but differs from the buffer unit 1422 in that it inputs a three-dimensional data generation model stored in a memory or the like to the difference calculation unit 1433 as a reference three-dimensional data generation model.

[0274] The difference calculation unit 1433 calculates difference information indicating the difference between the three-dimensional data generation models NNt0 to NNt5 for times t0 to t5 generated by the three-dimensional data generation model acquisition unit 1431 and each of the three-dimensional data generation models (hereinafter referred to as reference three-dimensional data generation models) generated by the three-dimensional data generation model acquisition unit 1431 before each time. Here, the difference information may include, for example, differences in weight parameters at the nodes of each network model. For example, the difference calculation unit 1433 acquires the three-dimensional data generation model NNt5 for time t5 from the three-dimensional data generation model acquisition unit 1431, and acquires the three-dimensional data generation model NNt4 for time t4 from the buffer unit 1432 as the reference three-dimensional data generation model.

[0275] The difference calculation unit 1433 may use the three-dimensional data generation model NNt5 and the three-dimensional data generation model NNt4 to calculate, for example, the difference (amount of change) between the weight parameters of the nodes of the network model in the three-dimensional data generation model NNt5 and the weight parameters of the nodes of the network model in the three-dimensional data generation model NNt4, and input difference information indicating this difference to the network model encoding unit 1434. The difference information is then encoded by the network model encoding unit 1434. In other words, the encoding device 1430 may reduce the amount of data by predicting information related to the network model in the three-dimensional data generation model NNt5 from the three-dimensional data generation model NNt4 and encoding the difference from the predicted value. Such predictive encoding reduces the value of the difference to be encoded when there is little change in the three-dimensional data generation model over time, such as when the target object is almost stationary, thereby improving encoding efficiency. For example, the encoding device 1430 may set RNNt0 = 0 and RNNtn = NNt(n-1) (n is an integer value from 1 to 5) and reduce the amount of bits by predictive encoding using the three-dimensional data generation model from the previous time as a reference three-dimensional data generation model.

[0276] In the second example, the encoding device 1430 predictively encodes information related to the network model in the 3D data generation model NNt5 from information related to the network model in the 3D data generation model NNt4, but this is not necessarily limited to this. For example, the encoding device 1430 may select a reference 3D data generation model to be used for prediction from one or more 3D data generation models stored in the buffer unit 1432 and perform predictive encoding using the selected 3D data generation model. In this case, the encoding device 1430 may add information indicating the selected 3D data generation model (reference 3D data generation model information) to the bitstream to communicate the selected 3D data generation model to the decoding device. This allows the encoding device 1430 to select the optimal reference 3D data generation model in terms of encoding efficiency, thereby improving encoding efficiency. Furthermore, the decoding device can appropriately decode a bitstream with improved encoding efficiency by decoding the reference 3D data generation model information.

[0277] When the encoding device 1430 performs predictive encoding by referring to two or more 3D data generation models stored in the buffer unit 1432, the encoding device 1430 may add information indicating the two or more reference 3D data generation models to the bitstream. This allows the encoding device 1430 to improve the encoding efficiency of predictive encoding using two or more reference 3D data generation models. Furthermore, the decoding device can appropriately decode the bitstream with improved encoding efficiency.

[0278] Note that in cases where a reference 3D data generation model is not stored in the buffer unit 1432, for example, when encoding the first 3D data generation model in data order (first frame), the encoding device 1430 may encode the 3D data generation model to be processed without prediction, without calculating the difference from a predicted value (hereinafter referred to as intra prediction), or may calculate and encode the difference from a predicted value set to 0. Furthermore, when setting a certain time t as a random access point, the encoding device 1430 may encode the 3D data generation model corresponding to time t using intra prediction, or may calculate and encode the difference from a predicted value set to 0. This allows the decoding device to start decoding the 3D data generation model from the first 3D data generation model in data order (first frame) or the random access point, thereby improving functionality during playback.

[0279] Furthermore, a set of multiple 3D data generation models (multiple frames) (hereinafter referred to as a GOF (Group of Frame)) may be defined, and the first frame of the GOF may be coded using intra-prediction. This allows the decoding device to randomly access the first frame of the GOF, and by decoding the first frame of the GOF, functionality such as fast-forward playback can be enhanced.

[0280] The encoding device 1430 may also add permission information to the bitstream, which indicates whether or not inter-GOF predictive reference is permitted. For example, if the bitstream includes permission information indicating that inter-GOF predictive reference is prohibited, the decoding device can determine that multiple GOFs can be decoded in parallel. Furthermore, for example, allowing inter-GOF predictive reference can improve encoding efficiency.

[0281] The network model encoding unit 1434 is similar to the network model encoding unit 1423 of the encoding device 1420, but differs in that it encodes the difference information d0 to d5 of the three-dimensional data generation models NNt0 to NNt5 input from the difference calculation unit 1433 and outputs a bit stream.

[0282] Note that although the encoding device 1430 has been described with the difference calculation unit 1433 and the network model encoding unit 1434 separated, this is not necessarily limited to this, and for example, the difference calculation unit 1433 may be included in the network model encoding unit 1434. In other words, the network model encoding unit 1434 may perform the processing of the difference calculation unit 1433.

[0283] The encoding device 1430 may add predictive coding information to the bitstream, indicating whether the 3D data generation model was encoded using intra prediction or predictive coding using a reference 3D data generation model (hereinafter referred to as inter prediction). This allows the decoding device to appropriately determine whether to use intra prediction or inter prediction to decode the 3D data generation model by decoding the predictive coding information.

[0284] FIG. 41 is a diagram illustrating a second example of the configuration of the decoding device according to the first embodiment of the present invention.

[0285] The decoding device 1435 includes a network model decoding unit 1436 , an adding unit 1437 , a buffer unit 1438 , and a rendering unit 1439 .

[0286] The network model decoding unit 1436 acquires the bit stream and, based on the acquired bit stream, decodes the difference information d0 to d5 of the three-dimensional data generation models NNt0 to NNt5 for times t0 to t5, as well as metadata such as time information.

[0287] The adder 1437 adds, for the corresponding times, the difference information d0 to d5 of the three-dimensional data generation model decoded by the network model decoding unit 1436 and corresponding to times t0 to t5, and the reference three-dimensional data generation models RNNt0 to RNNt5 acquired from the buffer unit 1438, to calculate three-dimensional data generation models NNt0 to NNt5. In this way, the decoding device 1435 may set RNNt0 = 0 and RNNtn = NNt(n-1) (n is a value from 1 to 5), and perform predictive decoding using the three-dimensional data generation model of the previous time as the reference three-dimensional data generation model.

[0288] Note that, in the decoding device 1435 in the second example, the addition unit 1437 and the network model decoding unit 1436 are described separately, but this is not necessarily limited to this, and for example, the addition unit 1437 may be included in the network model decoding unit 1436. In other words, the network model decoding unit 1436 may perform the processing of the addition unit 1437.

[0289] Note that in cases where a reference 3D data generation model is not stored in the buffer unit 1438, for example, when decoding the first 3D data generation model (first frame) in data order, the decoding device 1435 may perform decoding without prediction (hereinafter referred to as intra prediction) without the addition unit 1437 adding the difference information and the reference 3D data generation model, or may perform decoding by adding a prediction value set to 0 and the difference information. Furthermore, when setting a certain time t as a random access point, the decoding device 1435 may decode the 3D data generation model corresponding to time t using intra prediction, or may decode by adding a prediction value set to 0 and the difference information. Furthermore, when predictive coding information indicating that the 3D data generation model to be decoded has been coded using intra prediction is included in the bitstream, the 3D data generation model may be decoded using intra prediction, or may be decoded by adding a prediction value set to 0 and the difference information. This allows the decoding device 1435 to start decoding the three-dimensional data generation model from the first three-dimensional data generation model (first frame) in data order, a random access point, or a three-dimensional data generation model coded using intra-prediction, thereby improving functionality during playback.

[0290] In the second example, the decoding device 1435 predictively decodes information related to the network model in the 3D data generation model NNt5 from information related to the network model in the 3D data generation model NNt4, but this is not necessarily limited to this. For example, the decoding device 1435 may select a reference 3D data generation model to be used for prediction from one or more 3D data generation models stored in the buffer unit 1438 and perform predictive decoding using the selected 3D data generation model. In this case, the decoding device 1435 may decode information indicating the selected 3D data generation model (reference 3D data generation model information) from the bitstream. This allows the decoding device 1435 to appropriately decode a bitstream with improved coding efficiency by decoding the reference 3D data generation model information from the bitstream generated by the encoding device 1430, selecting the reference 3D data generation model that is optimal in terms of coding efficiency.

[0291] Note that when performing predictive decoding with reference to two or more 3D data generation models stored in the buffer unit 1438, the decoding device 1435 may decode information indicating the two or more reference 3D data generation models from the bitstream. This allows the decoding device 1435 to appropriately decode a bitstream in which the coding efficiency of predictive coding has been improved using two or more reference 3D data generation models.

[0292] The rendering unit 1439 is similar to the rendering unit 1427 of the decoding device 1425. The rendering unit 1439 does not necessarily need to generate a moving image, but may generate a still image of specified viewpoint information at a specified time.

[0293] [Example 2] Fig. 42 is a diagram for explaining a moving image generation method using an extended three-dimensional data generation model of Example 2 in Embodiment 2. Note that in this example, an example configuration of a device and a method for encoding or decoding extended three-dimensional data generation models NNt0-2 and NNt3-5 generated corresponding to periods t0 to t2 and periods t3 to t5, respectively, from time t0 to t5, will be described, but the present invention is not necessarily limited to this, and may also be applied to a device and a method for encoding or decoding an extended three-dimensional data generation model for any period.

[0294] This embodiment describes a method for generating moving images of a target object (subject) viewed from an arbitrary viewpoint using a three-dimensional data generation model. With this method, for example, as shown in FIG. 42 , by acquiring a three-dimensional data generation model (hereinafter referred to as an extended three-dimensional data generation model) capable of generating images from an arbitrary viewpoint within a certain time range (period), still images of the target object viewed from an arbitrary viewpoint at an arbitrary time within each period can be generated, and moving images can be generated by arranging the generated still images in chronological order. The extended three-dimensional data generation model is a three-dimensional data generation model generated, for example, by a method such as NeRF, similar to the three-dimensional data generation model of the first embodiment.

[0295] More specifically, when generating a moving image from time t0 to t5, an extended three-dimensional data generation model NNt0-2 capable of representing the period from time t0 to t2 and an extended three-dimensional data generation model NNt3-5 capable of representing the period from time t3 to t5 are generated by learning, and viewpoint information (camera posture, etc.) of viewpoint A from which a moving image is to be generated is input to the generated extended three-dimensional data generation model NNt0-2 and extended three-dimensional data generation model NNt3-5. As a result, generated images from viewpoint A from time t0 to t5 are output by the extended three-dimensional data generation model NNt0-2 and extended three-dimensional data generation model NNt3-5, and by connecting these images in time, a moving image from time t0 to t5 in which the target object is viewed from viewpoint A can be generated.

[0296] However, in this case, it is necessary to store an augmented 3D data generation model corresponding to each period (each time zone), which requires a huge storage capacity for storing the data of the augmented 3D data generation model in storage, or a huge network bandwidth for transmitting the data of multiple 3D data generation models over a network. Therefore, the data size may be reduced by data encoding the augmented 3D data generation model corresponding to each period using, for example, Neural Network Coding (NNC) of the Moving Picture Experts Group (MPEG) standard. This disclosure describes a method for more efficiently compressing this data.

[0297] With the above configuration, the information processing device may generate any viewpoint image at any time within the period from time t0 to time t5. For example, when acquiring the extended 3D data generation model NNt0-2, the information processing device generates the extended 3D data generation model NNt0-2 by machine learning based on multiple viewpoint images captured at times t0, t1, and t2 as learning data and camera postures corresponding to the multiple viewpoints. When generating a moving image of viewpoint A, the information processing device may generate not only viewpoint image A at times t0, t1, and t2, but also images of any viewpoint at times t0.5 and t1.5 between times t0, t1, and t2, for example. Time t0.5 is a time between times t0 and t1, and time t1.5 is a time between times t1 and t2.

[0298] This allows the information processing device to generate images from any viewpoint corresponding not only to the time corresponding to the image during learning, but also to a time shifted from the time corresponding to the image during learning, thereby enabling the generation of moving images from viewpoint A at a high frame rate.

[0299] The information processing device may perform learning using not only the learning data at times t0, t1, and t2 but also, for example, learning data at time t3 as learning data for the extended 3D data generation model NNt0-2. This allows for highly accurate generation of viewpoint images from any viewpoint after time t2, for example, an image from an arbitrary viewpoint at time t2.5.

[0300] Furthermore, the information processing device may use not only the learning data corresponding to times t3, t4, and t5 as the learning data for the extended 3D data generation model NNt3-5, but also, for example, learning data corresponding to times t2 and t6 for learning. This allows the information processing device to generate, with high accuracy, images from any viewpoint before time t3 or images from any viewpoint after time t5. Note that, for example, in the above example, when generating a viewpoint image at time 2.5, which is between times t2 and t3 when the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5 switch, the information processing device may generate a viewpoint image at time t2.5 for each of the extended 3D data generation model NNt0-2 and the extended 3D data generation model NNt3-5, and then generate an average image of the two generated viewpoint images at time t2.5 as the viewpoint image at time t2.5. This allows for the generation of a highly accurate viewpoint image at time t2.5.

[0301] In this way, the information processing device can generate an image of the target object viewed from the specified viewpoint at the specified time by specifying the time within the period corresponding to the extended three-dimensional data generation model and viewpoint information for the extended three-dimensional data generation model.

[0302] FIG. 43 is a diagram illustrating a first example of a configuration of an encoding device according to a second example of the second embodiment.

[0303] The encoding device 1450 includes an extended three-dimensional data generation model acquisition unit 1451 , a buffer unit 1452 , and a network model encoding unit 1453 .

[0304] The extended 3D data generation model acquisition unit 1451 acquires learning data for each of the periods t0 to t2 and t3 to t5, and generates an extended 3D data generation model NNt0-2 for the period t0 to t2 and an extended 3D data generation model NNt3-5 for the period t3 to t5 by learning using the acquired learning data for each period. The learning data includes multiple viewpoint images obtained by photographing the target object from one or more viewpoint positions and one or more line-of-sight directions for each of the times t0 to t5, and one or more pieces of viewpoint information indicating one or more viewpoint positions and one or more line-of-sight directions corresponding to the multiple viewpoint images. The one or more pieces of viewpoint information may be the position and orientation of the camera when each of the multiple viewpoint images was captured. Note that the learning data is not limited to this and may further include information obtained from other sensors. For example, the learning data may include point cloud data, depth images, etc. acquired at each time using a LiDAR or TOF sensor. This improves the accuracy of the extended 3D data generation model obtained by learning.

[0305] The buffer unit 1452 stores the extended 3D data generation model for the period tm-n from time tm (m is an integer) to time tn (n is an integer greater than m) generated by the extended 3D data generation model acquisition unit 1451. The buffer unit 1452 is realized by a storage device such as a memory. The extended 3D data generation model for the period tm-n stored in the buffer unit 1452 may be used, for example, as an initial model when the extended 3D data generation model acquisition unit 1451 acquires (generates) an extended 3D data generation model for a period after the period tm-n by learning. This makes it possible to shorten the learning time while improving the accuracy of the extended 3D data generation model for a period after the period tm-n.

[0306] The buffer unit 1452 may store multiple extended 3D data generation models corresponding to multiple time periods. As a result, for example, a single initial model may be generated based on the multiple extended 3D data generation models stored in the buffer unit 1452, for example, by processing such as averaging. The extended 3D data generation model acquisition unit 1451 can acquire a highly accurate extended 3D data generation model by learning an extended 3D data generation model for a period after the period tm-n using this initial model. If the extended 3D data generation model acquisition unit 1451 does not refer to extended 3D data generation models for past periods during learning, the encoding device 1450 may not need to include the buffer unit 1452. This reduces the amount of memory used by the buffer unit 1452.

[0307] The network model encoding unit 1453 encodes the extended three-dimensional data generation models NNt0-2 and NNt3-5 acquired by the extended three-dimensional data generation model acquisition unit 1451, and outputs a bit stream.

[0308] As a network model encoding method, for example, the data size may be reduced by data encoding using the NNC of the MPEG standard. That is, the network model encoding unit 1453 encodes the extended three-dimensional data generation models NNt0-2 and NNt3-5 using NNC and adds the encoding results to a bitstream. In other words, the network model encoding unit 1453 generates encoded data as the encoding result and generates a bitstream including the encoded data.

[0309] Specifically, the network model encoding unit 1453 first encodes the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 using an NNC and adds the encoding result to the bit stream. Next, the network model encoding unit 1453 encodes the extended three-dimensional data generation model NNt3-5 for the period t3 to t5 using an NNC and adds the encoding result to the bit stream. In this way, the network model encoding unit 1453 may reduce the amount of code by encoding the extended three-dimensional data generation models for each period in sequence using an NNC and adding each encoding result to the bit stream.

[0310] At this time, the network model encoding unit 1453 may add time information indicating to which period the encoded extended 3D data generation model corresponds to as metadata to the bitstream. This allows the decoding device to know to which period the decoded extended 3D data generation model corresponds by decoding and referring to the metadata included in the bitstream, and to appropriately generate moving images from any viewpoint of the target object.

[0311] The network model encoding unit 1453 may generate information indicating the period for which the extended 3D data generation model can generate viewpoint images as time information, and add the generated time information to the bitstream as metadata. This allows the decoding device to know the period for which the extended 3D data generation model can generate viewpoint images by decoding this metadata, and thereby generate a moving image appropriately.

[0312] The metadata is not limited to time information, and may include information related to the acquisition (generation) of learning data, or information required for the decoding device to generate a moving image.

[0313] For example, the network model encoding unit 1453 may add, as metadata, information about the frame rate of the camera when acquiring (generating) the training data, which allows the decoding device to decode the frame rate of the generated video from the bitstream and appropriately set the frame rate.

[0314] Alternatively, the network model encoding unit 1453 may add frame numbers corresponding to each period to the bit stream as metadata instead of time information, and link each frame number to the time information using another parameter. For example, the network model encoding unit 1453 may add the time information and frame rate of the first frame as metadata, and the decoding device may calculate the time information of each frame from the metadata, thereby reducing the amount of code by the amount of time information for each frame.

[0315] Furthermore, the network model encoding unit 1453 may add viewpoint information of the viewpoint image used in training or time information indicating the time when the viewpoint image was captured to the bitstream. This allows the decoding device to generate high-quality video by, for example, preferentially selecting a viewpoint close to the viewpoint position corresponding to the image used in training or a time close to the time corresponding to the image used in training. This is because the closer the viewpoint position or time is to the time during training, the more likely it is that the extended 3D data generation model will be able to generate viewpoint images with higher quality.

[0316] FIG. 44 is a diagram illustrating a first example of a configuration of a decoding device according to a second example of the second embodiment.

[0317] The decoding device 1455 includes a network model decoding unit 1456 and a rendering unit 1457 .

[0318] The network model decoding unit 1456 acquires the bit stream and, based on the acquired bit stream, decodes the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 and the extended three-dimensional data generation model NNt3-5 for the period t3 to t5, as well as metadata such as time information corresponding to these extended three-dimensional data generation models NNt0-2 and NNt3-5.

[0319] The rendering unit 1457 generates a moving image of viewpoint A based on viewpoint information of viewpoint A specified by a user or a system, using the extended three-dimensional data generation models NNt0-2 and NNt3-5 decoded by the network model decoding unit 1456 and metadata such as time information. Specifically, the rendering unit 1457 inputs the viewpoint information of viewpoint A and the times within the period t0 to t2 into the extended three-dimensional data generation model NNt0-2 for the period t0 to t2, and generates an image IMGt0 of viewpoint A at time t0, an image IMGt1 of viewpoint A at time t1, and an image IMGt2 of viewpoint A at time t2. The rendering unit 1457 applies the image generation process for the period t0 to t2 to the extended three-dimensional data generation model NNt3-5 for the period t3 to t5, thereby generating images IMGt3 to IMGt5 of viewpoint A at times t3 to t5. Then, the rendering unit 1457 uses the images IMGt0 to IMGt5 and metadata such as time information to generate a moving image of the target object from time t0 to t5 as viewed from viewpoint A. The moving image may include, for example, the images IMGt0 to IMGt5 and presentation time information for calculating the presentation times of the images IMGt0 to IMGt5 based on the times t0 to t5.

[0320] Note that the viewpoint information may be changed according to time. For example, viewpoint information of viewpoint A may be input to the extended three-dimensional data generation model NNt0-2 for the period t0 to t2, and viewpoint information of viewpoint B may be input to the extended three-dimensional data generation model NNt3-5 for the period t3 to t5. This allows the rendering unit 1457 to generate multiple images of the target object viewed from viewpoint A from time t0 to t2, and multiple images of the target object viewed from viewpoint B from time t3 to t5. In other words, the rendering unit 1457 can generate a moving image of the target object in which the viewpoint switches from viewpoint A to viewpoint B at time t3.

[0321] The rendering unit 1457 does not necessarily need to generate a moving image, but may generate a still image of a specified viewpoint at a specified time, thereby allowing the user to switch between generating a moving image or a still image depending on the application.

[0322] Note that the rendering unit 1457 is not limited to generating moving images or still images from the extended 3D data generation model. For example, the rendering unit 1457 may generate point cloud data or mesh data from the extended 3D data generation model for a period that the extended 3D data generation model can represent, and output the generated point cloud data or mesh data as dynamic point cloud data or dynamic mesh data. This allows the user to view dynamic 3D data of a dynamic target object on a head-mounted display (HMD) or the like, and also allows the user to measure the amount of movement of the target object using the dynamic 3D data.

[0323] FIG. 45 is a diagram illustrating a second example of the configuration of the encoding device according to the second embodiment.

[0324] The encoding device 1460 includes an extended three-dimensional data generation model acquisition unit 1461 , a buffer unit 1462 , a difference calculation unit 1463 , and a network model encoding unit 1464 .

[0325] The extended three-dimensional data generation model acquisition unit 1461 is the same as the extended three-dimensional data generation model acquisition unit 1451 of the encoding device 1450 .

[0326] The buffer unit 1462 is similar to the buffer unit 1452 of the encoding device 1450, but differs from the buffer unit 1452 in that it inputs an extended 3D data generation model stored in a memory or the like to the difference calculation unit 1463 as a reference extended 3D data generation model.

[0327] The difference calculation unit 1463 calculates difference information indicating the difference between the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 and the extended three-dimensional data generation model NNt3-5 for the period t3 to t5 generated by the extended three-dimensional data generation model acquisition unit 1461, and the extended three-dimensional data generation model (hereinafter referred to as the reference extended three-dimensional data generation model) generated by the extended three-dimensional data generation model acquisition unit 1461 before each period. Here, the difference information may include, for example, differences in weight parameters at the nodes of each network model. The difference calculation unit 1463 acquires, for example, the extended three-dimensional data generation model NNt3-5 for the period t3 to t5 from the extended three-dimensional data generation model acquisition unit 1461, and acquires the extended three-dimensional data generation model NNt0-2 for the period t0 to t2 from the buffer unit 1462 as the reference extended three-dimensional data generation model.

[0328] The difference calculation unit 1463 may use the extended three-dimensional data generation model NNt3-5 and the extended three-dimensional data generation model NNt0-2 to calculate, for example, the difference (amount of change) between the weight parameters of the nodes of the network model in the extended three-dimensional data generation model NNt3-5 and the weight parameters of the nodes of the network model in the extended three-dimensional data generation model NNt0-2, and input difference information indicating this difference to the network model encoding unit 1464. The difference information is then encoded by the network model encoding unit 1464. In other words, the encoding device 1460 may reduce the amount of data by predicting information related to the network model in the extended three-dimensional data generation model NNt3-5 from the extended three-dimensional data generation model NNt0-2 and encoding the difference from the predicted value. Such predictive encoding reduces the value of the difference to be encoded when there is little change in the extended three-dimensional data generation model over time, such as when the target object is almost stationary, thereby improving encoding efficiency. For example, the encoding device 1460 may set RNNt0-2=0 and RNNt3-5=NNt0-2, and reduce the amount of bits by predictive encoding using the extended 3D data generation model of the previous time period as the reference extended 3D data generation model.

[0329] Note that, in the second example, the encoding device 1460 predictively encodes information related to the network model in the extended 3D data generation model NNt3-5 from information related to the network model in the extended 3D data generation model NNt0-2, but this is not necessarily limited to this. For example, the encoding device 1460 may select a reference extended 3D data generation model to be used for prediction from one or more extended 3D data generation models stored in the buffer unit 1462, and perform predictive encoding using the selected extended 3D data generation model. In this case, the encoding device 1460 may add information indicating the selected extended 3D data generation model (reference extended 3D data generation model information) to the bitstream in order to communicate the selected extended 3D data generation model to the decoding device. This allows the encoding device 1460 to select the optimal reference extended 3D data generation model in terms of encoding efficiency, thereby improving encoding efficiency. Furthermore, the decoding device can appropriately decode a bitstream with improved encoding efficiency by decoding the reference extended 3D data generation model information.

[0330] When the encoding device 1460 performs predictive encoding by referring to two or more extended 3D data generation models stored in the buffer unit 1462, the encoding device 1460 may add information indicating the two or more reference extended 3D data generation models to the bitstream. This allows the encoding device 1460 to improve the encoding efficiency of predictive encoding using two or more reference extended 3D data generation models. Furthermore, the decoding device can appropriately decode the bitstream with improved encoding efficiency.

[0331] In addition, in cases where a reference extended 3D data generation model is not stored in the buffer unit 1462, for example, when encoding the first extended 3D data generation model in data order (first frame), the encoding device 1460 may encode the extended 3D data generation model to be processed without prediction, without calculating the difference from a predicted value (hereinafter referred to as intra prediction), or may calculate and encode the difference from a predicted value set to 0. Furthermore, when setting a certain period tm-n as a random access point, the encoding device 1460 may encode the extended 3D data generation model corresponding to the period tm-n using intra prediction, or may calculate and encode the difference from a predicted value set to 0. This allows the decoding device to start decoding the extended 3D data generation model from the first extended 3D data generation model in data order (first frame) or the random access point, thereby improving functionality during playback.

[0332] Furthermore, a set of multiple extended 3D data generation models (multiple frames) (hereinafter referred to as a GOF (Group of Frame)) may be defined, and the first frame of the GOF may be coded using intra prediction. This allows a decoding device to randomly access the first frame of the GOF, and by decoding the first frame of the GOF, functionality such as fast-forward playback can be enhanced.

[0333] The encoding device 1460 may also add permission information to the bitstream indicating whether or not inter-GOF predictive reference is permitted. For example, if the bitstream includes permission information indicating that inter-GOF predictive reference is prohibited, the decoding device can determine that multiple GOFs can be decoded in parallel. Furthermore, for example, allowing inter-GOF predictive reference can improve encoding efficiency.

[0334] The network model encoding unit 1464 is similar to the network model encoding unit 1453 of the encoding device 1450, but differs in that it encodes the difference information d0-2 and d3-5 of the extended three-dimensional data generation models NNt0-2 and NNt3-5 input from the difference calculation unit 1463 and outputs a bit stream.

[0335] Note that although the encoding device 1460 has been described with the difference calculation unit 1463 and the network model encoding unit 1464 separated, this is not necessarily limited to this, and for example, the difference calculation unit 1463 may be included in the network model encoding unit 1464. In other words, the network model encoding unit 1464 may perform the processing of the difference calculation unit 1463.

[0336] The encoding device 1460 may add predictive coding information to the bitstream, indicating whether the extended 3D data generation model was encoded using intra prediction or predictive coding using a reference extended 3D data generation model (hereinafter referred to as inter prediction). This allows the decoding device to appropriately determine whether to use intra prediction or inter prediction to decode the extended 3D data generation model by decoding the predictive coding information.

[0337] FIG. 46 is a diagram illustrating a second example of the configuration of a decoding device according to Example 2 in the second embodiment.

[0338] The decoding device 1465 includes a network model decoding unit 1466 , an adding unit 1467 , a buffer unit 1468 , and a rendering unit 1469 .

[0339] The network model decoding unit 1466 acquires the bit stream and, based on the acquired bit stream, decodes the differential information d0-2, d3-5 of the extended three-dimensional data generation model NNt0-2 and the extended three-dimensional data generation model NNt3-5 for the period t0 to t2, as well as metadata such as time information.

[0340] The adder 1467 adds, for the corresponding periods, difference information d0-2, d3-5 of the extended three-dimensional data generation models NNt0-2, NNt3-5 decoded by the network model decoding unit 1466 and corresponding to the periods t0 to t2, t3 to t5, to the reference extended three-dimensional data generation models RNNt0-2, RNNt3-5 acquired from the buffer unit 1468, to calculate the extended three-dimensional data generation models NNt0-2, NNt3-5. In this way, the decoding device 1465 may set RNNt0-2 = 0, RNNt3-5 = NNt0-2, and perform predictive decoding on the extended three-dimensional data generation model of the previous time period as the reference extended three-dimensional data generation model.

[0341] Note that, although the decoding device 1465 in the second example is described with the addition unit 1467 and the network model decoding unit 1466 separated, this is not necessarily limited thereto, and for example, the addition unit 1467 may be included in the network model decoding unit 1466. In other words, the network model decoding unit 1466 may perform the processing of the addition unit 1467.

[0342] In addition, in cases where a reference extended 3D data generation model is not stored in the buffer unit 1468, for example, when decoding the first extended 3D data generation model (first frame) in data order, the decoding device 1465 may perform decoding without prediction (hereinafter referred to as intra prediction) without the addition unit 1467 adding the difference information and the reference extended 3D data generation model, or may perform decoding by adding a prediction value set to 0 and the difference information. Furthermore, when a certain period tm-n is set as a random access point, the decoding device 1465 may decode the extended 3D data generation model corresponding to the period tm-n using intra prediction, or may decode by adding a prediction value set to 0 and the difference information. Furthermore, when predictive encoding information indicating that the extended 3D data generation model to be decoded has been encoded using intra prediction is included in the bitstream, the extended 3D data generation model may be decoded using intra prediction, or may be decoded by adding a prediction value set to 0 and the difference information. This allows the decoding device 1465 to start decoding the extended 3D data generation model from the first extended 3D data generation model (first frame) in data order, a random access point, or an extended 3D data generation model coded using intra prediction, thereby improving functionality during playback.

[0343] In the second example, the decoding device 1465 predictively decodes information related to the network model in the extended 3D data generation model NNt3-5 from information related to the network model in the extended 3D data generation model NNt0-2, but this is not necessarily limited to this. The decoding device 1465 may, for example, select a reference extended 3D data generation model to be used for prediction from one or more extended 3D data generation models stored in the buffer unit 1468, and perform predictive decoding using the selected extended 3D data generation model. In this case, the decoding device 1465 may decode information indicating the selected extended 3D data generation model (reference extended 3D data generation model information) from the bitstream. As a result, the decoding device 1465 can appropriately decode a bitstream with improved coding efficiency by decoding the reference extended 3D data generation model information from the bitstream generated by the encoding device 1460, in which the optimal reference extended 3D data generation model in terms of coding efficiency is selected.

[0344] Note that the decoding device 1465 may decode information indicating the two or more reference extended 3D data generation models from the bitstream when performing predictive decoding with reference to two or more extended 3D data generation models stored in the buffer unit 1468. This allows the decoding device 1465 to appropriately decode a bitstream in which the coding efficiency of predictive coding has been improved using two or more reference extended 3D data generation models.

[0345] The rendering unit 1469 is similar to the rendering unit 1427 of the decoding device 1425. The rendering unit 1469 does not necessarily need to generate a moving image, but may generate a still image of specified viewpoint information at a specified time.

[0346] [Modification] The encoding device 1460 may include information regarding the number of images that the extended 3D data generation model can generate (i.e., the upper limit of the number of images) in the metadata that is added to the bit stream of the extended 3D data generation model for the period tm-n. This allows the decoding device 1465 to know the number of images that the decoded extended 3D data generation model can generate, and, for example, to appropriately set the frame rate of the video to be generated or to calculate the number of delays before the video is displayed.

[0347] The encoding device 1460 may add, as time information to be added to the bitstream, information indicating up to what time unit (i.e., the smallest time unit) the extended 3D data generation model can generate viewpoint images. The encoding device 1460 may add, as time information to the bitstream, for example, whether viewpoint images can be generated up to a time unit of 1 msec, or whether viewpoint images can be generated up to a time unit of 1 μmsec. This allows the decoding device 1465 to know up to what time unit viewpoint information is to be generated, and can generate high-frame-rate video or 3D data accordingly.

[0348] Furthermore, the encoding device 1460 may add information about when the extended 3D data generation model was trained to the metadata to be added to the bitstream of the extended 3D data generation model for the period tm-n. For example, the encoding device 1460 adds time information or viewpoint information of the image used for training to the bitstream as metadata, and the decoding device 1465 can decode the metadata to know the time or viewpoint information at which the extended 3D data generation model can generate viewpoint images with high image quality, thereby creating high-quality moving images.

[0349] The width of the period of viewpoint images for which the extended 3D data generation model can be generated may be dynamically switched as shown in Fig. 47. Specifically, the width of the period may be switched depending on the subject. Fig. 47 is a diagram for explaining a moving image generation method using an extended 3D data generation model according to a modification of the second embodiment.

[0350] For example, in a scene with many still objects among the subjects (a scene in which the number of still objects among the multiple subjects is equal to or greater than a first number, or a scene in which the volume (area) occupied by the still objects among the multiple subjects is equal to or greater than a first amount), the encoding device 1460 widens the width of the period of the training data used during training (i.e., lengthens the period), thereby generating an extended 3D data generation model over a long period during which high-quality viewpoint images can be generated. For example, in a scene with many dynamic objects among the subjects (a scene in which the number of dynamic objects among the multiple subjects is equal to or greater than a first number, or a scene in which the volume (area) occupied by the dynamic objects among the multiple subjects is equal to or greater than a first amount), the encoding device 1460 narrows the width of the training data used during training, thereby generating an extended 3D data generation model over a short period during which high-quality viewpoint images can be generated, even for dynamic objects.

[0351] The encoding device 1460 may use training data for a certain period tm-n (e.g., a group of frames (GOF) representing a collection of frames within the period tm-n among the training images) to generate an extended 3D data generation model NNtm-n for that period tm-n. In this case, the encoding device 1460 buffers the training image frames for the period tm-n to generate an extended 3D data generation model, and compresses and transmits the data, resulting in a transmission delay equivalent to the GOF size. The encoding device 1460 may add information regarding the transmission delay, such as the number of GOF frames or the number of delayed frames, to the bitstream. This allows the decoding device 1465 to obtain delay information by decoding the bitstream and appropriately reproduce video or 3D data taking the delay into account.

[0352] In the above embodiment, an example has been described in which a still image from an arbitrary viewpoint at a certain time or period is generated using a 3D data generation model or an extended 3D data generation model, but this is not necessarily limited to this. For example, the 3D data generation model or the extended 3D data generation model may generate (output) 3D data such as point cloud data or mesh data at a certain time within a certain period, as shown in FIG. 48 . This allows a user to measure the dimensions of a target object or view higher-resolution 3D data. FIG. 48 is a diagram for explaining a moving image generation method using a 3D data generation model according to a modification of the second embodiment.

[0353] Furthermore, the encoding devices 1420 and 1460 may include, in the metadata of the bitstream, information indicating a recommended output format according to the use case, among output formats such as images, point cloud data, mesh data, etc. This allows the user to select a recommended output format according to the use case.

[0354] The encoding devices 1420 and 1460 may add one or more pieces of viewpoint information to the metadata added to the bitstream of the 3D data generation model or the extended 3D data generation model. For example, the encoding devices 1420 and 1460 may include, in the metadata, recommended viewpoint information for viewing the target object, or the user's viewpoint information when acquiring the training data. This allows the decoding devices 1425 and 1465 to generate moving images or 3D data using viewpoint information selected from the one or more pieces of viewpoint information added to the bitstream in accordance with the user's intentions, etc.

[0355] Note that a default viewpoint may be predetermined from one or more pieces of viewpoint information, and the decoding devices 1425 and 1465 may generate moving images or three-dimensional data using the predetermined default viewpoint unless specified by the user. This allows the decoding devices 1425 and 1465 to automatically generate moving images or three-dimensional data without user specification.

[0356] The following is an example of a use case that utilizes this embodiment.

[0357] First, the encoding devices 1420 and 1460 acquire data of a dynamic object to be transmitted to a remote location using a camera or sensor, and use the data of the dynamic object as learning data to generate a three-dimensional data generation model or an extended three-dimensional data generation model of the dynamic object.

[0358] Next, the encoding devices 1420 and 1460 encode the three-dimensional data generation model or the extended three-dimensional data generation model using the encoding method described in this embodiment, and transmit a bitstream containing the encoding results to a remote location.

[0359] The decoding devices 1425 and 1465 then decode the bitstream received at the remote location, generate moving images or 3D data from any viewpoint using the decoded 3D data generation model of the dynamic object or the extended 3D data generation model, and utilize the generated 3D data for viewing, measurement, etc. In this way, the present embodiment may be applied to general use cases in which information about a certain space is shared at remote locations.

[0360] If there are one or more objects in a space that you want to transmit to a remote location, the three-dimensional data generation modeling, encoding / transmission, decoding, and rendering processes described in this embodiment may be applied separately to each object. For example, a foreground dynamic object and a background static object in a space may be generated as three-dimensional data generation models and encoded / transmitted separately. This allows the application of three-dimensional data generation modeling or encoding methods that are optimal for each object, thereby improving encoding efficiency.

[0361] Furthermore, the present invention is not limited to this, and the three-dimensional data generation and modeling, encoding and transmission, decoding, and rendering processes shown in this embodiment may be applied separately to one or more multiple objects as a single object. This makes it possible to transmit one or more multiple objects to a remote location while reducing the amount of processing.

[0362] Fig. 49 is a diagram showing an example of a configuration of a coding device in Embodiment 2. Fig. 50 is a flowchart showing an example of a coding method performed by the coding device in Embodiment 2.

[0363] The encoding device 1470 includes a circuit 1471 and a memory 1472. The encoding device 1470 is a device that realizes the encoding devices 1420 and 1460.

[0364] The circuit 1471 performs the following operations.

[0365] The circuit 1471 acquires a first three-dimensional data generation model (e.g., three-dimensional data generation model NNt0) corresponding to a first time (e.g., time t0) and a second three-dimensional data generation model (e.g., three-dimensional data generation model NNt1) corresponding to a second time (e.g., time t1) (S1401). The circuit 1471 generates a bitstream by encoding the acquired first three-dimensional data generation model and second three-dimensional data generation model (S1402). When viewpoint information including a viewpoint and a line of sight is input, each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as seen from the viewpoint and the line of sight.

[0366] This makes it possible to generate a bitstream that includes a first three-dimensional data generation model that can obtain a two-dimensional image corresponding to a first time according to arbitrary viewpoint information, and a second three-dimensional data generation model that can obtain a two-dimensional image corresponding to a second time, thereby generating a bitstream that compresses data that can obtain moving images from arbitrary viewpoints. This makes it possible to reduce the memory capacity required to store data that can obtain moving images from arbitrary viewpoints, or the network bandwidth required to transmit that data.

[0367] For example, each of the first three-dimensional data generation model and the second three-dimensional data generation model is a learning model using a neural network.

[0368] For example, the bitstream includes first time information indicating the first time and second time information indicating the second time.

[0369] For example, the bitstream includes a first frame number corresponding to the first time and a second frame number corresponding to the second time.

[0370] For example, the bitstream includes frame rate information regarding frame rates of a plurality of training images used to generate the first 3D data generation model and the second 3D data generation model, the plurality of training images being two-dimensional images captured at a plurality of different times.

[0371] For example, the bitstream includes viewpoint information including viewpoints and line-of-sight directions of a plurality of learning images used to generate the first 3D data generation model and the second 3D data generation model.

[0372] For example, the plurality of learning images are two-dimensional images obtained by photographing the subject from different viewpoints and line-of-sight directions, and the viewpoint information includes the different viewpoints and line-of-sight directions.

[0373] For example, in encoding the second 3D data generation model, the circuit 1471 calculates difference information indicating a difference between the first 3D data generation model and the second 3D data generation model. The bit stream includes the difference information.

[0374] For example, the difference includes a difference between weight parameters associated with nodes included in the first three-dimensional data generation model and the second three-dimensional data generation model.

[0375] For example, the bitstream includes reference information indicating that the difference information has been calculated with reference to the first 3D data generation model.

[0376] For example, the first time corresponds to a random access point. The first 3D data generation model is coded using intra prediction, or is coded using inter prediction with a prediction value of 0.

[0377] For example, the first three-dimensional data generation model and the second three-dimensional data generation model are included in one set of a plurality of sets, and the first three-dimensional data generation model is the first in data order among the plurality of three-dimensional data generation models included in the one set.

[0378] For example, the bitstream includes permission information indicating whether or not, in encoding each of the plurality of three-dimensional data generation models, the three-dimensional data generation model is permitted to reference three-dimensional data generation models included in other sets.

[0379] For example, the first three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt0-2) corresponds to a first period (e.g., period t0 to t2) including the first time (e.g., time t0), and the second three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt3-5) corresponds to a second period (e.g., period t3 to t5) including the second time (e.g., time t3).

[0380] For example, the plurality of first learning images used to generate the first three-dimensional data generation model are two-dimensional images obtained by photographing at a plurality of different times during the first period.

[0381] For example, when a time included in the first period is input, the first three-dimensional data generation model outputs a two-dimensional image of the subject at the input time.

[0382] For example, the bitstream includes number information indicating the upper limit of the number of images that can be generated by the first 3D data generation model.

[0383] For example, the bitstream includes first information about the first training images, the first information including a plurality of viewpoints, a plurality of line-of-sight directions, and a plurality of different timings corresponding to the first training images.

[0384] For example, the first period or the second period is dynamically determined depending on the subject.

[0385] For example, the circuit 1471 stores the generated first three-dimensional data generation model in the memory 1472. The circuit 1471 generates the second three-dimensional data generation model based on the first three-dimensional data generation model stored in the memory 1472.

[0386] For example, the circuit 1471 stores the generated first and second three-dimensional data generation models in the memory 1472. The circuit 1471 generates an initial model based on the first and second three-dimensional data generation models stored in the memory 1472. The circuit 1471 generates a third three-dimensional data generation model (e.g., three-dimensional data generation model NNt2) corresponding to a third time (e.g., time t2) based on the initial model.

[0387] Fig. 51 is a diagram showing an example of the configuration of a decoding device in Embodiment 2. Fig. 52 is a flowchart showing an example of a decoding method performed by the decoding device in Embodiment 2.

[0388] The decoding device 1480 includes a circuit 1481 and a memory 1482. The decoding device 1480 is a device that realizes the decoding devices 1425 and 1465.

[0389] The circuit 1481 performs the following operations.

[0390] The circuit 1481 acquires a bitstream (S1411). The circuit 1481 decodes from the bitstream a first 3D data generation model (e.g., 3D data generation model NNt0) corresponding to a first time (e.g., time t0) and a second 3D data generation model (e.g., 3D data generation model NNt1) corresponding to a second time (e.g., time t1) (S1412). When viewpoint information including a viewpoint and a line of sight is input, each of the first 3D data generation model and the second 3D data generation model outputs a 2D image of a subject as seen from the viewpoint and the line of sight.

[0391] According to this, based on a bitstream that compresses data that allows for video images from any viewpoint, it is possible to decode a first three-dimensional data generation model that allows for a two-dimensional image corresponding to a first time point in accordance with any viewpoint information, and a second three-dimensional data generation model that allows for a two-dimensional image corresponding to a second time point. Therefore, it is possible to appropriately decode a bitstream that can reduce the memory capacity for storing data that allows for video images from any viewpoint to be obtained, or the network bandwidth for transmitting that data.

[0392] For example, each of the first three-dimensional data generation model and the second three-dimensional data generation model is a learning model using a neural network.

[0393] For example, the bitstream includes first time information indicating the first time and second time information indicating the second time.

[0394] For example, the bitstream includes a first frame number corresponding to the first time and a second frame number corresponding to the second time.

[0395] For example, the bitstream includes frame rate information regarding frame rates of a plurality of training images used to generate the first 3D data generation model and the second 3D data generation model, the plurality of training images being two-dimensional images captured at a plurality of different times.

[0396] For example, the bitstream includes viewpoint information including viewpoints and line-of-sight directions of a plurality of learning images used to generate the first 3D data generation model and the second 3D data generation model.

[0397] For example, the plurality of learning images are two-dimensional images obtained by photographing the subject from different viewpoints and line-of-sight directions, and the viewpoint information includes the different viewpoints and line-of-sight directions.

[0398] For example, the bitstream includes difference information indicating a difference between the first 3D data generation model and the second 3D data generation model.

[0399] For example, the difference includes a difference between weight parameters associated with nodes included in the first three-dimensional data generation model and the second three-dimensional data generation model.

[0400] For example, the bitstream includes reference information indicating that the difference information has been calculated with reference to the first 3D data generation model.

[0401] For example, the first time corresponds to a random access point. The first 3D data generation model is coded using intra prediction, or is coded using inter prediction with a prediction value of 0.

[0402] For example, the first three-dimensional data generation model and the second three-dimensional data generation model are included in one set of a plurality of sets, and the first three-dimensional data generation model is the first in data order among the plurality of three-dimensional data generation models included in the one set.

[0403] For example, the bitstream includes permission information indicating whether or not, in encoding each of the plurality of three-dimensional data generation models, the three-dimensional data generation model is permitted to reference three-dimensional data generation models included in other sets.

[0404] For example, the first three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt0-2) corresponds to a first period (e.g., period t0 to t2) including the first time (e.g., time t0), and the second three-dimensional data generation model (e.g., extended three-dimensional data generation model NNt3-5) corresponds to a second period (e.g., period t3 to t5) including the second time (e.g., time t3).

[0405] For example, the plurality of first learning images used to generate the first three-dimensional data generation model are two-dimensional images obtained by photographing at a plurality of different times during the first period.

[0406] For example, when a time included in the first period is input, the first three-dimensional data generation model outputs a two-dimensional image of the subject at the input time.

[0407] For example, the bitstream includes number information indicating the upper limit of the number of images that can be generated by the first 3D data generation model.

[0408] For example, the bitstream includes first information about the first training images, the first information including a plurality of viewpoints, a plurality of line-of-sight directions, and a plurality of different timings corresponding to the first training images.

[0409] For example, the first period or the second period is dynamically determined depending on the subject.

[0410] For example, the circuit 1471 stores the generated first three-dimensional data generation model in the memory 1472. The circuit 1471 generates the second three-dimensional data generation model based on the first three-dimensional data generation model stored in the memory 1472.

[0411] For example, the circuit 1471 stores the generated first and second three-dimensional data generation models in the memory 1472. The circuit 1471 generates an initial model based on the first and second three-dimensional data generation models stored in the memory 1472. The circuit 1471 generates a third three-dimensional data generation model (e.g., three-dimensional data generation model NNt2) corresponding to a third time (e.g., time t2) based on the initial model.

[0412] (Other) In one embodiment, a method for generating a moving image viewed from a predetermined viewpoint is disclosed. The generation of the moving image is realized, for example, by a device including a memory and a circuit connected to the memory. This example device stores a 3D data generation model (Neural Network) generated by learning in memory, and the circuit acquires the 3D data generation model (Neural Network) stored in the memory and generates a moving image based on the 3D data generation model. Note that the 3D data generation model or the extended 3D data generation model does not necessarily have to be stored in memory. For example, the encoding devices 1420 and 1460 may acquire specification information specifying a URL on a network and acquire the 3D data generation model based on the specification information.

[0413] FIG. 53 is a diagram illustrating an example of the configuration of an encoding device.

[0414] The encoding device 1490 includes a processor 1491 and a memory 1492 .

[0415] The processor 1491 is a circuit that performs information processing and is a circuit that can access the memory 1492. For example, the processor 1491 is a dedicated or general-purpose electronic circuit that encodes a three-dimensional data generation model. The processor 1491 may be a processor such as a CPU. The processor 1491 may also be a collection of multiple electronic circuits. For example, the processor 1491 may also fulfill the roles of multiple components of the aforementioned encoding device, excluding the component for storing information.

[0416] The memory 1492 is a dedicated or general-purpose memory that stores information used by the processor 1491 to encode the three-dimensional data generation model. The memory 1492 may be an electronic circuit and may be connected to the processor 1491. The memory 1492 may also be included in the processor 1491. The memory 1492 may also be a collection of multiple electronic circuits. The memory 1492 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as storage, a recording medium, or the like. The memory 1492 may also be a non-volatile memory or a volatile memory.

[0417] For example, the three-dimensional data generation model to be encoded may be stored in the memory 1492, or a stream corresponding to the encoded three-dimensional data generation model may be stored in the memory 1492. Furthermore, the memory 1492 may store a program that allows the processor 1491 to encode the three-dimensional data generation model.

[0418] It should be noted that not all of the components of the encoding device described above may be implemented, and not all of the processes described above may be performed, in the encoding device 1490. Some of the components may be included in other devices, and some of the processes described above may be performed by other devices.

[0419] FIG. 54 is a diagram illustrating an example of the configuration of a decoding device.

[0420] The decoding device 1495 includes a processor 1496 and a memory 1497 .

[0421] The processor 1496 is a circuit that processes information and is a circuit that can access the memory 1497. For example, the processor 1496 is a dedicated or general-purpose electronic circuit that decodes a stream. The processor 1496 may be a processor such as a CPU. The processor 1496 may also be a collection of multiple electronic circuits. For example, the processor 1496 may fulfill the roles of multiple components of the above-mentioned decoding device, excluding the component for storing information.

[0422] The memory 1497 is a dedicated or general-purpose memory that stores information for the processor 1496 to decode the stream. The memory 1497 may be an electronic circuit and may be connected to the processor 1496. The memory 1497 may also be included in the processor 1496. The memory 1497 may also be a collection of multiple electronic circuits. The memory 1497 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 1497 may also be a non-volatile memory or a volatile memory.

[0423] For example, a three-dimensional data generation model or a stream may be stored in memory 1497. Furthermore, memory 1497 may also store a program for processor 1496 to decode the stream.

[0424] Note that not all of the components of the above-described decoding device may be implemented, and not all of the above-described processes may be performed, in the decoding device 1495. Some of the components may be included in another device, and some of the above-described processes may be executed by another device.

[0425] The present disclosure is applicable to encoding devices and the like that can output three-dimensional data with different resolutions.

[0426] 1001 Three-dimensional data encoding system 1002 Three-dimensional data decoding system 1003 Sensor terminal 1004 External connection unit 1011 Three-dimensional data generation system 1012 Presentation unit 1013 Encoding unit 1014 Multiplexing unit 1015 Input / output unit 1016 Control unit 1017 Sensor information acquisition unit 1018 Three-dimensional data generation unit 1021 Sensor information acquisition unit 1022 Input / output unit 1023 Demultiplexing unit 1024 Decoding unit 1025 Presentation unit 1026 User interface 1027 Control unit 1031 Three-dimensional model learning unit 1032 Three-dimensional model encoding unit 1033 Three-dimensional model decoding unit 1034 Rendering reconstruction unit 1041 Data division unit 1042 Encoding unit 1051 Decoding unit 1052 Data combination unit 1070 Server 1071 Data generation unit 1072 Point cloud generation unit 1073 Mesh generation unit 1074 Model generation unit 1075 Synchronization unit 1076 Point cloud encoding unit 1077 Mesh encoding unit 1078 Model encoding unit 1079 Multiplexing unit 1080 Data extraction unit 1090 Terminal 1091 Control unit 1092 Decoding unit 1093 Presentation unit 1101 Point cloud sensor 1102 Camera 1110 Data generation unit 1111 Point cloud generation unit 1112 Mesh generation unit 1113 Model generation unit 1130 Decoding device 1131 Circuit 1132 Memory 1140 Encoding device 1141 Circuit 1142 Memory 1401 Three-dimensional data generation model 1402 Evaluation function 1403 Three-dimensional data generation model 1420 Encoding device 1421 3D data generation model acquisition unit 1422 Buffer unit 1423 Network model encoding unit 1425 Decoding device 1426 Network model decoding unit 1427 Rendering unit 1430 Encoding device 1431 3D data generation model acquisition unit 1432 Buffer unit 1433 Difference calculation unit 1434 Network model encoding unit 1435 Decoding device 1436 Network model decoding unit 1437 Addition unit 1438 Buffer unit 1439 Rendering unit 1450 Encoding device 1451 Extended 3D data generation model acquisition unit 1452 Buffer unit 1453 Network model encoding unit 1455 Decoding device 1456 Network model decoding unit1457 Rendering unit 1460 Encoding device 1461 Extended 3D data generation model acquisition unit 1462 Buffer unit 1463 Difference calculation unit 1464 Network model encoding unit 1465 Decoding device 1466 Network model decoding unit 1467 Addition unit 1468 Buffer unit 1469 Rendering unit 1470 Encoding device 1471 Circuit 1472 Memory 1480 Decoding device 1481 Circuit 1482 Memory 1490 Encoding device 1491 Processor 1492 Memory 1495 Decoding device 1496 Processor 1497 Memory

Claims

1. An encoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, acquires a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time, and generates a bit stream by encoding the acquired first three-dimensional data generation model and the second three-dimensional data generation model, and when viewpoint information including a viewpoint and a line of sight is input, each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as viewed from the viewpoint and the line of sight.

2. The encoding device according to claim 1, wherein each of the first three-dimensional data generation model and the second three-dimensional data generation model is a learning model using a neural network.

3. The encoding device according to claim 1, wherein the bit stream includes first time information indicating the first time and second time information indicating the second time.

4. The encoding device according to claim 3, wherein the bit stream includes a first frame number corresponding to the first time and a second frame number corresponding to the second time.

5. The encoding device according to any one of claims 1 to 4, wherein the bit stream includes frame rate information regarding frame rates of a plurality of learning images used in generating the first three-dimensional data generation model and the second three-dimensional data generation model, and the plurality of learning images are two-dimensional images obtained by photographing at a plurality of different times.

6. An encoding device described in any one of claims 1 to 4, wherein the bit stream includes viewpoint information including the viewpoint and line of sight direction of multiple learning images used to generate the first three-dimensional data generation model and the second three-dimensional data generation model.

7. The encoding device according to claim 6, wherein the plurality of learning images are two-dimensional images obtained by photographing the subject from mutually different viewpoints and viewing directions, and the viewpoint information includes the mutually different viewpoints and viewing directions.

8. An encoding device as claimed in any one of claims 1 to 4, wherein the circuit calculates differential information indicating a difference between the first three-dimensional data generation model and the second three-dimensional data generation model in encoding the second three-dimensional data generation model, and the bit stream includes the differential information.

9. The encoding device according to claim 8, wherein the difference includes a difference between weighting parameters associated with nodes included in the first three-dimensional data generation model and the second three-dimensional data generation model.

10. The encoding device according to claim 8, wherein the bit stream includes reference information indicating that the differential information has been calculated with reference to the first three-dimensional data generation model.

11. An encoding device described in any one of claims 1 to 4, wherein the first time corresponds to a random access point, and the first 3D data generation model is encoded using intra prediction or is encoded using inter prediction with a prediction value of 0.

12. The encoding device described in claim 11, wherein the first three-dimensional data generation model and the second three-dimensional data generation model are included in one set of a plurality of sets, and the first three-dimensional data generation model is the first in data order among the plurality of three-dimensional data generation models included in the one set.

13. The encoding device of claim 12, wherein the bit stream includes permission information indicating whether or not, in encoding each of the plurality of three-dimensional data generation models, the three-dimensional data generation model is permitted to reference three-dimensional data generation models included in other sets.

14. An encoding device according to any one of claims 1 to 4, wherein the first three-dimensional data generation model corresponds to a first period including the first time, and the second three-dimensional data generation model corresponds to a second period including the second time.

15. The encoding device described in claim 14, wherein the multiple first learning images used to generate the first three-dimensional data generation model are two-dimensional images obtained by photographing at multiple different times during the first period.

16. The encoding device according to claim 14, wherein the first three-dimensional data generation model, when a time included in the first period is input, outputs a two-dimensional image of the subject at the input time.

17. The encoding device according to claim 14, wherein the bit stream includes numerical information indicating an upper limit of the number of images that the first 3D data generation model can generate.

18. The encoding device of claim 15, wherein the bit stream includes first information regarding the plurality of first training images, and the first information includes a plurality of viewpoints and a plurality of line-of-sight directions, and a plurality of different timings, corresponding to the plurality of first training images.

19. The encoding device according to claim 14, wherein the first period or the second period is dynamically determined depending on the subject.

20. An encoding device as described in any one of claims 1 to 4, wherein the circuit stores the generated first three-dimensional data generation model in the memory, and generates the second three-dimensional data generation model based on the first three-dimensional data generation model stored in the memory.

21. The encoding device described in claims 1 to 4, wherein the circuit: stores the generated first three-dimensional data generation model and the second three-dimensional data generation model in the memory; generates an initial model based on the first three-dimensional data generation model and the second three-dimensional data generation model stored in the memory; and generates a third three-dimensional data generation model corresponding to a third time based on the initial model.

22. A decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in an operation, obtains a bit stream; and decodes from the bit stream a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time, wherein each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as viewed from the viewpoint and the line of sight when viewpoint information including the viewpoint and line of sight is input.

23. The decoding device according to claim 22, wherein each of the first three-dimensional data generation model and the second three-dimensional data generation model is a learning model using a neural network.

24. The decoding device according to claim 22, wherein the bit stream includes first time information indicating the first time and second time information indicating the second time.

25. The decoding device according to claim 24, wherein the bit stream includes a first frame number corresponding to the first time and a second frame number corresponding to the second time.

26. A decoding device as described in any one of claims 22 to 25, wherein the bit stream includes frame rate information regarding frame rates of a plurality of learning images used to generate the first three-dimensional data generation model and the second three-dimensional data generation model, and the plurality of learning images are two-dimensional images obtained by photographing at a plurality of different times.

27. A decoding device as described in any one of claims 22 to 25, wherein the bit stream includes viewpoint information including the viewpoint and line of sight direction of multiple learning images used to generate the first 3D data generation model and the second 3D data generation model.

28. The decoding device according to claim 27, wherein the plurality of learning images are two-dimensional images obtained by photographing the subject from mutually different viewpoints and viewing directions, and the viewpoint information includes the mutually different viewpoints and viewing directions.

29. A decoding device according to any one of claims 22 to 25, wherein the bit stream includes difference information indicating a difference between the first 3D data generation model and the second 3D data generation model.

30. The decoding device according to claim 29, wherein the difference includes a difference between weight parameters associated with nodes included in the first three-dimensional data generation model and the second three-dimensional data generation model.

31. The decoding device according to claim 29, wherein the bit stream includes reference information indicating that the differential information has been calculated with reference to the first 3D data generation model.

32. A decoding device according to any one of claims 22 to 25, wherein the first time corresponds to a random access point, and the first 3D data generation model is decoded by intra prediction or by inter prediction with a prediction value of 0.

33. The decoding device described in claim 32, wherein the first three-dimensional data generation model and the second three-dimensional data generation model are included in one set of a plurality of sets, and the first three-dimensional data generation model is the first in data order among the plurality of three-dimensional data generation models included in the one set.

34. The decoding device according to claim 33, wherein the bit stream includes permission information indicating whether or not, in decoding each of the plurality of three-dimensional data generation models, the three-dimensional data generation model is permitted to refer to three-dimensional data generation models included in other sets.

35. A decoding device as claimed in any one of claims 22 to 25, wherein the first three-dimensional data generation model corresponds to a first period including the first time, and the second three-dimensional data generation model corresponds to a second period including the second time.

36. The decoding device according to claim 35, wherein the plurality of first learning images used in generating the first three-dimensional data generation model are two-dimensional images obtained by photographing at a plurality of different times during the first period.

37. The decoding device according to claim 35, wherein the first three-dimensional data generation model, when a time included in the first period is input, outputs a two-dimensional image of the subject at the input time.

38. The decoding device according to claim 35, wherein the bit stream includes numerical information indicating an upper limit on the number of images that the first 3D data generation model can generate.

39. The decoding device of claim 36, wherein the bit stream includes first information regarding the plurality of first training images, and the first information includes a plurality of viewpoints and a plurality of line-of-sight directions, and a plurality of different timings, corresponding to the plurality of first training images.

40. The decoding device according to claim 35, wherein the first period or the second period is dynamically determined depending on the subject.

41. A decoding device as described in any one of claims 22 to 25, wherein the circuit stores the generated first three-dimensional data generation model in the memory, and generates the second three-dimensional data generation model based on the first three-dimensional data generation model stored in the memory.

42. A decoding device as described in claims 22 to 25, wherein the circuit: stores the generated first three-dimensional data generation model and the second three-dimensional data generation model in the memory; generates an initial model based on the first three-dimensional data generation model and the second three-dimensional data generation model stored in the memory; and generates a third three-dimensional data generation model corresponding to a third time based on the initial model.

43. An encoding method comprising: acquiring a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time; generating a bit stream by encoding the acquired first three-dimensional data generation model and the second three-dimensional data generation model; and when viewpoint information including a viewpoint and a line of sight is input, each of the first three-dimensional data generation model and the second three-dimensional data generation model outputs a two-dimensional image of a subject as viewed from the viewpoint and the line of sight.

44. A decoding method comprising: acquiring a bit stream; decoding from the bit stream a first three-dimensional data generation model corresponding to a first time and a second three-dimensional data generation model corresponding to a second time; and each of the first three-dimensional data generation model and the second three-dimensional data generation model, when viewpoint information including a viewpoint and a line of sight is input, outputting a two-dimensional image of a subject as viewed from the viewpoint and the line of sight.

Images