Information processing device, information processing method, and program

Abstract

This method reduces the data size of a 3D Gaussian model while obtaining high-precision virtual viewpoint images. [Solution] The information processing device 200 according to the present disclosure acquires a plurality of captured images obtained by photographing a target space from a plurality of different directions, and camera parameters corresponding to each of the plurality of captured images. Based on the captured images and the camera parameters, it sets initial values ​​for the parameters of a 3D Gaussian model that includes a plurality of 3D Gaussian distributions, each including a 3D Gaussian distribution of density values ​​corresponding to at least one negative opacity and a 3D Gaussian distribution of density values ​​corresponding to at least one positive opacity. By learning based on the captured images and the camera parameters, it optimizes the parameters of the 3D Gaussian model to obtain a trained 3D Gaussian model that reproduces the target space.

Claims

1. Image acquisition means for acquiring multiple captured images obtained by photographing the target space from multiple different directions, and camera parameters corresponding to each of the multiple captured images, A setting means for setting initial values ​​for parameters of a 3D Gaussian model that includes a plurality of 3D Gaussian distributions, each including a 3D Gaussian distribution of density values ​​corresponding to at least one negative opacity and a 3D Gaussian distribution of density values ​​corresponding to at least one positive opacity, based on the captured image and the camera parameters. A learning means that optimizes the parameters of the 3D Gaussian model by learning based on the captured image and the camera parameters, and acquires a trained 3D Gaussian model that reproduces the target space. An information processing device characterized by having the following features.

2. The setting means places a 3D Gaussian distribution of density values ​​corresponding to the initial positive opacity at a position corresponding to the three-dimensional surface of the object obtained by estimating the three-dimensional shape of the object present in the target space based on the captured image, and places a 3D Gaussian distribution of density values ​​corresponding to the initial negative opacity at a position a predetermined distance away from the 3D Gaussian distribution of density values ​​corresponding to the initial positive opacity. The information processing apparatus according to claim 1, characterized by the following:

3. The setting means arranges the initial plurality of 3D Gaussian distributions regularly or randomly in a space that encompasses the three-dimensional shape of the object obtained by estimation based on the captured image. The information processing apparatus according to claim 1, characterized by the following:

4. The setting means replaces two or more of the initial plurality of 3D Gaussian distributions that are close to each other and have similar initial color information set based on the captured image with a single initial 3D Gaussian distribution that encompasses them. The information processing apparatus according to claim 1, characterized by the following:

5. The learning means updates the parameters of the 3D Gaussian model and the total number of 3D Gaussian distributions included in the 3D Gaussian model when there are pixels in the learning process where the cumulative opacity or color value calculated in the rendering process using the 3D Gaussian model is negative. The information processing apparatus according to claim 1, characterized by the following:

6. The learning means updates the 3D Gaussian model in the learning process such that the total number of the multiple 3D Gaussian distributions included in the 3D Gaussian model decreases. The information processing apparatus according to claim 1, characterized by the following:

7. The learning means, in the learning process, removes from the 3D Gaussian model a 3D Gaussian distribution from which the absolute value of opacity is small among the plurality of 3D Gaussian distributions included in the 3D Gaussian model. The information processing apparatus according to claim 1, characterized by the following:

8. The learning means, in the learning process, removes pairs of 3D Gaussian distributions from among the plurality of 3D Gaussian distributions included in the 3D Gaussian model that are similar in position, variance-covariance, and color, and whose sum of densities is close to zero. The information processing apparatus according to claim 1, characterized by the following:

9. The learning means, in the learning process, places, among the plurality of 3D Gaussian distributions included in the 3D Gaussian model, 3D Gaussian distributions whose variance-covariance values ​​are large relative to the size of the object represented by the 3D Gaussian distribution, and 3D Gaussian distributions whose color or density values ​​are different from those of the 3D Gaussian distribution. The information processing apparatus according to claim 1, characterized by the following:

10. The learning means, in the learning process, places a 3D Gaussian distribution of density values ​​corresponding to negative opacity, which is associated with the 3D Gaussian distribution, within the region of the shape of the 3D Gaussian distribution of density values ​​corresponding to positive opacity. The information processing apparatus according to claim 1, characterized by the following:

11. The parameters of the trained 3D Gaussian model include position, covariance matrix, density information, and color information for each of the multiple 3D Gaussian distributions. The information processing apparatus according to claim 1, characterized by the following:

12. The system further includes a model output means that outputs the trained 3D Gaussian model acquired by the aforementioned learning means. The information processing apparatus according to claim 1, characterized by the following:

13. The model output means outputs information about the conversion function, associated with the data of the trained 3D Gaussian model, if a conversion function other than a predetermined function is used in the conversion process between density and opacity during training in the training means. The information processing apparatus according to claim 12, characterized by the above.

14. A data acquisition means that acquires a trained 3D Gaussian model containing multiple 3D Gaussian distributions containing information on position, variance-covariance, density, and color, and virtual viewpoint information containing information on a virtual viewpoint. A drawing means that determines the color value of each pixel in a virtual viewpoint image corresponding to the virtual viewpoint by accumulating the opacity and color corresponding to the density value of the 3D Gaussian distribution and the distance to the center of the 3D Gaussian distribution, in order of increasing distance from the virtual viewpoint, for each 3D Gaussian distribution projected onto the image plane corresponding to the virtual viewpoint from among a plurality of 3D Gaussian distributions included in the trained 3D Gaussian model and the virtual viewpoint information, the drawing means that generates the virtual viewpoint image by assigning a negative opacity to 3D Gaussian distributions whose density value is smaller than a predetermined threshold, An information processing device characterized by having the following features.

15. The drawing means replaces the color values ​​of pixels in the virtual viewpoint image whose cumulative opacity value is negative, or whose cumulative color value is negative, with predetermined color values. The information processing apparatus according to claim 14, characterized by the above.

16. An image acquisition step involves obtaining multiple captured images obtained by photographing the target space from multiple different directions, and camera parameters corresponding to each of the multiple captured images. A setting step of setting initial values ​​for the parameters of a 3D Gaussian model that includes a plurality of 3D Gaussian distributions, each including a 3D Gaussian distribution of density values ​​corresponding to at least one negative opacity and a 3D Gaussian distribution of density values ​​corresponding to at least one positive opacity, based on the captured image and the camera parameters. A learning process to optimize the parameters of the 3D Gaussian model by learning based on the captured image and the camera parameters, thereby obtaining a trained 3D Gaussian model that reproduces the target space; An information processing method characterized by including

17. A data acquisition process that acquires a trained 3D Gaussian model containing multiple 3D Gaussian distributions containing information on position, variance-covariance, density, and color, and virtual viewpoint information containing information on a virtual viewpoint. A drawing step in which, based on the trained 3D Gaussian model and the virtual viewpoint information, the opacity and color of each pixel in a virtual viewpoint image corresponding to the virtual viewpoint are determined by accumulating the opacity and color corresponding to the density value of the 3D Gaussian distribution and the distance to the center of the 3D Gaussian distribution, in order of increasing distance from the virtual viewpoint, for each 3D Gaussian distribution projected onto the image plane corresponding to the virtual viewpoint, the drawing step of which generates the virtual viewpoint image by assigning negative opacity to 3D Gaussian distributions whose density value is smaller than a predetermined threshold, An information processing method characterized by including

18. A program for causing a computer to function as an information processing device according to any one of claims 1 to 15.

Images