Program, information processing device, and adaptation density control method

Abstract

An extraction means 1151 extracts, from among 3D Gaussians subjected to parameter training by a learning means 114, only 3D Gaussians for which the maximum size of a projected Gaussian projected on each screen is greater than or equal to a threshold value. A selection means 1152 selects some 3D Gaussians, from the 3D Gaussians extracted by the extraction means 1151, with a probability proportional to the opacity of the extracted 3D Gaussians. A division means 1153 divides the 3D Gaussians selected by the selection means 1152 into a plurality of pieces so that the total volume of the 3D Gaussians is maintained. An arrangement means 1154 arranges each of the 3D Gaussians divided into the plurality of pieces by the division means 1153, on the basis of the positions of the original 3D Gaussians before the division.

Description

Program, Information Processing Apparatus, and Adaptive Density Control Method

[0001] The present invention relates to a technique for newly generating an image of an object as seen from an arbitrary viewpoint using images of the object taken from a plurality of viewpoints.

[0002] Photogrammetry is a technique for estimating the three-dimensional shape of an object from images (also referred to as captured images) of the object taken from a plurality of viewpoints. For example, Patent Document 1 discloses an apparatus and method for generating a three-dimensional (3D) model of a subject from images of a plurality of cameras as a photogrammetry technique.

[0003] Furthermore, techniques for generating an image (also referred to as a virtual image) of an object (also referred to as a virtual object) in a virtual space as seen from an arbitrary viewpoint (also referred to as a virtual viewpoint) based on captured images of the object taken from a plurality of viewpoints have been studied. As this technique, NeRF (Neural Radiance Fields) and NeuS (Neural Implicit Surface) are known, but there are problems such as a large load on machine learning and a long time required.

[0004] On the other hand, 3DGS (3D Gaussian Splatting) has attracted attention as a technique for realizing high-speed and high-quality rendering. 3DGS is a technique for reproducing a 3D scene by arranging ellipsoids called 3D Gaussians in a 3D (three-dimensional) virtual space and optimizing the parameters of those 3D Gaussians to match the original captured images.

[0005] Unlike the implicit expressions used by NeRF and the like, 3DGS explicitly represents a virtual object in a virtual space with an ellipsoid called a 3D Gaussian, so that rendering processing can be performed in a short time. In addition, since 3DGS uses differentiable rasterization, for example, by using a GPU (Graphics Processing Unit), the processing can be speeded up compared to NeRF and the like that use volume rendering. For example, Non-Patent Documents 1 to 3 disclose various methods related to this 3DGS.

[0006] Japanese Patent Application Laid-Open No. 2021-71749

[0007] https: / / arxiv.org / abs / 2308.04079 https: / / arxiv.org / abs / 2404.06109 https: / / ubc-vision.github.io / 3dgs-mcmc /

[0008] To recreate a 3D scene, a 3D Gaussian of a size corresponding to the details of the 3D scene is required. Obtaining such a 3D Gaussian has limitations if it is only optimized by comparing the size of the initial 3D Gaussian with the original captured image. Therefore, in addition to this optimization process, 3DGS duplicates or splits the 3D Gaussian placed during initialization according to predetermined conditions. The control that adjusts the density by splitting and duplicating the 3D Gaussian is generally called "adaptive density control."

[0009] However, in conventional technologies such as Non-Patent Document 3, adaptive density control of 3D Gaussians involves replicating 3D Gaussians smaller than a certain size in a virtual space and splitting only those 3D Gaussians larger than a certain size. Therefore, conventional technologies may not be able to reproduce the details of objects that were represented in the original captured image.

[0010] One of the objectives of the present invention is to realize adaptive density control of 3D Gaussian, which is more suitable than the prior art for reproducing the details of objects represented in captured images.

[0011] In one embodiment, the present invention provides a program for a computer to function as: an acquisition means for acquiring a plurality of images of an object and camera information indicating the position, orientation, and shooting range of the camera that captured each of the plurality of images; a calculation means for calculating a projected Gaussian by projecting each of a plurality of 3D Gaussians arranged in a virtual space onto a screen corresponding to each of the cameras in the virtual space; an extraction means for extracting from the plurality of 3D Gaussians the one whose largest projected Gaussian projected onto each of the screens is greater than or equal to a threshold; and a splitting means for splitting a 3D Gaussian selected from the extracted 3D Gaussians with a probability according to opacity into a plurality such that the total volume is maintained.

[0012] A diagram showing an example of the configuration of the information processing device 1 according to an embodiment of the present invention. A diagram showing an example of the image DB 121. A diagram showing an example of the camera information DB 122. A diagram showing the captured image and camera position. A diagram showing the correspondence between feature points in each of the captured images. A diagram showing three-dimensional feature points in a virtual space. A diagram showing an example of the Gaussian DB 123. A diagram for explaining 3D Gaussian. A diagram showing an example of the functional configuration of the information processing device 1. A diagram showing an example of the configuration of the adaptive density control means 115. A flowchart showing an example of the operation flow of the information processing device 1. A flowchart showing an example of the operation flow of the splitting process. A flowchart showing an example of the operation flow of the splitting process in a modified example. A diagram showing an example of the configuration of the adaptive density control means 115 in a modified example. A diagram showing an example of the functional configuration of the information processing device 1 in a modified example.

[0013] <Embodiment> <Configuration of Information Processing Device> Figure 1 is a diagram showing an example of the configuration of an information processing device 1 according to an embodiment of the present invention. The information processing device 1 is a device that generates a virtual image of a virtual object corresponding to an object in a virtual space, viewed from a virtual viewpoint, based on each of the captured images of an object taken from multiple viewpoints, and is, for example, a computer. The information processing device 1 has a processor 11, a memory 12, and an interface 13. These components are connected to each other in a way that allows them to communicate with one another, for example, by a bus.

[0014] The processor 11 controls each part of the information processing device 1 by reading and executing computer programs (hereinafter simply referred to as "programs") stored in the memory 12. The processor 11 is, for example, a CPU (Central Processing Unit) or a GPU.

[0015] Interface 13 is a communication circuit that connects the information processing device 1 to a communication line, an external device, etc., via wired or wireless means, enabling communication.

[0016] Memory 12 is a storage means that stores the operating system, various programs, data, etc., that are loaded into the processor 11. Memory 12 has RAM (Random Access Memory) or ROM (Read Only Memory). Memory 12 may also have a solid-state drive, a hard disk drive, etc. Memory 12 also stores an image DB 121, a camera information DB 122, and a Gaussian DB 123.

[0017] The information processing device 1 may have an operation unit and a display unit. The operation unit is configured to receive operations and send signals corresponding to the content of those operations to the processor 11. This operation unit may include, for example, operation buttons, a keyboard, a touch panel, a mouse, or other controls for giving various instructions.

[0018] This display unit is configured to display images under the control of the processor 11. This display unit may have a display screen such as a liquid crystal display. Furthermore, the transparent touch panel of the operation unit described above may be placed on top of this display screen. The information processing device 1 may be operated from an external device via the interface 13, or information may be presented to an external device.

[0019] <Image DB Configuration> Figure 2 shows an example of the image DB 121. The image DB 121 is a database that stores images of an object taken from multiple viewpoints. The image DB 121 shown in Figure 2 stores image data associated with an image ID. The image ID is identification information that uniquely identifies each of the captured images. The image data is data that indicates the content of the captured image. These captured images stored in the image DB 121 are images of a common object, etc., taken from different viewpoints. The image DB 121 may also be grouped by the object (subject) that was photographed, or by scene. In other words, the image DB 121 only needs to store a series of captured images (group of captured images) for each subject and scene so that the processor 11 can acquire them.

[0020] <Configuration of Camera Information DB> Figure 3 shows an example of the camera information DB 122. The camera information DB 122 is a database that stores information such as the position of the camera that took each captured image (referred to as camera information), associating it with the image ID of that captured image. The camera information DB 122 shown in Figure 3 stores the following items as associations: image ID, position, orientation, and shooting range.

[0021] In the camera information DB122, the "image ID" is identification information that uniquely identifies each captured image, and is information common to the image DB121 described above. In the camera information DB122, the "position" is information that indicates the position of the camera in the actual space (real space Sp) at the time the captured image identified by the corresponding image ID was taken, and is shown, for example, in three-dimensional coordinates in a Cartesian coordinate system.

[0022] For example, according to this camera information DB122, the captured image identified by image ID "I1" was taken from a camera located at the coordinates (x1, y1, z1).

[0023] In the camera information DB122, "orientation" indicates the direction in which the lens of the camera that captured the image is facing in real space Sp, and is represented, for example, by a three-dimensional vector. In the camera information DB122, "shooting range" is information such as parameters that determine the range captured by the camera that captured the image, such as focal length, angle of view, and the center position of the camera's image sensor.

[0024] The information processing device 1 shown in Figure 1 acquires images of an object taken from multiple viewpoints from a camera and stores them in the image DB 121. However, at that point, it does not store the position, orientation, shooting range, etc., of the camera that took those images. Therefore, the information processing device 1 calculates camera information, such as the position of the camera that took the images, from the captured images using, for example, SfM (Structure from Motion), and stores it in the camera information DB 122.

[0025] Figure 4 shows the captured image and camera position. In the following diagrams, the real space in which each component is arranged is represented as the xyz right-handed coordinate system, and the virtual space is represented as the XYZ right-handed coordinate system. In space, the direction along the x-axis is called the x-axis direction. Furthermore, within the x-axis direction, the direction in which the x component increases is called the +x direction, and the direction in which the x component decreases is called the -x direction. The y and z components, and the X, Y, and Z components are also defined according to the above definitions as follows: y-axis direction, +y direction, -y direction, z-axis direction, +z direction, -z direction, X-axis direction, +X direction, -X direction, Y-axis direction, +Y direction, -Y direction, Z-axis direction, +Z direction, and -Z direction.

[0026] Figure 4 illustrates how an object J existing in real space Sp is photographed by cameras from viewpoints (x1, y1, z1) and (x2, y2, z2), respectively, to obtain captured images I1 and I2.

[0027] Figure 5 shows the correspondence between feature points in each of the captured images. The information processing device 1 extracts multiple feature points from each of multiple captured images taken from different viewpoints. The information processing device 1 then infers the correspondence between the feature points based on the arrangement of these feature points in the captured images, the surrounding shape, the distribution of grayscale values, the gradient, etc. For example, the information processing device 1 estimates that the feature points Pa1, Pb1, and Pc1 in captured image I1 correspond to the feature points Pa2, Pb2, and Pc2 in captured image I2, respectively.

[0028] Figure 6 shows three-dimensional feature points in a virtual space. As described above, the information processing device 1 identifies feature points in each captured image and estimates the correspondence between these feature points. This correspondence is the relationship that a single point of an object in real space has been photographed from multiple viewpoints. By estimating the correspondence between feature points, the information processing device 1 identifies a single point of the object that is the basis for those feature points. Then, the information processing device 1 calculates a point in the virtual space Sv corresponding to this single point as a "three-dimensional feature point".

[0029] For example, in the example shown in Figure 6, the three-dimensional feature points Pa, Pb, and Pc correspond to the feature points Pa1, Pb1, and Pc1, respectively, in the captured image I1 shown in Figure 5. Also, the three-dimensional feature points Pa, Pb, and Pc correspond to the feature points Pa2, Pb2, and Pc2, respectively, in the captured image I2 shown in Figure 5. The information processing device 1 then calculates the camera position along the line connecting the calculated three-dimensional feature points and the feature points in the captured image, along with the camera orientation and shooting range.

[0030] <Gaussian DB Configuration> Figure 7 shows an example of Gaussian DB 123. Gaussian DB 123 is a database that stores 3D Gaussians. A 3D Gaussian is an ellipsoid used to reproduce virtual objects in a virtual space, and has various parameters such as position, orientation, shape, color, and opacity. This Gaussian DB 123 shown in Figure 7 stores the following items in association: Gaussian ID, center coordinates, rotation, scale, color, and opacity.

[0031] During the initialization phase, the 3D Gaussian is placed, for example, at the three-dimensional feature points mentioned above. As a result, the Gaussian DB123 stores the 3D Gaussian whose center coordinates coincide with the three-dimensional feature points. For example, as shown in Figure 7, the 3D Gaussian identified by Gaussian ID "E1" is placed at center coordinate "Pa". The center coordinate "Pa" is the three-dimensional feature point Pa corresponding to feature point Pa1 in captured image I1 and feature point Pa2 in captured image I2. The virtual object Jv shown in Figure 6 is constructed by the 3D Gaussians placed at the three-dimensional feature points in the virtual space Sv in this way.

[0032] Figure 8 is a diagram illustrating the 3D Gaussian. Figure 8(a) shows an example of the 3D Gaussian E. The 3D Gaussian E is an ellipsoid in the virtual space Sv that has size (i.e., scale) and central coordinates, and rotation angles with respect to the X, Y, and Z axes, respectively. This 3D Gaussian also possesses color information that depends on the viewing direction, and opacity. These can be represented, for example, by a three-dimensional covariance matrix, harmonic functions, constants, etc.

[0033] Figure 8(b) shows how three 3D Gaussians E1, E2, and E3 are projected onto the screen S. The information processing device 1 projects these multiple 3D Gaussians, which are placed in the virtual space Sv, onto the screen S. The screen S is the surface onto which virtual objects are reflected when any range is photographed from any viewpoint in the virtual space Sv. The 3D Gaussians E1, E2, and E3 are projected onto the screen S by the information processing device 1, becoming projected Gaussians Pj1, Pj2, and Pj3, which are two-dimensional ellipses. These projected Gaussians Pj1, Pj2, and Pj3 may also be displayed overlapping in the virtual space Sv, depending on the order and arrangement of the 3D Gaussians E1, E2, and E3 as seen from the screen S.

[0034] <Functional Configuration of Information Processing Device> Figure 9 shows an example of the functional configuration of the information processing device 1. The processor 11 of the information processing device 1 functions as the image acquisition means 111, bundle adjustment means 112, initialization means 113, learning means 114, and adaptive density control means 115 shown in Figure 9 by reading and executing a program stored in the memory 12.

[0035] The image acquisition means 111 acquires captured images from the image DB 121 in the memory 12. The image acquisition means 111 then supplies the acquired captured images to the bundle adjustment means 112 and the learning means 114.

[0036] The bundle adjustment means 112 calculates camera information from the supplied captured images using the aforementioned SfM, etc., to determine when the images were taken.

[0037] The bundle adjustment means 112 shown in Figure 9 includes a feature point extraction means 1121, a feature point matching means 1122, a verification means 1123, and an estimation means 1124.

[0038] The feature point extraction means 1121 extracts the outlines of objects depicted in the captured image from the position gradient of the grayscale values ​​of the captured image, and extracts feature points from those outlines.

[0039] The feature point matching means 1122 associates (also called matching) the feature points extracted from each of the multiple captured images and calculates the corresponding three-dimensional feature points in the virtual space.

[0040] The feature point extraction means 1121 and the feature point matching means 1122 are performed using feature point matching techniques such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF).

[0041] The verification means 1123 verifies the matched feature points using a method such as RANSAC (RANDOM SAmple Consensus). Based on the verification results, the verification means 1123 excludes feature points that were incorrectly matched due to the influence of outliers or the like.

[0042] The estimation means 1124 estimates camera information from the positional relationship between the calculated three-dimensional feature points in the virtual space and the corresponding feature points in the captured image, and stores this information in the camera information DB 122 of the memory 12.

[0043] As described above, the image acquisition unit 111 acquires a plurality of captured images of an object. Then, the bundle adjustment unit 112 acquires camera information indicating the position, orientation, and shooting range of the camera that captured each of the plurality of captured images acquired by the image acquisition unit 111 by estimating it. Therefore, the image acquisition unit 111 and the bundle adjustment unit 112 are examples of acquisition units that acquire a plurality of images of an object and camera information indicating the position, orientation, and shooting range of the camera that captured each of the plurality of images in association with each other.

[0044] The initialization unit 113 is a unit that arranges a 3D Gaussian in an initial state in a virtual space based on the captured image and the camera information. For example, the initialization unit 113 arranges the 3D Gaussian so that the center coordinates match the three-dimensional feature points in the virtual space calculated by the bundle adjustment unit 112. Then, the initialization unit 113 determines the color and opacity of the 3D Gaussian using the tone value and the like of the feature points of the captured image associated with the three-dimensional feature points. Further, the initialization unit 113 determines the rotation and scale of the 3D Gaussian in the initial state using the positional gradient of the tone value in the captured image or pseudo-random numbers. When the initialization unit 113 determines the initial values such as the center coordinates, rotation, scale, color, and opacity of the 3D Gaussian, it stores them in the Gaussian DB 123 of the memory 12.

[0045] The learning unit 114 performs machine learning in which the 3D Gaussian is projected onto the screen corresponding to the camera in the virtual space and the parameters of the 3D Gaussian are changed so that the obtained virtual image approaches the original captured image.

[0046] The learning unit 114 shown in FIG. 9 includes a calculation unit 1141 and an error minimization unit 1142. The calculation unit 1141 reads out all the 3D Gaussians arranged in the virtual space from the Gaussian DB 123 of the memory 12, projects them onto the screens on the virtual space corresponding to the respective cameras that captured the captured images, and calculates the projected Gaussians respectively.

[0047] Therefore, this calculation means 1141 is an example of a calculation means that calculates a projected Gaussian obtained by projecting each of a plurality of 3D Gaussians arranged in a virtual space onto a screen corresponding to each camera that captured a captured image in the virtual space.

[0048] The error minimization means 1142 compares the virtual image generated by superimposing the calculated projected Gaussians with the captured image acquired by the image acquisition means 111, and changes the parameters of the 3D Gaussian so that the error therebetween (also referred to as a rendering error) is reduced. The calculation means 1141 projects the 3D Gaussian whose parameters have been changed by the error minimization means 1142 onto the above-described screen, and calculates a projected Gaussian. The error minimization means 1142 continues this interaction with the calculation means 1141 until a determined condition is satisfied. In this case, the determined condition is, for example, that the number of times the parameters of the 3D Gaussian are changed becomes equal to or greater than a threshold value.

[0049] The adaptive density control means 115 adjusts the density of the 3D Gaussian by selecting and splitting those that satisfy the condition from among the 3D Gaussians after the learning means 114 has learned by changing the parameters of the 3D Gaussian until the above-described determined condition is satisfied.

[0050] FIG. 10 is a diagram showing an example of the configuration of the adaptive density control means 115. The adaptive density control means 115 shown in FIG. 10 includes an extraction means 1151, a selection means 1152, a splitting means 1153, and an arrangement means 1154.

[0051] The extraction means 1151 extracts only those 3D Gaussians among the 3D Gaussians that have undergone parameter learning by the learning means 114 and for which the maximum size of the projected Gaussian projected onto each of the above-described screens is equal to or greater than a threshold value. That is, this extraction means 1151 is an example of an extraction means that extracts those 3D Gaussians among a plurality of 3D Gaussians for which the maximum size of the projected Gaussian projected onto each screen is equal to or greater than a threshold value.

[0052] The "threshold" used to compare the maximum size of the projected Gaussian, as used here, is a reference value for size based on the resolution of the screen on which the projected Gaussian is projected, such as one pixel. In other words, the extraction means 1151 extracts only those projected Gaussians projected onto all screens that have the largest size of one pixel or more. As a result, at least 3D Gaussians whose projected Gaussians are all less than one pixel in size are excluded from the extraction target of the extraction means 1151.

[0053] The selection means 1152 selects a 3D Gaussian from among the 3D Gaussians extracted by the extraction means 1151 with a probability corresponding to the parameters of those 3D Gaussians. This probability is, for example, a probability corresponding to the opacity of the 3D Gaussians. Here, the selection means 1152 selects several 3D Gaussians from among the 3D Gaussians extracted by the extraction means 1151 with a probability proportional to the opacity of those 3D Gaussians. Therefore, the higher the opacity of the 3D Gaussian, the higher the probability with which the selection means 1152 will select that 3D Gaussian.

[0054] Note that "probability proportional to opacity" is one form of "probability according to opacity" and can be replaced by other forms. For example, the selection means 1152 may select a 3D Gaussian with a probability proportional to the logarithm of the opacity of the 3D Gaussian. Alternatively, for example, the selection means 1152 may select a 3D Gaussian with a probability proportional to a monotonically increasing function with the opacity of the 3D Gaussian as the independent variable.

[0055] The splitting means 1153 splits the 3D Gaussian selected by the selection means 1152. At this time, the splitting means 1153 splits the 3D Gaussian selected by the selection means 1152 into multiple parts such that its total volume is maintained.

[0056] In other words, the selection means 1152 and the splitting means 1153 are examples of splitting means that split a 3D Gaussian selected from the extracted 3D Gaussians with a probability corresponding to the opacity into multiple parts while maintaining the total volume.

[0057] The arrangement means 1154 arranges the 3D Gaussian, which has been split into multiple parts by the splitting means 1153, based on the position of the original 3D Gaussian before splitting. For example, the arrangement means 1154 arranges the split 3D Gaussian so that each part becomes a portion of the original 3D Gaussian. This arrangement means 1154 is an example of an arrangement means that arranges each of the split 3D Gaussian based on the position of the original 3D Gaussian.

[0058] The 3D Gaussian is adjusted in density by the adaptive density control means 115 and stored in the Gaussian DB 123 of the memory 12. The learning means 114 reads the density-adjusted 3D Gaussian from the Gaussian DB 123 and repeatedly optimizes the parameters of these 3D Gaussians based on the difference between the virtual image and the captured image until predetermined conditions are met. This calculates a 3D Gaussian capable of generating a virtual image that is close to the captured image.

[0059] Figure 11 is a flowchart showing an example of the operation flow of the information processing device 1. The processor 11 of the information processing device 1 performs adaptive density control as shown in Figure 11.

[0060] Before performing adaptive density control, the processor 11 retrieves a series of captured images of a 3D scene from the image database 121 in memory 12. The processor 11 then estimates the camera information used when these images were captured, associates it with the captured images, and places an initial state of 3D Gaussian in the virtual space. Furthermore, the processor 11 projects these 3D Gaussian models onto a screen corresponding to the camera in the virtual space to generate a virtual image, for example, until a predetermined condition is met, and optimizes (learns) the parameters of the 3D Gaussian model so that it closely resembles the captured image.

[0061] When the learning process satisfies the conditions described above, the processor 11 starts the adaptive density control operation shown in Figure 11. First, the processor 11 focuses on one of the undetermined 3D Gaussians (step S101). Then, the processor 11 projects the 3D Gaussian of interest onto all screens and compares each of the resulting projected Gaussians with a threshold (step S102).

[0062] The processor 11 identifies the largest projected Gaussian among those described above and determines whether its size (maximum size) is greater than or equal to a threshold (step S103). If it is determined that the maximum size of the projected Gaussian is greater than or equal to the threshold (step S103; YES), the processor 11 extracts that 3D Gaussian of interest as the target of the splitting process described later (step S104) and proceeds to step S105.

[0063] On the other hand, if it is determined that the maximum size of the projected Gaussian is not greater than or equal to a threshold (step S103; NO), the processor 11 proceeds to step S105 without performing step S104 described above. In other words, in this case, the 3D Gaussian of interest is not extracted as a target for splitting.

[0064] Processor 11 determines that the 3D Gaussian that it was focusing on in steps S102 to S104 has been determined, and then determines whether there are any undetermined 3D Gaussians remaining (step S105). If it determines that there are any undetermined 3D Gaussians remaining (step S105; YES), processor 11 returns to step S101.

[0065] On the other hand, if it is determined that there are no undetermined 3D Gaussians remaining (step S105; NO), the processor 11 determines whether or not there are any 3D Gaussians extracted in step S104 (step S106).

[0066] If it is determined that there are no extracted 3D Gaussians (step S106; NO), the processor 11 terminates the process. On the other hand, if it is determined that there are extracted 3D Gaussians (step S106; YES), the processor 11 performs a splitting process on those extracted 3D Gaussians (step S200).

[0067] Figure 12 is a flowchart showing an example of the operation flow of the splitting process. Processor 11 identifies the opacity of each 3D Gaussian extracted in step S104 and selects several 3D Gaussians with a probability proportional to their opacity (step S201). Processor 11 selects 3D Gaussians by sampling according to, for example, a multinomial distribution of the identified opacities.

[0068] Next, the processor 11 splits each of the selected 3D Gaussians into multiple parts while maintaining their total volume (step S202).

[0069] Furthermore, the processor 11 arranges the newly obtained 3D Gaussians based on the positions of the original 3D Gaussians before splitting (step S203).

[0070] In this way, the information processing device 1 performs adaptive density control, which splits the 3D Gaussian initialized according to the feature points of the captured image. This improves spatial resolution and shape representation, making it possible to represent the details of the 3D scene more accurately.

[0071] In particular, this information processing device 1 selects the split 3D Gaussian based on the opacity of the 3D Gaussian optimized during the learning process, thereby precisely improving the spatial resolution of the parts with high opacity that have a high proportion of influence on the virtual image reconstructed (generated) on the screen. As a result, this information processing device 1 can achieve high-speed yet high-definition virtual image generation.

[0072] Furthermore, since the information processing device 1 does not perform 3D Gaussian replication, it can eliminate the processing time required for replication and improve rendering speed.

[0073] <Modifications> The above describes the embodiment, but the contents of this embodiment can be modified as follows. Furthermore, the following modifications may be combined.

[0074] <1> In the embodiments described above, the processor 11 was a CPU or a GPU, but it may have other configurations. For example, the processor 11 may be an FPGA (Field Programmable Gate Array) or may include an FPGA. The processor 11 may also have an ASIC (Application Specific Integrated Circuit) or other programmable logic device. The information processing device 1 may also have multiple processors, multiple memories, and multiple interfaces. The information processing device 1 may be, for example, a mobile terminal such as a smartphone or a slate PC. The information processing device 1 may also be, for example, a virtual machine realized by the dynamic cooperation of multiple computing resources on the cloud via a communication line such as the Internet.

[0075] <2> In the embodiments described above, the information processing device 1 selected a 3D Gaussian with a probability corresponding to the opacity and split it so that the total volume of the selected 3D Gaussian was maintained. However, the criteria for selecting the 3D Gaussian to be split are not limited to this.

[0076] Figure 13 is a flowchart showing an example of the operation flow of the splitting process in a modified example. In step S200 shown in Figure 13, step S201a is executed instead of step S201 shown in Figure 12. That is, the processor 11 selects a 3D Gaussian with a probability proportional to the opacity as well as the maximum size of the projected Gaussian (step S201a). As a result, the information processing device 1 becomes more likely to split the 3D Gaussian as the opacity increases, and also as the maximum size of the projected Gaussian increases. Therefore, the information processing device 1 can improve the spatial resolution of parts with high opacity that have a high proportion of influence on the virtual image reconstructed (generated) on the screen, as well as parts that have a high proportion of influence on the virtual image because they occupy a large area on the screen when projected.

[0077] In other words, the splitting means 1153 implemented by the processor 11 in this modified example is an example of a splitting means that splits a 3D Gaussian selected from the extracted 3D Gaussians with a probability corresponding to the opacity and the maximum size of the projected Gaussian, so as to maintain the total volume.

[0078] <3> In the embodiments described above, the information processing device 1 only splits the 3D Gaussian and does not perform replication. However, if certain conditions are met, one of the 3D Gaussians may be replicated.

[0079] Figure 14 shows an example of the configuration of the adaptive density control means 115 in a modified example. The adaptive density control means 115 shown in Figure 14 differs from that shown in Figure 10 in that it has a computation load determination means 1155 and a replication means 1156.

[0080] The computation load determination means 1155 is configured to monitor the computation load of the information processing device 1. This computation load determination means 1155 determines whether the load on the processor 11, or the computation resources controlled by the processor 11, is below a predetermined level.

[0081] If the computation load determination means 1155 determines that the computation load is below a predetermined level, it supplies the 3D Gaussian that was not extracted by the extraction means to the replication means 1156. The replication means 1156 replicates the 3D Gaussian supplied by the computation load determination means 1155 that meets the predetermined conditions.

[0082] For example, the replication means 1156 replicates the 3D Gaussian in the virtual space if its size (three-dimensional size) is smaller than a certain threshold. This allows the information processing device 1 to improve the accuracy of rendering over a wide range, which is difficult to achieve with splitting, and to enhance the general shape and low-frequency characteristics when reproducing an object as a virtual object. Furthermore, since the information processing device 1 only replicates the 3D Gaussian when sufficient computing resources are available, it is less likely to interfere with the splitting process of the 3D Gaussian in adaptive density control, and is less likely to reduce the rendering speed.

[0083] In this modified example, the computation load determination means 1155 and the duplication means 1156 are examples of duplication means that, when the computation load is below a predetermined level, duplicates a plurality of 3D Gaussians that satisfy certain conditions.

[0084] <4> In the embodiment described above, the processor 11 of the information processing device 1 functioned as a bundle adjustment means 112 that calculated camera information at the time of capture from the captured images supplied by the image acquisition means 111. However, the camera information may be stored in the memory 12 in advance in association with the captured images.

[0085] Figure 15 shows an example of the functional configuration of the information processing device 1 in a modified example. Camera information DB 122 stored in memory 12 has camera information pre-stored for each image ID that identifies an image in image DB 121. In this case, the processor 11 does not need to estimate camera information from the feature points of the captured image. As shown in Figure 15, the processor 11 simply needs to function as a camera information acquisition means 112a that reads the camera information associated with the image ID in camera information DB 122.

[0086] The image database 121 may be integrated with the camera information database 122. For example, when a camera photographs an object, it may store its own camera information as metadata for the captured image. The camera information may be generated when photographing an object by a barometer, accelerometer, gyro sensor, GNSS, etc., attached to the camera itself. If there is a moving device for moving the camera, the camera information may be generated using a measuring instrument for the amount of movement provided on this moving device. Furthermore, the camera information may be pre-associated with the position where the camera is attached, or with camera identification information, etc.

[0087] <5> In the above-described embodiment, the information processing device 1 extracted only those 3D Gaussians whose maximum size projected onto each screen was greater than or equal to a threshold, from among the 3D Gaussians whose parameters had been learned, and selected several 3D Gaussians from among the extracted 3D Gaussians as targets for splitting with a probability corresponding to the opacity. However, the order of extraction and selection may be reversed.

[0088] For example, the adaptive density control means 115 shown in Figure 15 is configured such that the extraction means 1151 is executed after the selection means 1152, followed by the splitting means 1153 and the placement means 1154. In this case, the selection means 1152 selects several 3D Gaussians from the learned 3D Gaussians with a probability corresponding to the opacity.

[0089] Then, the extraction means 1151 extracts a 3D Gaussian from the 3D Gaussians selected by the selection means 1152 such that the maximum size of the projected Gaussian is equal to or greater than a threshold. The splitting means 1153 splits the 3D Gaussian extracted by the extraction means 1151 into multiple parts so as to maintain the total volume.

[0090] Therefore, the selection means 1152 shown in Figure 15 is an example of a selection means that selects a 3D Gaussian from among a plurality of 3D Gaussians with a probability corresponding to the opacity.

[0091] Furthermore, the extraction means 1151 and the splitting means 1153 shown in Figure 15 are examples of splitting means that, from among 3D Gaussians selected according to opacity, etc., split a 3D Gaussian whose maximum size projected onto each screen exceeds a threshold, so as to maintain its total volume.

[0092] <6> In the embodiment described above, the program executed by the processor 11 is conceivable as a program that causes the computer to function as: an acquisition means that acquires a plurality of images of an object and camera information indicating the position, orientation, and shooting range of the camera that took each of the plurality of images; a calculation means that calculates a projected Gaussian by projecting each of the plurality of 3D Gaussians arranged in a virtual space onto a screen corresponding to each camera in that virtual space; an extraction means that extracts from the plurality of 3D Gaussians the one whose largest projected Gaussian projected onto each screen is greater than or equal to a threshold; and a splitting means that splits the 3D Gaussians selected from the extracted 3D Gaussians with a probability according to the opacity into a plurality so as to maintain the total volume.

[0093] <7> The method of causing the processor 11 described above to execute the program described above can also be conceived as an adaptive density control method that causes the computer to control the density of 3D Gaussians. That is, the adaptive density control method of 3D Gaussians described above is an example of an adaptive density control method of 3D Gaussians characterized by including the steps of: the computer acquiring a plurality of images of an object and camera information indicating the position, orientation, and shooting range of the camera that took each of the plurality of images in association with each other; the computer calculating a projected Gaussian by projecting each of the plurality of 3D Gaussians arranged in a virtual space onto a screen corresponding to each camera in that virtual space; the computer extracting from the plurality of 3D Gaussians the one whose largest projected Gaussian projected onto each screen is greater than or equal to a threshold; and the computer splitting the 3D Gaussians selected from the extracted 3D Gaussians with a probability according to the opacity into a plurality so as to maintain the total volume.

[0094] 1... Information processing device, 11... Processor, 111... Image acquisition means, 112... Bundle adjustment means, 1121... Feature point extraction means, 1122... Feature point matching means, 1123... Verification means, 1124... Estimation means, 112a... Camera information acquisition means, 113... Initialization means, 114... Learning means, 1141... Calculation means, 1142... Error minimization means, 115... Adaptive density control means, 1151... Extraction means, 1152... Selection means, 1153... Splitting means, 1154... Arrangement means, 1155... Computation load determination means, 1156... Replication means, 12... Memory, 121... Image DB, 122... Camera information DB, 123... Gaussian DB, 13... Interface.

Claims

1. A program to cause a computer to perform the following steps:

1. Acquire multiple images of an object, and camera information indicating the position, orientation, and shooting range of the camera that took each of the multiple images, by associating them with each other; 2. Calculate a projected Gaussian by projecting each of the multiple 3D Gaussians placed in a virtual space onto a screen corresponding to each of the cameras in the virtual space; 3. Extract from the multiple 3D Gaussians the one whose largest projected Gaussian projected onto each of the screens is greater than or equal to a threshold; and 4. From the extracted 3D Gaussians, divide the 3D Gaussians selected with a probability according to opacity into multiple parts so as to maintain their total volume.

2. The program according to claim 1, wherein in the splitting step, a 3D Gaussian selected from the extracted 3D Gaussians with a probability corresponding to the opacity and the maximum size of the projected Gaussian, respectively, is split into multiple parts such that the total volume is maintained.

3. The program according to claim 1, which causes the computer to further perform the step of duplicating the plurality of 3D Gaussian models that satisfy certain conditions when the computational load is below a predetermined level.

4. The program according to claim 1, which causes the computer to further perform the step of arranging each of the divided 3D Gaussians based on the positions of the original 3D Gaussian.

5. A program for causing a computer to perform the following steps: acquire multiple images of an object and camera information indicating the position, orientation, and shooting range of the camera that took each of the multiple images, associating them with each other; calculate a projected Gaussian by projecting each of the multiple 3D Gaussians placed in a virtual space onto a screen corresponding to each of the cameras in the virtual space; select a 3D Gaussian from the multiple 3D Gaussians with a probability according to the opacity; and split the 3D Gaussian from the selected 3D Gaussians into multiple 3D Gaussians such that the largest projected Gaussian projected onto each of the screens is greater than or equal to a threshold, while maintaining the total volume.

6. An information processing device comprising: an acquisition means for acquiring multiple images of an object and camera information indicating the position, orientation, and shooting range of the camera that captured each of the multiple images, in association with each of these; a calculation means for calculating a projected Gaussian by projecting each of the multiple 3D Gaussians arranged in a virtual space onto a screen corresponding to each of the cameras in the virtual space; an extraction means for extracting from the multiple 3D Gaussians the one whose largest projected Gaussian projected onto each of the screens is greater than or equal to a threshold; and a splitting means for splitting the 3D Gaussians selected from the extracted 3D Gaussians with a probability according to opacity into multiple parts while maintaining their total volume.

7. A method for controlling the adaptive density of a 3D Gaussian using a computer, comprising the steps of: acquiring multiple images of an object and camera information indicating the position, orientation, and shooting range of the camera that took each of the multiple images, in association with each of them; calculating a projected Gaussian by projecting each of the multiple 3D Gaussians placed in a virtual space onto a screen corresponding to each of the cameras in the virtual space; extracting from the multiple 3D Gaussians the one whose largest projected Gaussian projected onto each of the screens is greater than or equal to a threshold; and splitting the 3D Gaussian selected by the computer from among the extracted 3D Gaussians with a probability according to opacity into multiple parts such that the total volume is maintained.

Images