Program, information processing device, and adaptive density control method

Abstract

This realizes adaptive density control of 3D Gaussian suitable for reproducing the details of objects depicted in captured images. [Solution] An extraction unit 1151 extracts only those 3D Gaussians whose maximum size when projected onto each screen is equal to or greater than a threshold, from among the 3D Gaussians whose parameters have been learned by a learning unit 114. A selection unit 1152 selects several 3D Gaussians from the 3D Gaussians extracted by the extraction unit 1151 with a probability proportional to the opacity of the 3D Gaussians. A division unit 1153 divides the 3D Gaussian selected by the selection unit 1152 into multiple pieces so that the total volume is maintained. An arrangement unit 1154 arranges each of the multiple 3D Gaussians divided by the division unit 1153 based on the position of the original 3D Gaussian before the division.

Description

[Technical Field]

[0001] The present invention relates to a technique for generating a new image of an object viewed from an arbitrary viewpoint using images of the object photographed from multiple viewpoints. [Background technology]

[0002] Photogrammetry is a technology for estimating the three-dimensional shape of an object from images (also called photographed images) of the object taken from multiple viewpoints. For example, Patent Document 1 discloses, as a photogrammetry technology, an apparatus and method for generating a three-dimensional (3D) model of a subject from images taken by multiple cameras.

[0003] Furthermore, research is being conducted on a technology that generates an image (also called a virtual image) of an object in a virtual space (also called a virtual object) as seen from an arbitrary viewpoint (also called a virtual viewpoint) based on images of the object photographed from multiple viewpoints. Known technologies for this purpose include NeRF (Neural Radiance Fields) and NeuS (Neural Implicit Surface), but these technologies have the problem of requiring a large load and time for machine learning.

[0004] Meanwhile, 3DGS (3D Gaussian Splatting) is attracting attention as a technology that can achieve high-speed, high-quality rendering. 3DGS is a technology that recreates a 3D scene by placing ellipsoids called 3D Gaussians in a 3D virtual space and optimizing the parameters of these 3D Gaussians to match the original captured image.

[0005] Unlike the implicit representation used by NeRF and the like, 3DGS explicitly represents virtual objects in virtual space using ellipsoids called 3D Gaussians, enabling rendering processing to be completed in a short time. Furthermore, because 3DGS uses differentiable rasterization, using a GPU (Graphics Processing Unit) can speed up processing compared to NeRF and the like, which use volume rendering. For example, Non-Patent Documents 1 to 3 disclose various techniques related to 3DGS. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-71749 [Non-patent literature]

[0007] [Non-Patent Document 1] https: / / arxiv.org / abs / 2308.04079 [Non-patent document 2] https: / / arxiv.org / abs / 2404.06109 [Non-patent document 3] https: / / ubc-vision.github.io / 3dgs-mcmc / Summary of the Invention [Problem to be solved by the invention]

[0008] To recreate a 3D scene, a 3D Gaussian of a size appropriate to the details of the 3D scene is required. Obtaining such a 3D Gaussian is limited by simply optimizing the size of the initial 3D Gaussian by comparing it with the original captured image. Therefore, in addition to this optimization process, 3DGS replicates or splits the 3D Gaussian placed during initialization according to specified conditions. The control of splitting and replicating 3D Gaussians and adjusting their density is generally called "adaptive density control."

[0009] However, in conventional techniques such as those described in Non-Patent Document 3, adaptive density control of 3D Gaussians involves duplicating 3D Gaussians smaller than a certain size in a virtual space and splitting only 3D Gaussians larger than a certain size. As a result, conventional techniques may not be able to reproduce the details of objects depicted in the original captured image.

[0010] One of the objects of the present invention is to realize adaptive density control of a 3D Gaussian that is more suitable for reproducing details of an object depicted in a captured image than conventional techniques. [Means for solving the problem]

[0011] In one aspect, the present invention provides a program for causing a computer to function as an acquisition means for acquiring a plurality of images of an object and camera information indicating the position, orientation, and shooting range of the camera that captured each of the plurality of images, each of which is associated with the other; a calculation means for calculating a projected Gaussian by projecting each of a plurality of 3D Gaussians placed in a virtual space onto a screen in the virtual space corresponding to each of the cameras; an extraction means for extracting, from the plurality of 3D Gaussians, those projected Gaussians whose maximum size when projected onto each of the screens is equal to or greater than a threshold; and a division means for selecting from the extracted 3D Gaussians with a probability according to opacity and dividing the selected 3D Gaussians into multiple 3D Gaussians so that the total volume is maintained. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram showing an example of the configuration of an information processing device 1 according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of an image DB 121. [Figure 3] FIG. 4 is a diagram showing an example of a camera information DB 122. [Figure 4] FIG. 2 is a diagram showing a captured image and a camera position. [Figure 5] 4A and 4B are diagrams showing correspondences between feature points in each captured image. [Figure 6] FIG. 1 is a diagram showing three-dimensional feature points in a virtual space. [Figure 7] A diagram showing an example of Gaussian DB123. [Figure 8] A diagram to explain 3D Gaussian. [Figure 9] FIG. 2 is a diagram showing an example of the functional configuration of the information processing device 1. [Figure 10] FIG. 2 is a diagram showing an example of the configuration of adaptive density control means 115. [Figure 11] FIG. 3 is a flowchart showing an example of the flow of operations of the information processing device 1. [Figure 12] FIG. 10 is a flowchart showing an example of the operational flow of a division process. [Figure 13] FIG. 10 is a flowchart showing an example of the operation flow of a division process in a modified example. [Figure 14] FIG. 10 is a diagram showing an example of the configuration of adaptive density control means 115 in a modified example. [Figure 15] FIG. 10 is a diagram showing an example of the functional configuration of an information processing device 1 according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0013] <Embodiment> <Configuration of information processing device> 1 is a diagram showing an example of the configuration of an information processing device 1 according to an embodiment of the present invention. The information processing device 1 is a device, such as a computer, that generates, based on each of captured images of an object taken from a plurality of viewpoints, a virtual image of a virtual object in a virtual space, viewed from a virtual viewpoint, corresponding to the object. The information processing device 1 has a processor 11, a memory 12, and an interface 13. These components are connected to each other so that they can communicate with each other, for example, by a bus.

[0014] The processor 11 reads and executes a computer program (hereinafter simply referred to as a program) stored in the memory 12 to control each part of the information processing device 1. The processor 11 is, for example, a CPU (Central Processing Unit) or a GPU.

[0015] The interface 13 is a communication circuit that connects the information processing device 1 to a communication line, an external device, etc., in a wired or wireless manner so that the information processing device 1 can communicate with each other.

[0016] The memory 12 is a storage means for storing an operating system, various programs, data, etc., which are loaded into the processor 11. The memory 12 includes a random access memory (RAM) or a read only memory (ROM). The memory 12 may include a solid state drive, a hard disk drive, etc. The memory 12 also stores an image DB 121, a camera information DB 122, and a Gaussian DB 123.

[0017] The information processing device 1 may have an operation unit and a display unit. The operation unit is configured to receive an operation and send a signal corresponding to the operation content to the processor 11. The operation unit may include, for example, an operation button, a keyboard, a touch panel, a mouse, or other operator for issuing various instructions.

[0018] The display unit is configured to display images under the control of the processor 11. The display unit may have a display screen such as a liquid crystal display. Furthermore, the transparent touch panel of the operation unit described above may be placed on top of the display screen. The information processing device 1 may be operated from an external device via the interface 13, or may present information to an external device.

[0019] <Image DB configuration> FIG. 2 is a diagram showing an example of image DB 121. Image DB 121 is a database that stores photographed images of an object taken from multiple viewpoints. This image DB 121 shown in FIG. 2 stores image data in association with an image ID. The image ID is identification information that uniquely identifies each photographed image. The image data is data that indicates the content of the photographed image. These photographed images stored in image DB 121 are images of a common object or the like taken from different viewpoints. Note that image DB 121 may be grouped by object (subject) that was the subject of the photograph or by scene. In other words, image DB 121 only needs to store a series of photographed images (a group of photographed images) for each subject and scene so that processor 11 can acquire them.

[0020] <Camera information DB configuration> Fig. 3 is a diagram showing an example of camera information DB 122. Camera information DB 122 is a database that stores information (called camera information) such as the position of the camera that captured each captured image, in association with the image ID of the captured image. Camera information DB 122 shown in Fig. 3 stores the image ID, position, orientation, and capture range in association with each other.

[0021] The "image ID" in the camera information DB 122 is identification information that uniquely identifies each captured image, and is information common to the above-mentioned image DB 121. The "position" in the camera information DB 122 is information that indicates the position of the camera in the actual space (real space Sp) when the captured image identified by the corresponding image ID was captured, and is indicated by, for example, three-dimensional coordinates in a Cartesian coordinate system.

[0022] For example, the camera information DB 122 indicates that the captured image identified by the image ID "I1" was captured by a camera located at the coordinates indicated by (x1, y1, z1).

[0023] The "orientation" in the camera information DB 122 is information indicating the direction in which the lens of the camera that captured the captured image is facing in the real space Sp, and is represented by, for example, a three-dimensional vector. The "capture range" in the camera information DB 122 is information such as parameters that determine the range captured by the camera that captured the captured image, such as the focal length, the angle of view, and the center position of the camera's image sensor.

[0024] 1 acquires captured images of an object from multiple viewpoints using a camera and stores them in an image DB 121, but does not store the position, orientation, shooting range, etc. of the camera that captured the captured images at that time. Therefore, the information processing device 1 calculates camera information such as the position of the camera that captured the captured images from the captured images using, for example, SfM (Structure from Motion) or the like, and stores the information in a camera information DB 122.

[0025] FIG. 4 is a diagram showing a captured image and a camera position. In the diagrams shown below, the real space in which each component is arranged is represented as an xyz right-handed coordinate space, and the virtual space is represented as an XYZ right-handed coordinate space. The direction along the x-axis in space is referred to as the x-axis direction. Furthermore, within the x-axis direction, the direction in which the x component increases is referred to as the +x direction, and the direction in which the x component decreases is referred to as the -x direction. Regarding the y- and z-components, as well as the X-, Y-, and Z-components, the y-axis direction, +y-direction, -y-direction, z-axis direction, +z-direction, -z-direction, X-axis direction, +X-direction, -X-direction, Y-axis direction, +Y-direction, -Y-direction, Z-axis direction, +Z-direction, and -Z-direction are defined in accordance with the above definitions.

[0026] FIG. 4 illustrates how an object J existing in a real space Sp is photographed by a camera from viewpoints (x1, y1, z1) and (x2, y2, z2), respectively, to obtain a photographed image I1 and a photographed image I2.

[0027] 5 is a diagram showing the correspondence between feature points in each captured image. The information processing device 1 extracts multiple feature points from each of multiple captured images captured from different viewpoints. The information processing device 1 then estimates the correspondence between the feature points based on the positional relationship of the feature points in the captured image, the shape of their surroundings, the distribution and gradient of gradation values, etc. For example, the information processing device 1 estimates that feature points Pa1, Pb1, and Pc1 in captured image I1 correspond to feature points Pa2, Pb2, and Pc2 in captured image I2, respectively.

[0028] FIG. 6 is a diagram showing three-dimensional feature points in a virtual space. As described above, the information processing device 1 identifies feature points in each of the captured images and estimates the correspondence between those feature points. This correspondence is a relationship in which a single point on an object in real space is photographed from multiple viewpoints. By estimating the correspondence between the feature points, the information processing device 1 identifies a single point on the object that is the basis of those feature points. Then, the information processing device 1 calculates a point in the virtual space Sv that corresponds to this single point as a "three-dimensional feature point."

[0029] For example, in the example shown in Fig. 6, three-dimensional feature points Pa, Pb, and Pc correspond to feature points Pa1, Pb1, and Pc1, respectively, in captured image I1 shown in Fig. 5. Furthermore, three-dimensional feature points Pa, Pb, and Pc correspond to feature points Pa2, Pb2, and Pc2, respectively, in captured image I2 shown in Fig. 5. Then, the information processing device 1 calculates the position of the camera on a line connecting the calculated three-dimensional feature points and the feature points in the captured image, together with the camera orientation and shooting range.

[0030] <Gaussian DB configuration> Fig. 7 is a diagram showing an example of the Gaussian DB 123. The Gaussian DB 123 is a database that stores 3D Gaussians. A 3D Gaussian is an ellipsoid used to reproduce a virtual object in a virtual space, and has various parameters such as position, orientation, shape, color, and opacity. The Gaussian DB 123 shown in Fig. 7 stores the Gaussian ID, center coordinates, rotation, scale, color, and opacity in association with each other.

[0031] At the initialization stage, the 3D Gaussians are placed at, for example, the three-dimensional feature points described above. As a result, the Gaussian DB 123 stores 3D Gaussians whose center coordinates coincide with the three-dimensional feature points. For example, as shown in FIG. 7, a 3D Gaussian identified by a Gaussian ID "E1" is placed at center coordinate "Pa." The center coordinate "Pa" is the three-dimensional feature point Pa corresponding to the feature point Pa1 in the captured image I1 and the feature point Pa2 in the captured image I2. The 3D Gaussians placed at the three-dimensional feature points in the virtual space Sv in this way constitute a virtual object Jv shown in FIG. 6.

[0032] FIG. 8 is a diagram for explaining a 3D Gaussian. FIG. 8(a) shows an example of a 3D Gaussian E. The 3D Gaussian E is an ellipsoid that has a size (i.e., scale) and center coordinates in the virtual space Sv, and has rotation angles relative to the X-axis, Y-axis, and Z-axis. This 3D Gaussian has color information and opacity that depend on the viewing direction. These are expressed, for example, by a three-dimensional covariance matrix, a harmonic function, a constant, etc.

[0033] FIG. 8(b) shows three 3D Gaussians E1, E2, and E3 being projected onto a screen S. The information processing device 1 projects these multiple 3D Gaussians arranged in a virtual space Sv onto the screen S. The screen S is a surface on which virtual objects are reflected when an arbitrary range is photographed from an arbitrary viewpoint in the virtual space Sv. The 3D Gaussians E1, E2, and E3 are projected onto the screen S by the information processing device 1, respectively, to become projected Gaussians Pj1, Pj2, and Pj3, which are two-dimensional ellipses. These projected Gaussians Pj1, Pj2, and Pj3 may be displayed overlapping each other in the virtual space Sv depending on the order and positional relationship of the 3D Gaussians E1, E2, and E3 as viewed from the screen S.

[0034] <Functional configuration of information processing device> Fig. 9 is a diagram showing an example of the functional configuration of the information processing device 1. The processor 11 of the information processing device 1 reads and executes programs stored in the memory 12, thereby functioning as image acquisition means 111, bundle adjustment means 112, initialization means 113, learning means 114, and adaptive density control means 115 shown in Fig. 9.

[0035] The image acquisition means 111 acquires a photographed image from the image DB 121 in the memory 12. Then, the image acquisition means 111 supplies the acquired photographed image to the bundle adjustment means 112 and the learning means 114.

[0036] The bundle adjustment means 112 calculates the camera information from the supplied captured images when they were captured using the above-mentioned SfM or the like.

[0037] The bundle adjustment means 112 shown in FIG. 9 includes a feature point extraction means 1121, a feature point matching means 1122, a verification means 1123, and an estimation means 1124.

[0038] The feature point extraction means 1121 extracts the contours of objects shown in the captured image from the position gradient of the gradation values ​​of the captured image, and extracts feature points from these contours.

[0039] The feature point matching means 1122 associates (also called matching) the feature points extracted from each of the plurality of captured images, and calculates the three-dimensional feature points in the virtual space that correspond to them.

[0040] The feature point extraction means 1121 and the feature point matching means 1122 are implemented using feature point matching techniques such as SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF).

[0041] The verification unit 1123 verifies the matched feature points using a method such as RANSAC (RANdom SAmple Consensus), etc. As a result of the verification, the verification unit 1123 excludes feature points that have been erroneously matched due to the influence of outliers, etc.

[0042] The estimation means 1124 estimates camera information from the positional relationship between the calculated three-dimensional feature points in the virtual space and the corresponding feature points in the captured image, and stores this in the camera information DB 122 in the memory 12.

[0043] As described above, the image acquisition means 111 acquires a plurality of captured images of an object. Then, the bundle adjustment means 112 acquires the plurality of captured images acquired by the image acquisition means 111 by estimating camera information indicating the position, orientation, and capturing range of the camera that captured each of the plurality of captured images. Therefore, the image acquisition means 111 and the bundle adjustment means 112 are examples of acquisition means that acquire a plurality of captured images of an object and camera information indicating the position, orientation, and capturing range of the camera that captured each of the plurality of images in association with each other.

[0044] The initialization means 113 is a means for arranging an initial 3D Gaussian in a virtual space based on the captured image and camera information. For example, the initialization means 113 arranges the 3D Gaussian so that its center coordinates coincide with the three-dimensional feature point in the virtual space calculated by the bundle adjustment means 112. The initialization means 113 then determines the color and opacity of the 3D Gaussian using, for example, the gradation values ​​of the feature point in the captured image associated with the three-dimensional feature point. The initialization means 113 also determines the rotation and scale of the 3D Gaussian in the initial state using, for example, the position gradient of the gradation values ​​in the captured image or pseudo-random numbers. After determining the initial values ​​of the center coordinate, rotation, scale, color, opacity, etc. of the 3D Gaussian, the initialization means 113 stores them in the Gaussian DB 123 of the memory 12.

[0045] The learning means 114 performs machine learning by projecting a 3D Gaussian onto a screen corresponding to a camera in a virtual space and changing the parameters of the 3D Gaussian so that the obtained virtual image approaches the original captured image.

[0046] 9 includes a calculation means 1141 and an error minimization means 1142. The calculation means 1141 reads out all 3D Gaussians arranged in the virtual space from the Gaussian DB 123 in the memory 12, projects these onto a screen in the virtual space corresponding to each camera that captured the captured image, and calculates each projected Gaussian.

[0047] Therefore, this calculation means 1141 is an example of a calculation means that calculates a projected Gaussian by projecting each of multiple 3D Gaussians placed in a virtual space onto a screen corresponding to each camera that captured the captured image in that virtual space.

[0048] The error minimization means 1142 compares a virtual image generated by superimposing the calculated projected Gaussians with the captured image acquired by the image acquisition means 111, and changes the parameters of the 3D Gaussians so as to reduce the error (also called rendering error) between them. The calculation means 1141 projects the 3D Gaussians whose parameters have been changed by the error minimization means 1142 onto the above-mentioned screen, and calculates the projected Gaussians. The error minimization means 1142 continues this exchange with the calculation means 1141 until a predetermined condition is satisfied. In this case, the predetermined condition is, for example, that the number of times the parameters of the 3D Gaussians have been changed is equal to or exceeds a threshold value.

[0049] The adaptive density control means 115 adjusts the density of the 3D Gaussians by changing and learning the parameters of the 3D Gaussians until the learning means 114 satisfies the above-mentioned predetermined conditions, and then selecting and splitting the 3D Gaussians that satisfy the conditions from among the 3D Gaussians.

[0050] 10 is a diagram showing an example of the configuration of the adaptive density control means 115. The adaptive density control means 115 shown in FIG.

[0051] The extraction means 1151 extracts only those 3D Gaussians whose maximum size when projected onto each of the above-mentioned screens is equal to or greater than a threshold value, from among the 3D Gaussians whose parameters have been learned by the learning means 114. In other words, the extraction means 1151 is an example of an extraction means that extracts those 3D Gaussians whose maximum size when projected onto each of the screens is equal to or greater than a threshold value, from among a plurality of 3D Gaussians.

[0052] The "threshold" compared with the maximum size of the projected Gaussian here is a reference value of size based on the resolution of the screen onto which the projected Gaussian is projected, such as 1 pixel. That is, the extraction means 1151 extracts, for example, only the projected Gaussian whose largest size is 1 pixel or more among all the projected Gaussians projected onto the screen. As a result, at least 3D Gaussians whose projected Gaussians are all less than 1 pixel in size are excluded from the extraction targets of the extraction means 1151.

[0053] The selection means 1152 selects 3D Gaussians from among the 3D Gaussians extracted by the extraction means 1151 with a probability according to the parameters of the 3D Gaussians, etc. This probability is, for example, a probability according to the opacity of the 3D Gaussians. Here, the selection means 1152 selects some 3D Gaussians from among the 3D Gaussians extracted by the extraction means 1151 with a probability proportional to the opacity of the 3D Gaussians. Therefore, the higher the opacity of a 3D Gaussian, the higher the probability that the selection means 1152 will select that 3D Gaussian.

[0054] Note that the "probability proportional to opacity" is one aspect of the "probability according to opacity" and may be replaced by other aspects. For example, the selection unit 1152 may select a 3D Gaussian with a probability proportional to the logarithm of the opacity of the 3D Gaussian. Also, for example, the selection unit 1152 may select a 3D Gaussian with a probability proportional to a monotonically increasing function with the opacity of the 3D Gaussian as an independent variable.

[0055] The dividing means 1153 divides the 3D Gaussian selected by the selecting means 1152. At this time, the dividing means 1153 divides the 3D Gaussian selected by the selecting means 1152 into a plurality of pieces so that the total volume of the 3D Gaussian is maintained.

[0056] In other words, the selection means 1152 and the division means 1153 are examples of division means that divide a 3D Gaussian selected from the extracted 3D Gaussians with a probability according to the opacity into multiple pieces so that the total volume is maintained.

[0057] The arranging means 1154 arranges each of the 3D Gaussians split into multiple pieces by the splitting means 1153 based on the position of the original 3D Gaussian before the split. For example, the arranging means 1154 arranges the split 3D Gaussians so that each becomes a part of the original 3D Gaussian. The arranging means 1154 is an example of an arrangement means that arranges each of the split 3D Gaussians based on the position of the original 3D Gaussian.

[0058] The density of the 3D Gaussians is adjusted by the adaptive density control means 115 and stored in the Gaussian DB 123 of the memory 12. The learning means 114 reads the density-adjusted 3D Gaussians from the Gaussian DB 123 and repeatedly optimizes the parameters of these 3D Gaussians based on the difference between the virtual image and the photographed image until a predetermined condition is satisfied. In this way, 3D Gaussians capable of generating a virtual image similar to the photographed image are calculated.

[0059] 11 is a flow diagram showing an example of the flow of operations of the information processing device 1. The processor 11 of the information processing device 1 performs adaptive density control shown in FIG.

[0060] Before performing adaptive density control, processor 11 acquires a series of captured images of a 3D scene from image DB 121 in memory 12. Then, processor 11 estimates camera information used when these captured images were taken, associates them with the captured images, and places initial 3D Gaussians in virtual space. Furthermore, processor 11 generates virtual images by projecting these 3D Gaussians onto a screen corresponding to the camera in virtual space until, for example, a predetermined condition is met, and optimizes (learns) the parameters of the 3D Gaussians so that the virtual images approach the captured images.

[0061] When the learning process satisfies the above-mentioned conditions, processor 11 starts the operation of adaptive density control shown in Fig. 11. First, processor 11 focuses on one of the undetermined 3D Gaussians (step S101). Then, processor 11 projects the focused 3D Gaussian onto all screens and compares each of the obtained projected Gaussians with a threshold (step S102).

[0062] Processor 11 identifies the largest of the above-mentioned projected Gaussians and determines whether its size (maximum size) is equal to or larger than a threshold (step S103). If it is determined that the maximum size of the projected Gaussians is equal to or larger than a threshold (step S103; YES), processor 11 extracts the 3D Gaussian of interest as a target for division processing, which will be described later (step S104), and proceeds to step S105.

[0063] On the other hand, if it is determined that the maximum size of the projected Gaussians is not equal to or greater than the threshold (step S103; NO), the processor 11 skips step S104 and proceeds to step S105. That is, in this case, the 3D Gaussian of interest is not extracted as a target for division processing.

[0064] Processor 11 determines that the 3D Gaussian focused on in the processes of steps S102 to S104 has been determined, and determines whether or not any undetermined 3D Gaussian remains (step S105). If it is determined that any undetermined 3D Gaussian remains (step S105; YES), processor 11 returns the process to step S101.

[0065] On the other hand, if it is determined that no undetermined 3D Gaussians remain (step S105; NO), the processor 11 determines whether or not there are any 3D Gaussians extracted in step S104 (step S106).

[0066] If it is determined that no 3D Gaussians have been extracted (step S106; NO), processor 11 ends the process. On the other hand, if it is determined that some 3D Gaussians have been extracted (step S106; YES), processor 11 performs a division process on the extracted 3D Gaussians (step S200).

[0067] 12 is a flow diagram showing an example of the operation flow of the division processing. Processor 11 specifies the opacity of each of the 3D Gaussians extracted in step S104, and selects several 3D Gaussians with probabilities proportional to the opacity (step S201). Processor 11 performs sampling according to, for example, a multinomial distribution of the specified opacity, and selects the 3D Gaussians.

[0068] Next, the processor 11 divides each of the selected 3D Gaussians into multiple pieces so that the total volume of each Gaussian is maintained (step S202).

[0069] Furthermore, the processor 11 positions the newly obtained 3D Gaussians based on the positions of the original 3D Gaussians before division (step S203).

[0070] In this way, the information processing device 1 performs adaptive density control to split the initialized 3D Gaussian according to the feature points of the captured image, thereby improving spatial resolution and shape representation, and making it easier to represent the details of a 3D scene more accurately.

[0071] In particular, this information processing device 1 selects 3D Gaussians to be split according to the opacity of the 3D Gaussians optimized in the learning process, thereby pinpointing and improving the spatial resolution of areas with high opacity that have a high impact on the virtual image reconstructed (generated) on the screen. As a result, this information processing device 1 can generate high-resolution virtual images at high speed.

[0072] Furthermore, since the information processing device 1 does not replicate 3D Gaussians, it is possible to save the processing time required for replication and improve the rendering speed.

[0073] <Modification> The above is a description of the embodiment, but the contents of this embodiment can be modified as follows. In addition, the following modifications can be combined.

[0074] <1> In the above-described embodiment, the processor 11 is a CPU or a GPU, but may have other configurations. For example, the processor 11 may be or may include an FPGA (Field Programmable Gate Array). The processor 11 may also have an ASIC (Application Specific Integrated Circuit) or other programmable logic device. The information processing device 1 may also have multiple processors, multiple memories, and multiple interfaces. The information processing device 1 may be, for example, a mobile terminal such as a smartphone or a slate PC. The information processing device 1 may also be, for example, a virtual machine realized by dynamic cooperation of multiple computing resources on a cloud via a communication line such as the Internet.

[0075] <2> In the above-described embodiment, the information processing device 1 selects a 3D Gaussian with a probability according to the opacity and splits the selected 3D Gaussian so that the total volume of the selected 3D Gaussian is maintained, but the criteria for selecting the 3D Gaussian to be split are not limited to this.

[0076] FIG. 13 is a flow diagram showing an example of the operational flow of the division processing in the modified example. In the division processing of step S200 shown in FIG. 13, step S201a is executed instead of step S201 shown in FIG. 12. That is, the processor 11 selects a 3D Gaussian with a probability proportional to not only the opacity but also the maximum size of the projected Gaussian (step S201a). As a result, the information processing device 1 is more likely to divide a 3D Gaussian the higher the opacity, and is also more likely to divide a 3D Gaussian the larger the maximum size of the projected Gaussian. Therefore, the information processing device 1 can improve the spatial resolution of a portion that has a high opacity and therefore a high proportion of influence on the virtual image reconstructed (generated) on the screen, as well as improve the spatial resolution of a portion that occupies a large area on the screen when projected and therefore a high proportion of influence on the virtual image.

[0077] In other words, the division means 1153 realized by the processor 11 in this modified example is an example of a division means that divides a 3D Gaussian selected from the extracted 3D Gaussians with a probability corresponding to the opacity and the maximum size of the projected Gaussian into multiple pieces so that the total volume is maintained.

[0078] <3> In the above-described embodiment, the information processing device 1 only divides the 3D Gaussian and does not replicate it, but may replicate any of the 3D Gaussians when a predetermined condition is met.

[0079] Fig. 14 is a diagram showing an example of the configuration of the adaptive density control means 115 in a modified example. The adaptive density control means 115 shown in Fig. 14 differs from that shown in Fig. 10 in that it includes a calculation load determination means 1155 and a duplication means 1156.

[0080] The calculation load determining means 1155 is configured to monitor the calculation load of the information processing device 1. The calculation load determining means 1155 determines whether the load on the processor 11 or the calculation resources controlled by the processor 11 is below a predetermined level.

[0081] When the calculation load determination means 1155 determines that the calculation load is less than a predetermined level, it supplies the 3D Gaussians that were not extracted by the extraction means to the duplication means 1156. The duplication means 1156 duplicates the 3D Gaussians that satisfy predetermined conditions among the 3D Gaussians supplied from the calculation load determination means 1155.

[0082] For example, if the size (three-dimensional size) of a 3D Gaussian in the virtual space is smaller than a certain threshold, the duplication means 1156 duplicates the 3D Gaussian. This allows the information processing device 1 to improve the rendering accuracy over a wide range that is difficult to achieve by division, and to enhance the rough shape and low-frequency features when reproducing an object as a virtual object. Furthermore, since the information processing device 1 duplicates the 3D Gaussian only when sufficient computational resources are secured, it is less likely to interfere with the division process of the 3D Gaussian in adaptive density control, and is less likely to slow down the rendering speed.

[0083] The calculation load determining means 1155 and the duplicating means 1156 in this modified example are examples of duplicating means that, when the calculation load is below a predetermined level, duplicates one of a plurality of 3D Gaussians that satisfies a condition.

[0084] <4> In the above-described embodiment, the processor 11 of the information processing device 1 functions as a bundle adjustment means 112 that calculates camera information at the time the captured images supplied by the image acquisition means 111 were captured, but the camera information may be stored in advance in the memory 12 in association with the captured images.

[0085] Fig. 15 is a diagram showing an example of the functional configuration of information processing device 1 in a modified example. Camera information DB 122 stored in memory 12 stores camera information in advance for each image ID that identifies an image in image DB 121. In this case, processor 11 does not need to estimate camera information from feature points of a captured image. As shown in Fig. 15, processor 11 simply functions as camera information acquisition means 112a that reads out camera information associated with an image ID in camera information DB 122.

[0086] The image DB 121 may be integrated with the camera information DB 122. For example, when a camera photographs an object, it may store its own camera information as metadata of the photographed image. The camera information may be generated by a barometer, acceleration sensor, gyro sensor, GNSS, or the like attached to the camera when photographing an object. Furthermore, if there is a mobile device that moves the camera, the camera information may be generated using a movement amount measuring device attached to the mobile device. Furthermore, the camera information may be associated in advance with the position where the camera is attached, the camera's identification information, or the like.

[0087] <5> In the above-described embodiment, the information processing device 1 extracts only those 3D Gaussians whose maximum size when projected onto each screen is equal to or greater than a threshold value from among the 3D Gaussians that have undergone parameter learning, and selects some 3D Gaussians from the extracted 3D Gaussians as targets for division with a probability according to their opacity. However, the order of extraction and selection may be reversed.

[0088] 15 is configured such that the selection means 1152 is followed by the extraction means 1151, which is then followed by the division means 1153 and the arrangement means 1154. In this case, the selection means 1152 selects some 3D Gaussians from the learned 3D Gaussians with a probability according to the opacity.

[0089] Then, the extraction means 1151 extracts a 3D Gaussian whose maximum size of the projected Gaussian is equal to or greater than a threshold from the 3D Gaussians selected by the selection means 1152. The division means 1153 divides the 3D Gaussian extracted by the extraction means 1151 into multiple pieces so that the total volume is maintained.

[0090] Therefore, the selection means 1152 shown in FIG. 15 is an example of a selection means that selects a 3D Gaussian from a plurality of 3D Gaussians with a probability according to the opacity.

[0091] In addition, the extraction means 1151 and the division means 1153 shown in Figure 15 are examples of division means that divide, from among 3D Gaussians selected according to opacity, etc., 3D Gaussians whose maximum size when projected onto each screen is equal to or exceeds a threshold value, into multiple 3D Gaussians so that the total volume is maintained.

[0092] <6> In the above-described embodiment, the program executed by the processor 11 is conceived as a program that causes a computer to function as: an acquisition means that acquires multiple images of an object and camera information indicating the position, orientation, and shooting range of the camera that captured each of the multiple images, each associated with the multiple images; a calculation means that calculates projected Gaussians by projecting each of multiple 3D Gaussians placed in a virtual space onto a screen corresponding to each camera in the virtual space; an extraction means that extracts, from the multiple 3D Gaussians, those projected Gaussians whose maximum size when projected onto each screen is equal to or greater than a threshold; and a division means that selects from the extracted 3D Gaussians with a probability according to opacity and divides the selected 3D Gaussians into multiple 3D Gaussians so that the total volume is maintained.

[0093] <7> The method of causing the processor 11 to execute the program can also be conceived as an adaptive density control method for causing a computer to control the density of 3D Gaussians. That is, the adaptive density control method for 3D Gaussians includes the steps of: acquiring, by a computer, a plurality of images of an object and camera information indicating the position, orientation, and shooting range of a camera that captured each of the plurality of images, in association with each other; calculating projected Gaussians by projecting each of the plurality of 3D Gaussians arranged in a virtual space onto a screen corresponding to each camera in the virtual space; extracting, by the computer, from the plurality of 3D Gaussians, those projected onto the screen whose maximum size is equal to or greater than a threshold; and dividing, by the computer, a plurality of 3D Gaussians selected from the extracted 3D Gaussians with a probability according to opacity so as to maintain the total volume. [Explanation of symbols]

[0094] 1...information processing device, 11...processor, 111...image acquisition means, 112...bundle adjustment means, 1121...feature point extraction means, 1122...feature point matching means, 1123...verification means, 1124...estimation means, 112a...camera information acquisition means, 113...initialization means, 114...learning means, 1141...calculation means, 1142...error minimization means, 115...adaptive density control means, 1151...extraction means, 1152...selection means, 1153...division means, 1154...placement means, 1155...computational load determination means, 1156...replication means, 12...memory, 121...image DB, 122...camera information DB, 123...Gaussian DB, 13...interface.

Claims

1. Computer, an acquisition means for acquiring a plurality of images of an object and camera information indicating the position, orientation, and shooting range of a camera that captured each of the plurality of images in association with each other; a calculation means for calculating a projected Gaussian by projecting each of a plurality of 3D Gaussians arranged in a virtual space onto a screen corresponding to each of the cameras in the virtual space; an extraction means for extracting, from the plurality of 3D Gaussians, those projected onto the screens whose maximum size is equal to or greater than a threshold; a division means for dividing a 3D Gaussian selected from the extracted 3D Gaussians with a probability according to the opacity into a plurality of 3D Gaussians so that the total volume is maintained; A program to function as a

2. The splitting means splits a 3D Gaussian selected from the extracted 3D Gaussians with a probability corresponding to the opacity and the maximum size of the projected Gaussians into a plurality of 3D Gaussians so that the total volume is maintained. The program according to claim 1.

3. The computer a replicating means for replicating, when a calculation load is less than a predetermined level, one of the plurality of 3D Gaussians that satisfies a condition; The program according to claim 1 for causing the program to function as:

4. The computer an arrangement means for arranging each of the split 3D Gaussians based on the position of the original 3D Gaussian; The program according to claim 1 for causing the program to function as:

5. Computer, an acquisition means for acquiring a plurality of images of an object and camera information indicating the position, orientation, and shooting range of a camera that captured each of the plurality of images in association with each other; a calculation means for calculating a projected Gaussian by projecting each of a plurality of 3D Gaussians arranged in a virtual space onto a screen corresponding to each of the cameras in the virtual space; a selection means for selecting a 3D Gaussian from the plurality of 3D Gaussians with a probability according to opacity; a division means for dividing, from among the selected 3D Gaussians, a 3D Gaussian whose maximum size when projected onto each of the screens is equal to or larger than a threshold into a plurality of 3D Gaussians while maintaining the total volume; A program to function as a

6. an acquisition means for acquiring a plurality of images of an object and camera information indicating the position, orientation, and shooting range of a camera that captured each of the plurality of images in association with each other; a calculation means for calculating a projected Gaussian by projecting each of a plurality of 3D Gaussians arranged in a virtual space onto a screen corresponding to each of the cameras in the virtual space; an extraction means for extracting, from the plurality of 3D Gaussians, those projected onto the screens whose maximum size is equal to or greater than a threshold; a division means for dividing a 3D Gaussian selected from the extracted 3D Gaussians with a probability according to the opacity into a plurality of 3D Gaussians so that the total volume is maintained; An information processing device having the above.

7. a step in which a computer acquires a plurality of images of an object and camera information indicating the position, orientation, and shooting range of a camera that captured each of the plurality of images, in association with each other; A step in which a computer calculates projected Gaussians by projecting each of a plurality of 3D Gaussians arranged in a virtual space onto a screen in the virtual space corresponding to each of the cameras; A step in which a computer extracts, from the plurality of 3D Gaussians, Gaussians whose maximum size when projected onto each of the screens is equal to or greater than a threshold; A step in which a computer divides a 3D Gaussian selected from the extracted 3D Gaussians with a probability according to opacity into multiple pieces so that the total volume is maintained; 1. A method for adaptive density control of a 3D Gaussian, comprising:

Images