Three-dimensional space generation device, method, and program

Abstract

An embodiment of a three-dimensional space generation device according to the present invention comprises an image acquisition unit, an image classification unit, an image selection unit, and a camera parameter estimation unit. The image acquisition unit acquires an original image obtained by capturing, at a plurality of viewpoints, the region of an object for which a three-dimensional space model is to be created. The image classification unit classifies each input image constituting the original image. The image selection unit selects whether or not to adopt each of the input images on the basis of the classified results. The parameter estimation unit estimates a camera parameter on the basis of the input image adopted by the selection.

Description

Three-dimensional space generation device, method, and program

[0001] One aspect of the present invention relates to a three-dimensional space generation device, method, and program for generating a three-dimensional space based on an acquired image.

[0002] In recent years, there has been a demand for services that reconstruct three-dimensional space from video captured outdoors, allowing users to experience outdoor scenes while indoors by viewing the reconstructed outdoor scenery in three-dimensional space. To realize this service, technology for generating three-dimensional space from two-dimensional images is required.

[0003] One such technique is the Neural Radiance Fields (NeRF) technique (see, for example, Non-Patent Document 1). The NeRF technique creates a representation domain (a cubic grid) for representing a three-dimensional space, and a deep learning model learns the radiance and opacity at each grid in the representation domain from a two-dimensional image and the camera parameters of that image, thereby creating a radiance field called Neural Radiance Fields (NeRF). The technique then reconstructs the three-dimensional shape and color of an object or landscape based on the radiance field. In addition to NeRF, 3D Gaussian Splatting and Multi-View Stereo are also known as representative techniques for generating three-dimensional space.

[0004] B. Mildenhall, et al. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Proc. of European Conference on Computer Vision, 2020.

[0005] However, with existing NeRF techniques, for example, when capturing outdoor video of a location from which a three-dimensional space is to be reconstructed and acquiring a two-dimensional image, various factors can cause unnecessary information other than the three-dimensional information to be reconstructed to be mixed into the two-dimensional image. For example, the photographer may be captured in the image due to careless shooting, or an unwanted moving object may be mixed in due to shooting in a crowded area. Furthermore, when shooting in a location with heavy traffic or foot traffic or a location with many structures, the background or building area necessary for reconstruction may be obstructed by obstacles such as cars, people, or structures that are not necessary for reconstruction. In these cases, existing NeRF techniques require a long time to estimate camera parameters from a two-dimensional image. In addition, when an area is obstructed by unnecessary structures, the three-dimensional space cannot be correctly reconstructed.

[0006] The present invention has been made in consideration of the above circumstances, and provides a technology that can quickly estimate camera parameters and improve the quality of the generated three-dimensional space even when unnecessary information is mixed in the captured image.

[0007] In order to solve the above problems, one aspect of a three-dimensional space generation device according to the present invention includes an image acquisition unit, an image classification unit, an image selection unit, and a camera parameter estimation unit. The image acquisition unit acquires original images of a target area for creating a three-dimensional space model, photographed from multiple viewpoints. The image classification unit classifies each input image that constitutes the original image. The image selection unit selects whether to accept each input image based on the classification results. The parameter estimation unit estimates camera parameters based on the input image that is accepted by the selection.

[0008] According to one aspect of the present invention, by classifying each input image that constitutes a captured original image and rejecting input images that contain unnecessary information, it is possible to reduce the number of input images to be adopted and improve their quality. As a result, it is possible to quickly estimate camera parameters based on the adopted input images, thereby improving the quality of the generated three-dimensional space.

[0009] In other words, according to one aspect of the present invention, even if unnecessary information is mixed in the captured image, camera parameters can be estimated quickly, thereby improving the quality of the generated three-dimensional space.

[0010] FIG. 1 is a block diagram showing an example of the configuration of a three-dimensional space generation device according to a first embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of a control unit shown in FIG. 1. FIG. 3 is a flowchart for explaining an example of operation in the first embodiment. FIG. 4 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 5 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 6 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 7 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 8 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 9 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 10 is a diagram for explaining a specific example and a comparative example of the first embodiment. FIG. 11 is a block diagram showing an example of the configuration of a control unit in a second embodiment. FIG. 12 is a flowchart for explaining an example of operation in the second embodiment. FIG. 13 is a block diagram showing an example of the configuration of a control unit in a third embodiment. FIG. 14 is a flowchart for explaining an example of operation in the third embodiment.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, a three-dimensional space generation device may be appropriately interpreted as a three-dimensional space reconstruction device. Similarly, a new viewpoint image may be appropriately interpreted as a free viewpoint image. In other words, terms with the same meaning may be appropriately interpreted as they are.

[0012] 1 is a block diagram showing an example of the configuration of a three-dimensional space generation device according to a first embodiment of the present invention. This three-dimensional space generation device 1 is formed, for example, by a personal computer. Note that the three-dimensional space generation device 1 may also be formed by a server computer installed on the web or on the cloud.

[0013] The three-dimensional space generating device 1 comprises a control unit 10 using a hardware processor such as a central processing unit (CPU), and a storage unit having a program storage unit 20 and a data storage unit 30, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 40 are connected to this control unit 10 via a bus 50.

[0014] A camera 2, an input device 3, and a display device 4 are connected to the input / output I / F unit 40. The camera 2 captures an image of a location including the target space to be reconstructed, and transmits the captured image data to the three-dimensional space generation device 1. Note that multiple cameras 2 may be placed in different locations. The image data may be acquired from an external storage device, or may be acquired via a network from another terminal device or a server computer on the web or cloud.

[0015] The input device 3 consists of, for example, a keyboard and a mouse, and is used to input information specifying the target area that the user or system administrator wishes to reconstruct, commands to instruct the three-dimensional space generation device 1 to perform various operations, and the like.

[0016] The display device 4 is used to display information representing various operating states of the three-dimensional space generation device 1 and information necessary for input operations, as well as to display new viewpoint images, which are three-dimensional space images reconstructed from the constructed three-dimensional space model.

[0017] The input / output I / F unit 40 may include a communication interface, which allows the video data transmitted from the camera 2 to be acquired via a network even when the camera 2 is located far away.

[0018] The program storage unit 20 is configured, for example, by combining a nonvolatile memory such as a solid-state drive (SSD) that can be written and read at any time as a storage medium with a nonvolatile memory such as a read-only memory (ROM). It stores middleware such as an operating system (OS) as well as application programs necessary for executing various control operations according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program." The program may be installed on the computer in advance from a network or a non-transitory computer-readable storage medium, or may be pre-recorded on the computer. In either case, the program is executed by the processor to cause the computer to function as the three-dimensional space generation device 1.

[0019] The data storage unit 30 is, for example, a combination of a nonvolatile memory such as an SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium.

[0020] 2, the control unit 10 includes an image acquisition unit 11, an image classification unit 12, an image selection unit 13, and a camera parameter estimation unit 14. The control unit 10 is not limited to the processes of the above units 11 to 14, and can execute any process in response to an operation by an operator.

[0021] Each of the above units 11 to 14 is realized by causing a hardware processor of the control unit 10 to execute an application program stored in the program storage unit 20. Note that some or all of the above units 11 to 14 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0022] The image acquisition unit 11 acquires original images of a target area for generating a three-dimensional space model, taken from a plurality of viewpoints.

[0023] The image classification unit 12 classifies each input image that constitutes the original image. For example, the image classification unit 12 can use a Vision Transformer (ViT) and a Convolutional Neural Network (CNN) for image classification. ViT can extract global features and is thought to be able to reduce excessively redundant images and reconstruct a 3D scene with an appropriate number of images. ViT is described in, for example, References 1 and 2. CNN can extract local features and is thought to be able to reduce redundant images based on the local features.

[0024] The image selection unit 13 selects whether to accept each input image based on the classification result.

[0025] The camera parameter estimation unit 14 estimates camera parameters based on the input image adopted by this selection. Here, the camera parameters include internal parameters related to lens characteristics and external parameters related to a coordinate system, such as the viewpoint position. However, the camera parameters in this embodiment mainly relate to external parameters. The external parameters represent the difference between the positions of two viewpoints, and include, for example, five parameters (x, y, z, θ, φ) that represent the viewpoint position and orientation (posture). A specific example of an external parameter is a 3×4 matrix or a 4×4 matrix, generally referred to as a camera transformation matrix. In the case of a 4×4 matrix, the camera transformation matrix consists of a 3×3 rotation matrix and a 3×1 translation matrix, with the last row of the 4×4 matrix being

[0001] . In this case, the rotation matrix represents the orientation of the camera (viewpoint), and the translation matrix represents the camera position.

[0026] As a method for estimating camera parameters, for example, Structure from Motion (SfM) can be used. SfM is described, for example, in Reference 3. SfM is a technology for estimating the structure of a 3D scene from a 2D image, simultaneously estimating the camera motion and the 3D structure of the scene. COLMAP's SfM pipeline consists of three main steps: feature extraction, feature matching, and sparse reconstruction. Feature extraction involves extracting feature points from an image using Scale Invariant Feature Transform (SIFT). SITF is described, for example, in Reference 4. Feature matching involves matching feature points. Sparse reconstruction involves estimating camera parameters and 3D points. These processes tend to take a long time, especially when there are a large number of images, because the number of matching feature points increases. In contrast, in this embodiment, by eliminating unnecessary images through image selection, camera parameters can be efficiently estimated even when using methods such as SfM, which require long processing times.

[0027] Next, an example of the operation of the three-dimensional space generating device configured as described above will be described using the flowchart in FIG. 3 and the schematic diagrams in FIGS. 4 to 9. The following description will be given taking an example of capturing and reconstructing a wide-area 3D scene within a limited time frame, from the perspective of being able to reconstruct a 3D scene of an outdoor event. It should be noted that at outdoor event venues, it may be desirable to capture a wide area of ​​the event venue environment in a short period of time before the event begins and recreate the three-dimensional space. For example, bicycle road race venues have extensive courses spanning several kilometers, so they require capture within a short period of time. Because it is necessary to capture such a wide area in a short period of time, multiple bicycles are photographed running side by side. The following will explain each step in turn.

[0028] (1) Acquisition of Original Images In step ST10, the image acquisition unit 11 acquires original images of a target area for generating a three-dimensional space model, taken from multiple viewpoints. This step ST10 includes steps ST11 to ST13.

[0029] That is, the image acquisition unit 11 reads each input image 6i constituting the original image 6 as shown in FIG. 4 (step ST11).

[0030] After step ST11, the image acquisition unit 11 waits for a readable input image 6i (step ST12) and reads the newly received input image 6i. For example, as shown in FIG. 5, the input image 6i is transmitted from two cameras 2 mounted on frames 8 fixed to the carriers 7 of each of the bicycles 5A, 5B, and 5C operated by the rider T. Specifically, as shown in FIG. 6, the input images 6i are transmitted to the three-dimensional space generating device 1 from a total of six cameras 2 mounted on three bicycles 5A, 5B, and 5C traveling side by side in three lanes L1, L2, and L3. The image acquisition unit 11 receives the input image 6i via the input / output I / F unit 40 and reads the received input image 6i into the data storage unit 30. Additionally, the two cameras 2 for each of the bicycles 5A, 5B, and 5C are installed, for example, 200 cm above the ground and 60 cm apart from each other. The format of the original images 6 (video) captured by the cameras 2 mounted on the bicycles 5A, 5B, and 5C may be a standard format (perspective format), equirectangular format (equirectangular projection) captured by a 360-degree camera, or other formats. Images captured by a 360-degree camera can be used as is, or images captured by a 360-degree camera converted into a standard perspective format can be used. The six cameras 2 in this embodiment use GoPRO MAX (registered trademark), a 360-degree camera. This allows a single bicycle to capture a wide area from multiple viewpoints. Furthermore, by having the three bicycles 5A-5C travel side by side, it is possible to capture images in a short time, capture images from different viewpoints, and reconstruct a wide-area 3D scene by combining images from multiple viewpoints at some distance.

[0031] After step ST12, the image acquisition unit 11 determines whether or not the process is complete, depending on whether or not all input images 6i have been read (step ST13). If the result of this determination is "no," the processes of steps ST12 to ST13 are repeated. On the other hand, if the result of the determination in step ST13 is "complete," step ST10 is completed and the process proceeds to step ST20.

[0032] (2) Classification of Input Images After step ST10, the image classification unit 12 classifies (step ST20) each of the input images 6i that make up the original image 6. For example, as shown in Figures 7 and 8, a scene visible from bicycles 5A, 5B, and 5C traveling side by side includes an area 9 that is visible from all bicycles 5A, 5B, and 5C, and an area 9c that is not visible from some of the bicycles 5A and 5B.

[0033] In the subsequent camera parameter estimation, corresponding points are detected by capturing the same area in multiple input images 6i. Therefore, it is desirable that the same object be captured in multiple input images 6i. In FIG. 7, the shooting direction di indicates the area 9 visible from each of the bicycles 5A, 5B, and 5C. Since multiple input images 6i captured under such circumstances capture the same area 9, this information is useful for camera parameter estimation. On the other hand, in FIG. 8, the shooting direction di indicates the area 9c that is not visible from the front bicycles 5A and 5B due to the inclusion (reflection) of the rear bicycles 5B and 5C, making it difficult to determine correspondence between the multiple input images 6i. Furthermore, since the bicycles 5B and 5C are captured, it may be difficult to estimate the camera parameters necessary for reconstructing the 3D scene and perform the NeRF processing in the subsequent stage. Input images 6i containing such unnecessary information are unnecessary for camera parameter estimation, and should therefore be deleted. Therefore, the image classification unit 12 performs image classification on the input images 6i to determine whether unnecessary information is present.

[0034] In order to classify the input images 6i, the input images 6i must be labeled in advance. Here, as shown in FIG. 9 , the labeling may be performed using a combination of multiple attributes, such as a bicycle ID indicating which bicycle it is, a shooting direction di indicating the direction of the camera image, and the time (time zone) at which the image was taken. This combination may further include an attribute such as a camera number. In other words, a combination of multiple attributes, such as the bicycle ID, camera number, shooting direction, and time, constitutes one label. Each label is a different label if at least one of the multiple attributes is different. Each label corresponds to a different viewpoint.

[0035] This labeling results in a dataset in which the input images 6i are used as input data and feature quantities (similarity) corresponding to the labels assigned to the input images 6i are used as output data. An image classifier is trained in advance using this dataset. The trained image classifier outputs feature quantities (similarity) corresponding to each label based on the input images 6i. For example, multiple input images 6i captured from similar viewpoints are images with a high degree of overlap, and each image has a maximum similarity to its own label and a similarity to the other label that is also close to the maximum. Therefore, if the classification result of an input image 6i contains many high similarities, it can be determined that the input image 6i is an image with a high degree of overlap. On the other hand, if the classification result of an input image 6i does not contain many high similarities, it can be determined that the input image 6i does not overlap much with other input images 6i and that unnecessary information may have been mixed in. Note that input images 6i with a high degree of overlap are desirable because they contribute to estimating camera parameters. On the other hand, it is desirable to reduce the number of input images 6i that do not overlap much, since they contribute to the estimation of camera parameters. By using such a trained image classifier, the image classification unit 12 classifies the input images 6i based on the similarity with each label.

[0036] (3) Selecting whether to accept or reject the input image After step ST20, the image selector 13 selects whether to accept or reject each input image based on the classification results (step ST30). This step ST30 includes steps ST31 to ST34.

[0037] That is, the image selection unit 13 reads each input image 6i together with the classification result (step ST31).

[0038] After step ST31, the image selection unit 13 waits for a readable input image 6i (step ST32), and reads the newly received input image 6i and its classification result.

[0039] After step ST32, the image selection unit 13 selects whether to accept or reject each input image 6i corresponding to the read classification result (step ST33). That is, the image selection unit 13 determines which labels the classified input images 6i are similar to, retains input images 6i with many similar labels, and discards input images 6i with few similar labels. Specifically, for example, the image selection unit 13 may sort the input images 6i in order of label similarity and then select whether to accept or reject each input image 6i based on whether it satisfies a first criterion. The first criterion may be, for example, having the top eight similarities. In this case, the image selection unit 13 retains input images 6i with the top eight similarities and discards the remaining input images 6i. In either case, the image selection unit 13 selects unnecessary input images 6i to be thinned out based on the image classification, thereby eliminating unnecessary images.

[0040] After step ST33, the image selection unit 13 determines whether or not the process is complete, depending on whether all input images 6i have been read (step ST34). If the result of this determination is "no," the processes of steps ST32 to ST34 are repeated. On the other hand, if the result of the determination in step ST34 is "complete," step ST30 is completed and the process proceeds to step ST40.

[0041] (4) Estimation of Camera Parameters The camera parameter estimation unit 14 estimates camera parameters based on the input image 6i adopted by the selection (step ST40). That is, after unnecessary images have been removed, the camera parameter estimation unit 14 estimates camera parameters based on the remaining high-quality input images 6i. Then, step ST40 is terminated.

[0042] After step ST40 is completed, the three-dimensional space generation device 1 may terminate processing. In other words, the use of the three-dimensional space generation device 1 is not limited to typical cases such as generating a three-dimensional space model or generating a new viewpoint image, but may also include cases where camera parameters are simply desired to be acquired. An example of the latter case may be a case where the user wishes to confirm the location of a certain input image 6i. Thus, depending on the use, the three-dimensional space generation device 1 may terminate processing by estimating camera parameters without necessarily proceeding to image generation. Conversely, depending on the use, the three-dimensional space generation device 1 may generate a three-dimensional space model based on the estimated camera parameters and the adopted input image. Similarly, depending on the use, the three-dimensional space generation device 1 may generate a new viewpoint image based on the generated three-dimensional space model and input image generation parameters.

[0043] (5) Specific Examples and Comparative Examples Next, specific examples and comparative examples in which the three-dimensional space generation device 1 generates new viewpoint images will be described. FIG. 10 is a diagram for explaining specific examples and comparative examples of the first embodiment. In the first embodiment, three bicycles 5A, 5B, and 5C were each equipped with two cameras 2, and images were captured using six cameras 2. The cameras 2 used were GoPro MAX (registered trademark), a 360-degree camera. Four input images 6i were extracted from each camera 2, shifted horizontally by 60 degrees from the equirectangular image. The four input images 6i were adjusted to minimize the view angle of the rider T. The input images 6i were extracted at a frame rate of 6 fps, and 5 seconds' worth of images were extracted. In other words, 3 bicycles x 2 bicycles x 4 images x 5 seconds x 6 fps = 720 input images 6i were extracted.

[0044] For camera parameter estimation and 3D scene reconstruction, images were generated using COLMAP and the nerfacto 3D scene reconstruction method on NeRFStudio, which is described, for example, in Reference 5. The nerfacto model was created to serve as a default for real-world data capture of static scenes, and is an implementation created by combining many publicly available methods that have been found to work well on real-world data.

[0045] Here, six types of methods were prepared as specific examples (#1 to #4) of this embodiment and comparative examples (Δ1, Δ2) thereof.

[0046] The comparative example (Δ1) is a method that uses all the data, and the comparative example (Δ2) is a method that evenly thins out the data by half.

[0047] On the other hand, specific example (#1) is a method of thinning the entire data in half using ViT. Specific example (#2) is a method of thinning the entire data in half using CNN. Specific example (#3) is a method of thinning the data to an equal density using ViT. Specific example (#4) is a method of thinning the data to an equal density using CNN.

[0048] In the specific examples (#1) to (#4), after sorting in descending order of labels with the highest similarity, the input images 6i are thinned in half, starting with the eighth most similar. That is, in the specific examples (#1) to (#4), the input images 6i with the highest eight similarities are retained, and the remaining input images 6i are thinned out, thereby thinning the entire image in half. Furthermore, the techniques (#3) and (#4) for thinning out images at equal densities thin out the images in the same manner as described above so that the density per unit time (per second) is equal.

[0049] The experimental results for the six methods are shown in Fig. 10. In Fig. 10, PSNR is an abbreviation for Peak Signal to Noise Ratio, SSIM is an abbreviation for Structural Similarity, and LPIPS is an abbreviation for Learned Perceptual Image Patch Similarity. For PSNR and SSIM, lower values ​​indicate better results. For LPIPS, higher values ​​indicate better results.

[0050] The PSNR was best for the comparative example (△1), which used all the data (new viewpoint image), but the processing time was the longest. The processing time was shortest for the comparative example (△2), which simply thinned out the entire image by half.

[0051] For both SSIM and LPIPS, the best results were obtained when the CNN of the specific example (#4) was used and the entire data was thinned out to half at equal density.

[0052] As described above, according to the first embodiment, the image acquisition unit 11 acquires original images 6 captured from multiple viewpoints of a target area for creating a three-dimensional space model. The image classification unit 12 classifies each input image 6i constituting the original image 6. The image selection unit 13 selects whether to accept each input image 6i based on the classification results. The camera parameter estimation unit 14 estimates camera parameters based on the selected input images 6i. In this manner, by classifying each input image 6i constituting the captured original image 6 and rejecting input images 6i containing unnecessary information, it is possible to reduce the number of input images 6i to be adopted and improve their quality. Therefore, even if unnecessary information is mixed in the captured images, camera parameters can be estimated quickly. Furthermore, when the three-dimensional space generation device 1 generates a three-dimensional space model, it is possible to improve the quality of the generated three-dimensional space.

[0053] 11 is a block diagram showing an example of the configuration of a control unit in a second embodiment, and parts that are substantially the same as those in Fig. 1 and Fig. 2 are given the same reference numerals and detailed explanations thereof are omitted, and only different parts will be described here. Duplicate explanations will also be omitted for each of the following embodiments.

[0054] In addition to the above-described units 11 to 14, the control unit 10 also includes a three-dimensional space generation unit 15 and an image generation unit 16. As described above, each of these units 11 to 16 is realized by causing a hardware processor in the control unit 10 to execute an application program stored in the program storage unit 20. Similarly, some or all of the above-described units 11 to 16 may be realized using hardware such as an LSI or an ASIC.

[0055] The three-dimensional space generation unit 15 generates a three-dimensional space model based on the estimated camera parameters and the adopted input image 6i.

[0056] The image generator 16 generates a new viewpoint image based on the generated three-dimensional space model and the input image generation parameters. The image generation parameters include, for example, the size and viewpoint position of the new viewpoint image to be generated.

[0057] Next, an example of the operation of the three-dimensional space generating device configured as above will be described with reference to the flowchart of FIG.

[0058] Now, it is assumed that steps ST10 to ST40 have been executed in the same manner as described above.

[0059] After step ST40, in step ST60, the three-dimensional space generation unit 15 generates a three-dimensional space model based on the estimated camera parameters and the adopted input image 6i.

[0060] After step ST60, in step ST70, the image generation unit 16 generates a new viewpoint image based on the generated three-dimensional space model and the input image generation parameters. This step ST70 includes steps ST71 to ST75.

[0061] That is, the image generating unit 16 reads the image generating parameters together with the three-dimensional space model (step ST71).

[0062] After step ST71, the image generating unit 16 waits for image generating parameters that can be read (step ST72), and reads the newly received image generating parameters.

[0063] After step ST72, the image generating section 16 generates a new viewpoint image based on the three-dimensional space model and the read image generation parameters (step ST73).

[0064] After step ST73, the image generation unit 16 outputs the new viewpoint image to the display device 4 via the input / output I / F unit 40 (step ST74). The display device 4 displays the output new viewpoint image.

[0065] After step ST74, the image generating unit 16 determines whether or not the process is finished, depending on whether or not all image generation parameters have been read (step ST75). If the result of this determination is "no," the processes of steps ST72 to ST75 are repeated. On the other hand, if the result of the determination in step ST75 is "finish," step ST70 is finished.

[0066] As described above, according to the second embodiment, the three-dimensional space generation unit 15 generates a three-dimensional space model based on the estimated camera parameters and the adopted input image 6i. The image generation unit 16 generates a new viewpoint image based on the generated three-dimensional space model and the input image generation parameters. Therefore, in addition to the effects described above, it is possible to generate a three-dimensional space model and a new viewpoint image within the three-dimensional space generation device 1.

[0067] 13 is a block diagram showing an example of the configuration of a control unit according to a third embodiment. This control unit 10 has the configuration shown in FIG. 11 in which a second processing by an image selection unit 13 is added.

[0068] That is, in addition to the above-mentioned functions, the image selection unit 13 executes a process of selecting whether to accept or reject each input image depending on whether or not the image satisfies a second criterion that is higher than the first criterion used for the above selection.

[0069] Accordingly, the three-dimensional space generation unit 15 generates a three-dimensional space model based on the input image that satisfies the second criterion.

[0070] The image generation unit 16 generates a new viewpoint image based on the generated three-dimensional space model and the input image generation parameters.

[0071] Next, an example of the operation of the three-dimensional space generating device configured as above will be described with reference to the flowchart of FIG.

[0072] Now, it is assumed that steps ST10 to ST40 have been executed in the same manner as described above.

[0073] After step ST40, in step ST50, the image selection unit 13 selects whether to accept each input image 6i depending on whether it satisfies a second criterion that is higher than the first criterion used for the selection in step ST30. For example, the image selection unit 13 sorts the input images 6i in descending order of similarity, and then selects whether to accept each input image 6i depending on whether it has one of the top four similarities higher than the first criterion. In this case, the image selection unit 13 retains the input images 6i with the top four similarities and discards the remaining input images 6i. Furthermore, when the image selection unit 13 has performed selection based on the second criterion for all input images 6i adopted in step ST30, it ends step ST50.

[0074] After step ST50, steps ST60 to ST70 are executed in the same manner as described above.

[0075] That is, in step ST60, the three-dimensional space generation unit 15 generates a three-dimensional space model based on the input image that satisfies the second criterion.

[0076] After step ST60, in step ST70, the image generating section 16 generates a new viewpoint image based on the generated three-dimensional space model and the input image generation parameters.

[0077] As described above, according to the third embodiment, the image selection unit 13 executes a process of selecting whether to accept or reject each input image depending on whether the input image satisfies a second criterion that is higher than the first criterion used in the previous selection. The three-dimensional space generation unit 15 generates a three-dimensional space model based on the input image that satisfies the second criterion. The image generation unit 16 generates a new viewpoint image based on the generated three-dimensional space model and input image generation parameters. Therefore, in addition to the effects described above, the three-dimensional space model is generated using the input image 6i that is of higher quality than the input image 6i used to estimate the camera parameters, thereby further improving the quality of the three-dimensional space.

[0078] Additionally, among the input images 6i used when estimating camera parameters, there are some input images 6i that should not be used when generating a three-dimensional space model. For example, when estimating camera parameters, the presence of the input image 6i may result in many images being superimposed, which may facilitate the estimation of other images. In this case, it is preferable to retain the input image 6i when estimating the camera parameters. On the other hand, when generating a three-dimensional space model, using the input image 6i may result in unnecessary blurring or the inclusion of unnecessary images, so it may be better not to use the input image 6i. Therefore, in order to properly use this type of input image, in the third embodiment, a stricter second criterion is used to determine whether only high-quality input images are used when generating a three-dimensional space model. In other words, by using input images 6i that satisfy the first criterion for camera parameter estimation and input images 6i that satisfy the second criterion for three-dimensional space generation, the selection of images to be used for each process can be optimized.

[0079] <Other Embodiments> The functional configuration of the three-dimensional space generation device 1, its processing procedure and processing content, the type of neural network and its algorithm used in a machine learning model such as an image classifier, the type and use of images to be generated, and the like can be modified and implemented without departing from the spirit and scope of the present invention. For example, the method of generating a three-dimensional space is not limited to NeRF, and any method such as 3D Gaussian Splatting or Multi-View Stereo can be used. 3D Gaussian Splatting is described, for example, in Reference 6. Multi-View Stereo is described, for example, in Reference 7.

[0080] Furthermore, for example, the number of bicycles 5A to 5C used to collect the original images and the number of cameras 2 for each bicycle may be changed as appropriate. In addition, any moving object such as a motorcycle or an automobile may be used instead of a bicycle.

[0081] Furthermore, the original images to be thinned out do not necessarily have to be images extracted at equal intervals in time series, but may be images randomly sampled from a moving image, for example.

[0082] In addition, additional processing may be performed, such as re-sampling images from the video that correspond to the deleted and reduced labels, re-classifying the images, and determining whether or not to accept them. This can avoid inconveniences such as deleting input images (input images that are originally highly similar) taken from a similar direction because unnecessary information was captured for just a moment, resulting in a decrease in the quality of the 3D space model based on the adopted input images. Such additional processing may be performed, for example, when the number of deleted images exceeds a threshold, or when the similarity of the eighth image from the top is below a threshold.

[0083] Furthermore, for example, the elements included in the image classification label are not limited to those described above. For example, instead of the bicycle ID attribute included in the label, the image classification unit 12 may include a camera set ID that indicates a set of cameras 2 mounted on the same bicycle or GPS (Global Positioning System) subdivided zone information corresponding to the bicycle location as an attribute. The GPS zone information is information that indicates a zone that includes the bicycle position (latitude and longitude) obtained by a GPS device attached to the bicycle. In other words, the above embodiment can be implemented in the same way and similar effects can be obtained even if the attributes included in the label are appropriately modified.

[0084] For example, in the third embodiment, the eighth similarity from the top is used as the first criterion, and the fourth similarity from the top is used as the second criterion. However, this is not limiting. For example, the Nth similarity from the top may be used as the first criterion, and the Mth similarity from the top may be used as the second criterion, and the first and / or second criterion may be modified within a range that satisfies the relationship M<N (e.g., 4<8). In other words, the first and / or second criterion may be modified as appropriate as long as the top M input images 6i having relatively high similarities among the top N input images 6i used for camera parameter estimation are used for spatial image generation. Even with such a modification, the above-described embodiment can be implemented in the same manner and similar effects can be obtained.

[0085] For example, in the third embodiment, the selection of images that satisfy the second criterion is performed between the camera parameter estimation and the three-dimensional space generation. However, this is not limiting. For example, the selection of images that satisfy the second criterion may be performed between the image classification and the camera parameter estimation. In this way, even if the timing of the selection of images that satisfy the second criterion is appropriately changed, the above embodiment can be implemented in the same way and the same effect can be obtained.

[0086] Furthermore, for example, the selection of images that satisfy the second criterion may be performed by narrowing down each input image 6 i that satisfies the first criterion, or may be performed for each of the input images 6 i that have undergone image classification. In this way, even if the input images 6 i that are the subject of the selection of images that satisfy the second criterion are appropriately modified, the same effect can be obtained by implementing the above embodiment in the same way. However, from the perspective of performing image selection quickly, it is preferable to select images that satisfy the second criterion by narrowing down each input image 6 i that satisfies the first criterion.

[0087] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0088] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0089] (References) (1) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. (2) Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? arXiv preprint arXiv:2108.08810, 2022. (3) Johannes Lutz Sch¨onberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. (4) David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, Vol. 60, No. 2, pp. 91-110, 2004.(5)Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH ’23, 2023. (6)Kerbl, Bernhard and Kopanas, Georgios and Leimk¨uhler, Thomas and Drettakis, George. “3D Gaussian Splatting for Real-Time Radiance Field Rendering”, ACM Transactions on Graphics, Vol. 42, No. 4, July 2023. (7)Sch¨onberger, Johannes Lutz and Zheng, Enliang and Pollefeys, Marc and Frahm, Jan-Michael. “Pixelwise view selection for unstructured multi-view stereo”, European Conference on Computer Vision, Springer, pp. 501-518, 2016.

[0090] DESCRIPTION OF SYMBOLS 1...3D space generation device 2...Camera 3...Input device 4...Display device 5A, 5B, 5C...Bicycle 6...Original image 6i...Input image 7...Carrier 8...Frame 9, 9c...Area 10...Control unit 11...Image acquisition unit 12...Image classification unit 13...Image selection unit 14...Camera parameter estimation unit 15...3D space generation unit 16...Image generation unit 20...Program storage unit 30...Data storage unit 40...Input / output I / F unit 50...Bus

Claims

1. A three-dimensional space generation device comprising: an image acquisition unit that acquires original images taken from multiple viewpoints of a target area for generating a three-dimensional space model; an image classification unit that classifies each input image that constitutes said original image; an image selection unit that selects whether to adopt each input image based on the results of said classification; and a camera parameter estimation unit that estimates camera parameters based on the input image adopted by said selection.

2. A three-dimensional space generation device as described in claim 1, further comprising: a three-dimensional space generation unit; and an image generation unit, wherein the image selection unit selects whether to accept each input image depending on whether it satisfies a second criterion higher than the first criterion used for the selection; the three-dimensional space generation unit generates the three-dimensional space model based on the input image that satisfies the second criterion; and the image generation unit generates a new viewpoint image based on the generated three-dimensional space model and input image generation parameters.

3. A method executed by a three-dimensional space generation device, comprising: acquiring original images of a target area for creating a three-dimensional space model, photographed from multiple viewpoints; classifying each of the input images that make up the original images; selecting whether to adopt each of the input images based on the results of the classification; and estimating camera parameters based on the input image adopted by the selection.

4. A program for causing a computer to function as the three-dimensional space generating device according to claim 1 or 2.

Images