Three-dimensional space generation device, method, and program

Abstract

This three-dimensional space generation device includes: an image acquisition unit for acquiring, with respect to a target for which a three-dimensional space model is to be generated, original images captured under at least two conditions; an image editing unit that edits the original images to generate edited images; and a three-dimensional space generation unit for generating the three-dimensional space model on the basis of the original images and the edited images.

Description

Three-dimensional space generation device, method, and program

[0001] The present invention relates to an apparatus, a method, and a program for generating a three-dimensional space based on collected images.

[0002] In recent years, there has been a demand for services that reconstruct three-dimensional space from video captured outdoors, allowing users to experience outdoor scenes while indoors by viewing the reconstructed outdoor scenery in three-dimensional space. To realize this type of service, technology is needed to reconstruct three-dimensional space from two-dimensional images.

[0003] Regarding methods for generating three-dimensional space, as in Non-Patent Document 3, a method is known in which a representation region (a cubic grid) for representing the three-dimensional space is generated, and a deep learning model learns the radiance and opacity at each grid in the representation region from a two-dimensional image and the camera parameters of that image, thereby generating a radiance field called Neural Radiance Fields (NeRF), and reconstructing the three-dimensional shape and color of an object or landscape based on the radiance field.

[0004] K. He, et al. Mask R-CNN. Proc. of Computer Vision and Pattern Recognition, 2017. Z. Li, et al. DynIBaR: Neural Dynamic Image-Based Rendering. Proc. of Computer Vision and Pattern Recognition, 2023. B. Mildenhall, et al. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Proc. of European Conference on Computer Vision, 2020.A. Mirzaei, et al. SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields. Proc. of Computer Vision and Pattern Recognition, 2023.F. Reda, et al. FILM: Frame Interpolation for Large Motion. Proc. of European Conference on Computer Vision, 2022.

[0005] However, in the method of Non-Patent Document 3, when applying 3D space reconstruction technology to services such as NeRF, for example, when considering reconstructing a 3D space of a specific outdoor location, outdoor video is captured to obtain a 2D image. However, various factors can cause information other than the 3D information to be reconstructed to be mixed into the 2D image. For example, the photographer may enter the image, or in a congested area, unwanted moving objects may be mixed in. Furthermore, in areas with heavy traffic or foot traffic or many structures, the background or buildings necessary for reconstruction may not be captured because they are blocked by cars, people, structures, etc. This occluded area presents a problem in that 3D space reconstruction technology cannot correctly reconstruct the 3D space.

[0006] Furthermore, the method of Non-Patent Document 4 performs interpolation by generating results that are considered to be appropriate for the landscape or building to be reconstructed, so while sufficient quality can be obtained when the area to be interpolated is small, there is a problem in that when the area to be interpolated is large, the interpolation process results in an effect that is not representative of the actual landscape or building.

[0007] The present invention has been made in consideration of the above circumstances, and aims to provide a three-dimensional space generation device, method, and program that can improve the quality of the reconstructed three-dimensional space when reconstructing the three-dimensional space based on collected images.

[0008] In order to achieve the above object, a three-dimensional space generation device according to a first aspect of the present invention includes an image acquisition unit that acquires original images taken under at least two conditions of an object for which a three-dimensional space model is to be generated, an image editing unit that edits the original images to generate edited images, and a three-dimensional space generation unit that generates a three-dimensional space model based on the original images and edited images.

[0009] According to the three-dimensional space generation device, method, and program of the present invention, when a three-dimensional space is reconstructed based on collected images, it is possible to improve the quality of the reconstructed three-dimensional space.

[0010] FIG. 1 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a first embodiment is applied. FIG. 2 is a flowchart showing an example of the operation of a three-dimensional space generation device to which a three-dimensional space generation method according to the first embodiment is applied. FIG. 3 is a perspective view showing acquisition of an input image using a camera mounted on a bicycle. FIG. 4 is a top view showing acquisition of an input image using a camera mounted on a bicycle. FIG. 5 is a diagram showing an example of an image captured by a camera. FIG. 6 is a diagram showing an example of a mask image. FIG. 7 is a diagram showing an image after mask processing. FIG. 8 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a second embodiment is applied. FIG. 9 is a flowchart showing an example of the operation of a three-dimensional space generation device to which a three-dimensional space generation method according to the second embodiment is applied. FIG. 10 shows two images captured under different color conditions. FIG. 11 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a third embodiment is applied. FIG. 12 is a flowchart showing an example of the operation of a three-dimensional space generation device to which a three-dimensional space generation method according to the third embodiment is applied. FIG. 13 is a diagram showing a crowd scene and its occluded area captured under different conditions. FIG. 14 is a diagram illustrating similar image selection. FIG. 15 is a diagram illustrating image editing. FIG. 16 is a block diagram illustrating an example of the configuration of a three-dimensional space generation device to which the three-dimensional space generation method according to the fourth embodiment is applied. FIG. 17 is a flowchart illustrating an example of the operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the fourth embodiment is applied. FIG. 18 is a diagram illustrating a restored image. FIG. 19 is a diagram illustrating projective transformation. FIG. 20 is a diagram illustrating estimation of a movement direction. FIG. 21 is a diagram illustrating disparity estimation. FIG. 22 is a diagram illustrating disparity estimation. FIG. 23 is a diagram illustrating pixel value estimation. FIG. 24 is a block diagram illustrating an example of the configuration of a three-dimensional space generation device to which the three-dimensional space generation method according to the fifth embodiment is applied. FIG. 25 is a flowchart illustrating an example of the operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the fifth embodiment is applied. FIG. 26 is a block diagram illustrating an example of the configuration of a three-dimensional space generation device to which the three-dimensional space generation method according to the sixth embodiment is applied.FIG. 27 is a flowchart showing an example of operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the sixth embodiment is applied. FIG. 28 is a diagram showing an example of an image including a moving object captured by a camera. FIG. 29 is a diagram showing an example of a mask image. FIG. 30 is a diagram showing an example of an image after mask processing. FIG. 31 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which the three-dimensional space generation method according to the seventh embodiment is applied. FIG. 32 is a flowchart showing an example of operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the seventh embodiment is applied. FIG. 33 is a diagram showing an example of an image of the first frame captured by the first camera. FIG. 34 is a diagram showing an example of an image of the second frame captured by the first camera. FIG. 35 is a diagram showing an example of an image of the first frame captured by the second camera. FIG. 36 is a diagram showing an example of an image of the second frame captured by the second camera. FIG. 37 is a diagram showing an example of a frame-interpolated image obtained from the frames of FIGS. 33 and 34. FIG. 38 is a diagram showing an example of a frame-interpolated image obtained from the frames of FIGS. 35 and 36.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc., are not necessarily the same as those in reality. Furthermore, even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In this specification and each drawing, elements similar to those described above with reference to the previous drawings are designated by the same reference numerals, and detailed and redundant explanations will be omitted as appropriate.

[0012] First Embodiment FIG. 1 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a first embodiment is applied.

[0013] FIG. 2 is a flowchart showing an example of the operation of the three-dimensional space generation device to which the three-dimensional space generation method according to the first embodiment is applied.

[0014] The three-dimensional space generating device 10A according to the first embodiment is configured to include an image acquisition unit 12, an image editing unit 14, a three-dimensional space generating unit 16, and an image generating unit 18.

[0015] The image acquisition unit 12 acquires original images I captured under at least two conditions of an object for which a three-dimensional space model is to be generated (S1, S2, S3). These original images are input images that form the basis of the three-dimensional space.

[0016] In step S3, when the image acquisition unit 12 has completed the acquisition of all the original images I (S3: Yes), the image editing unit 14 edits the acquired original images I to generate edited images.

[0017] For this editing, the image editing unit 14 includes a camera placement mask processing unit 14a.

[0018] The camera arrangement mask processing unit 14a performs mask processing on the original image I based on the camera arrangement of the cameras used for shooting, and generates an edited image E (S4a).

[0019] The three-dimensional space generation unit 16 generates a three-dimensional space model M based on the original image I and the edited image E (S5).

[0020] Image generator 18 reads image generation parameters P from an external device (S6, S7). Once the reading is complete, image generator 18 generates an image based on the three-dimensional space model M and image generation parameters P (S8), and outputs generated image R (S9).

[0021] The image generating unit 18 repeatedly executes the operations from steps S7 to S9 until all image generating parameters P are acquired (S10).

[0022] Next, the processing performed by the camera placement mask processing unit 14a will be described using a specific example.

[0023] Here, as an example, a description will be given of generating an NeRF model using images captured by a camera mounted on a moving vehicle.

[0024] 3 and 4 are perspective and top views showing the acquisition of input images using cameras mounted on a bicycle as a moving vehicle. That is, Fig. 3 shows the bicycle 30 as seen from an oblique angle, and Fig. 4 shows the bicycle 30 as seen from directly above. Multiple cameras 20 (four in this example) are mounted on the rear wheel side of the bicycle 30.

[0025] In this way, the format of the image captured by the camera 20 mounted on the bicycle 30 may be an equirectangular format captured by a 360-degree camera in addition to a general format (perspective format), and an equirectangular format image may be used as is, or an equirectangular image converted into a general perspective format may be used.

[0026] Here, each camera 20 is a camera capable of capturing 360-degree images in equirectangular format.

[0027] FIG. 5 is a diagram showing an example of an image captured by a camera.

[0028] FIG. 6 is a diagram showing an example of a mask image.

[0029] FIG. 7 is a diagram showing an image after the mask processing.

[0030] As shown in Fig. 5, the occupant T appears in the images captured by each camera 20, and so it is desirable to avoid the appearance of the occupant T in order to generate an NeRF model. Therefore, based on the arrangement of each camera 20, the area in which the occupant T appears is prepared in advance as a mask image K as shown in Fig. 6, and an image in which the occupant T is avoided from appearing by mask processing is generated and used as shown in Fig. 7.

[0031] According to the three-dimensional space generating device 10A of this embodiment, by performing such editing processing on each image, it is possible to improve the accuracy of constructing a three-dimensional space using NeRF.

[0032] As described above, the three-dimensional space generation device 10A to which the three-dimensional space generation method according to this embodiment is applied makes it possible to improve the quality of the reconstructed three-dimensional space when reconstructing the three-dimensional space based on collected images.

[0033] Second Embodiment FIG. 8 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a second embodiment is applied.

[0034] FIG. 9 is a flowchart showing an example of the operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the second embodiment is applied.

[0035] The three-dimensional space generation device 10B according to the second embodiment is a partial modification of the configuration of the three-dimensional space generation device 10A according to the first embodiment.

[0036] That is, the three-dimensional space generation device 10B has a configuration in which the camera placement mask processing unit 14a in the three-dimensional space generation device 10A is replaced with a color adjustment unit 14b. Therefore, in Fig. 8, the same components as in Fig. 1 are denoted by the same reference numerals, and only different components will be described, avoiding redundant explanations. Similarly, in Fig. 9, the same steps as in Fig. 2 are denoted by the same step numbers, and only different components will be described, avoiding redundant explanations.

[0037] The image editing unit 14 includes a color adjustment unit 14b for editing the original image I to generate an edited image.

[0038] The color adjustment unit 14b performs color adjustment on the original image I based on color adjustment information prepared in advance, and generates an edited image E (S4b).

[0039] FIG. 10 shows two images taken under different color conditions.

[0040] The two images, i.e., the two images α shown in FIG. 10( a) and the image β shown in FIG. 10( b), have different shooting conditions, such as white balance, shutter speed, exposure sensitivity, and aperture. As a result, even if the subject is the same, the colors will be different. Therefore, it is possible to adjust the colors of the images using shooting parameters such as white balance, shutter speed, exposure sensitivity, and aperture when the images were acquired, and create images under the same conditions.

[0041] The color adjustment by the color adjustment unit 14b can be performed based on, for example, the relationship between the white balance, shutter speed, exposure sensitivity, aperture, and other RGB gains at the time of image acquisition, which is determined in advance.

[0042] Another example of different shooting conditions is when the lighting is different. In such cases, a subject of known color can be used as a reference for color adjustment, and the color can be adjusted by projecting a color target into a specific location on the camera, allowing the color to be sampled and adjusted to create an image under the same conditions.

[0043] According to the three-dimensional space generating device 10B of this embodiment, the accuracy of constructing a three-dimensional space using NeRF can be improved by the above-described color adjustment process.

[0044] As described above, according to the three-dimensional space generation device 10B to which the three-dimensional space generation method of this embodiment is applied, when reconstructing a three-dimensional space based on collected images, it is possible to improve the quality of the reconstructed three-dimensional space.

[0045] Third Embodiment FIG. 11 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a third embodiment is applied.

[0046] FIG. 12 is a flowchart showing an example of the operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the third embodiment is applied.

[0047] The three-dimensional space generation device 10C according to the third embodiment is a partial modification of the configuration of the three-dimensional space generation device 10A according to the first embodiment.

[0048] That is, the three-dimensional space generation device 10C has a configuration in which the camera placement mask processing unit 14a in the three-dimensional space generation device 10A is replaced with a similar image mixing unit 14c. Therefore, in Fig. 11, the same components as in Fig. 1 are denoted by the same reference numerals, and only different components will be described, avoiding redundant explanations. Similarly, in Fig. 12, the same steps as in Fig. 2 are denoted by the same step numbers, and only different components will be described, avoiding redundant explanations.

[0049] The image editing unit 14 includes a similar image mixing unit 14c for editing the original image I to generate an edited image.

[0050] The similar image mixing unit 14c mixes the images selected based on the similarity to generate an edited image E. Specifically, the original image I and an image captured under conditions different from those of the original image I are mixed based on the similarity. This process is performed by searching for similar images and partially replacing them, as shown in step S20 in FIG. 12 .

[0051] Next, we will explain the processing performed by the similar image mixing unit 14c when constructing a three-dimensional spatial model using NeRF, using an image taken under condition B during a time when there are no people around, to cover an area of ​​background buildings and scenery that is blocked by many people in an image taken under condition A.

[0052] FIG. 13 shows a crowd scene and its occluded areas photographed under different conditions.

[0053] FIG. 14 is a diagram illustrating similar image selection.

[0054] FIG. 15 is a diagram illustrating image editing.

[0055] Here we consider a crowd scene captured under different conditions.

[0056] 13(a) and 13(b), the similar image mixing unit 14c generates an image set A' by extracting images that have areas blocked by people from the image set generated under condition A. Next, the similar image mixing unit 14c generates an image indicating an editing area for each image in the image set A' shown in FIGS. 13(a) and 13(b), and generates an editing area image set A' linked to each image in the image set A', as shown in the two examples shown in FIGS.

[0057] The image set A' containing occluded areas, as illustrated in Figures 13(a) and (b), is generated by a user selecting images from the image set A', or by extracting images in which people, etc. are detected using an existing object detection model, etc.

[0058] Images showing occluded areas can be generated by the user using a UI for each image, or by using an existing segmentation model (such as Mask R-CNN) or a multi-view segmentation model.

[0059] Next, the similar image mixing unit 14c uses an image set B taken under different conditions as shown in FIGS. 14(b), (c), and (d) to perform interpolation processing on the editing area in the image set A' shown in FIG. 14(a).

[0060] In this interpolation process, feature points are calculated for the images in image set A' shown in Figure 14(a) and the images in image set B shown in Figures 14(b), (c), and (d) using an existing image processing library (such as OpenCV), and a similarity determination is performed to generate an image pair consisting of each image a1 in image set A' and the image b1 in image set B that has the highest similarity.

[0061] Next, for the pair of generated images a1 and b1, as shown in Figures 15(a) and (b), the similar image mixing unit 14c generates a pseudo-editing area image that matches the angle of view of the images from the editing area image corresponding to the images, as shown in Figures 15(c) and (d), based on the feature points of images a1 and b1.

[0062] The similar image mixing unit 14c then acquires feature amounts inside the editing area and feature amounts outside the editing area from the image and the editing area image. Similarly, it acquires feature amounts inside the editing area and feature amounts outside the editing area from the image and the pseudo-editing area image. The similar image mixing unit 14c may be equipped with a feature extractor (not shown) to perform these processes.

[0063] Next, the similar image mixing unit 14c performs scaling processing so that the mean and variance of the distribution of the feature quantities become the mean and variance of the distribution of the feature quantities. As a result, an image is obtained by interpolating the obtained feature quantities inside the editing region and the feature quantities outside the variable region using the editing region image. The similar image mixing unit 14c may be equipped with an image restorer (not shown) to perform this processing.

[0064] The similar image blending unit 14c then receives the image set and the color-corrected image set Z', and performs training of the NeRF model using the camera parameters of each image. Note that the image set A is a combination of the image set without the initial edited area and the image set Z' with the edited area subjected to interpolation processing.

[0065] Such NeRF model training can be performed using each image and camera parameter by deriving camera parameters for image set A from information acquired using the COLMAP method or a measurement device such as photographic equipment.

[0066] The similar image mixing unit 14c may include an NeRF learning unit (not shown) for learning such an NeRF model, and may also include a color correction unit (not shown) for generating a color-corrected image set Z′.

[0067] The similar image mixing unit 14 c outputs the NeRF model thus trained to the three-dimensional space generation unit 16 .

[0068] The three-dimensional space generation unit 16 converts the NeRF model into a format that can display a three-dimensional space model such as a 3D engine, and outputs it to the image generation unit 18 .

[0069] As described above, the three-dimensional space generation device 10C to which the three-dimensional space generation method according to this embodiment is applied makes it possible to improve the quality of the reconstructed three-dimensional space when reconstructing the three-dimensional space based on collected images.

[0070] Fourth Embodiment FIG. 16 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a fourth embodiment is applied.

[0071] FIG. 17 is a flowchart showing an example of the operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the fourth embodiment is applied.

[0072] The three-dimensional space generation device 10D according to the fourth embodiment is a partial modification of the configuration of the three-dimensional space generation device 10A according to the first embodiment.

[0073] That is, the three-dimensional space generation device 10D has a configuration in which the camera placement mask processing unit 14a in the three-dimensional space generation device 10A is replaced with a geometric transformation unit 14d. Therefore, in Fig. 16, the same components as in Fig. 1 are denoted by the same reference numerals, and only different components will be described, avoiding redundant explanations. Similarly, in Fig. 17, the same steps as in Fig. 2 are denoted by the same step numbers, and only different components will be described, avoiding redundant explanations.

[0074] The image editing unit 14 includes a geometric transformation unit 14d for editing the original image I to generate an edited image.

[0075] The geometric transformation unit 14d performs geometric transformation on the original image I to obtain an edited image (S22), thereby obtaining a restored image.

[0076] FIG. 18 is a diagram illustrating a restored image.

[0077] Fig. 18(a) shows a real scene, and Fig. 18(b) shows an original image corresponding to the real scene. Here, the cars shown in Figs. 18(a) and 18(b) are assumed to be unwanted objects. Figs. 18(c) and 18(d) show two reference images 1 and 2 that are used as references for removing unwanted objects. Fig. 18(e) shows a restored image in which unwanted objects have been removed using reference images 1 and 2 through geometric transformation by the geometric transformation unit 14d.

[0078] The original image shown in Figure 18(b) and reference images 1 and 2 shown in Figures 18(c) and (d) are input to the image acquisition unit 12, and output from the image acquisition unit 12 to the geometric transformation unit 14d of the image editing unit 14.

[0079] FIG. 19 is a diagram illustrating projective transformation.

[0080] FIG. 20 is a diagram illustrating estimation of the moving direction.

[0081] 21 and 22 are diagrams illustrating disparity estimation.

[0082] FIG. 23 is a diagram illustrating pixel value estimation.

[0083] The geometric transformation unit 14d performs projective transformation as shown in FIG. 19, movement direction estimation as shown in FIG. 20, parallax estimation as shown in FIGS. 21 and 22, and pixel value estimation as shown in FIG. 23 on the original image (FIG. 18(b)) and reference images 1 and 2 (FIGS. 18(c) and 18(d)), thereby obtaining a restored image (FIG. 18(e)).

[0084] The geometric transformation unit 14d performs projective transformation by generating a projection image 1 from a reference image 1 as shown in Figures 19(c) to 19(d), and generating a projection image 2 from a reference image 2 as shown in Figures 19(a) to 19(b), thereby generating a projection image as shown in Figure 19(e).

[0085] After such projective transformation, the geometric transformation unit 14d performs movement direction estimation. Using FIG. 20 as an example, the geometric transformation unit 14d calculates the movement direction and distance from the positional relationship between reference points (here, as an example, center points) in the projected image 1 shown in FIG. 20(d) obtained by projective transformation from FIG. 20(c) and the restored image shown in FIG. 20(e). Similarly, the geometric transformation unit 14d calculates the movement direction and distance from the positional relationship between reference points (here, as an example, center points) in the projected image 2 shown in FIG. 20(b) obtained by projective transformation from FIG. 20(a) and the restored image shown in FIG. 20(e). The geometric transformation unit 14d then determines the average value of these two movement directions as the movement direction. In this way, the geometric transformation unit 14d performs movement direction estimation.

[0086] After estimating the movement direction in this way, the geometric transformation unit 14d performs disparity estimation. Specifically, as illustrated in Fig. 21 and Fig. 22 , the geometric transformation unit 14d calculates the correspondence between the target region of the projection image 1 and the target region of the projection image 2 along the estimated movement direction by dynamic time warping (edit distance), and performs disparity estimation. Fig. 21 illustrates a case where the movement direction is diagonal to the region, and Fig. 22 illustrates a case where the movement direction is parallel to the region.

[0087] After estimating the disparity, the geometric transformation unit 14d estimates pixel values. To estimate pixel values, the geometric transformation unit 14d predicts the positional deviation in the target region of the restoration image by extrapolating or interpolating the disparity using the distance between target region 1 and target region 2 and the distance between target region 1 and the restoration image, as shown in Fig. 23 for example. The geometric transformation unit 14d estimates pixel values ​​in this manner.

[0088] As described above, the three-dimensional space generation device 10D to which the three-dimensional space generation method according to this embodiment is applied makes it possible to improve the quality of the reconstructed three-dimensional space when reconstructing the three-dimensional space based on collected images.

[0089] Fifth Embodiment FIG. 24 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a fifth embodiment is applied.

[0090] FIG. 25 is a flowchart showing an example of the operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the fifth embodiment is applied.

[0091] The three-dimensional space generation device 10E according to the fifth embodiment is a partial modification of the configuration of the three-dimensional space generation devices 10A, 10B, 10C and 10D according to the first to fourth embodiments.

[0092] That is, the three-dimensional space generation device 10E includes, in the image editing unit 14, a geometric transformation unit 14d in the three-dimensional space generation device 10D, a similar image mixing unit 14c in the three-dimensional space generation device 10C, a camera placement mask processing unit 14a in the three-dimensional space generation device 10A, and a color adjustment unit 14b in the three-dimensional space generation device 10B, in that order of processing.

[0093] The three-dimensional space generation device 10E further includes a clustering unit 15 between the image editing unit 14 and the three-dimensional space generation unit 16.

[0094] Therefore, in Fig. 24, the same parts as those already mentioned are denoted by the same reference numerals, and only different parts will be described to avoid duplicated explanations. Similarly, in Fig. 25, the same steps as those already mentioned are denoted by the same step numbers, and only different parts will be described to avoid duplicated explanations.

[0095] That is, in the three-dimensional space generating device 10E, the image editing unit 14 performs the following processing.

[0096] The geometric transformation unit 14d performs a geometric transformation on the original image I (S22).

[0097] The similar image mixing unit 14c searches for similar images in the image that has been geometrically transformed by the geometric transformation unit 14d and partially replaces the image (S20).Then, the process proceeds to step S3.

[0098] After step S3, the camera arrangement mask processing unit 14a performs mask processing based on the camera arrangement on the image that has been processed in step S20 (S4a).

[0099] The color adjustment unit 14b performs color adjustment on the image that has been subjected to the masking process by the camera arrangement mask processing unit 14a based on color adjustment information prepared in advance, and generates an edited image E (S4b).

[0100] Thereafter, the clustering unit 15 performs clustering on the edited image E generated by the color adjustment unit 14b based on the shooting conditions at the time of shooting (S40).

[0101] Thereafter, the three-dimensional space generation unit 16 generates a three-dimensional space model M based on the original image I and the edited image E obtained by clustering by the clustering unit 15 (S5).

[0102] To specifically explain the processing by the clustering unit 15 and the three-dimensional space generation unit 16, the clustering unit 15 divides a large number of two-dimensional images by clustering according to predetermined criteria, and the three-dimensional space generation unit 16 generates a three-dimensional space from the divided set of images.

[0103] As the predetermined criterion, information representing location information such as GPS acquired when an image is captured can be used.

[0104] Furthermore, the k-means method can be used as a clustering method. When the k-means method is used, it is possible to divide the data into a predetermined number of clusters using information on latitude, longitude, and altitude. Then, the three-dimensional space generation unit 16 generates a three-dimensional space in units of division, thereby enabling the efficient generation of a three-dimensional space.

[0105] According to the three-dimensional space generation device 10E to which the three-dimensional space generation method of this embodiment is applied, it is possible to improve the quality of the reconstructed three-dimensional space when reconstructing the three-dimensional space based on collected images as described above.

[0106] Sixth Embodiment FIG. 26 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a sixth embodiment is applied.

[0107] FIG. 27 is a flowchart showing an example of the operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the sixth embodiment is applied.

[0108] The three-dimensional space generation device 10F according to the sixth embodiment is a partial modification of the configuration of the three-dimensional space generation device 10A according to the first embodiment.

[0109] That is, the three-dimensional space generation device 10F has a configuration in which an animated object mask processing unit 14e is added after the camera placement mask processing unit 14a in the image editing unit 14 of the three-dimensional space generation device 10A. Therefore, in Fig. 26, the same parts as in Fig. 1 are indicated by the same reference numerals, and duplicated explanations will be avoided, and only different parts will be described. Similarly, in Fig. 27, the same steps as in Fig. 2 are indicated by the same step numbers, and duplicated explanations will be avoided, and only different parts will be described.

[0110] The moving object mask processing unit 14e performs a moving object masking process on the edited image E that has been masked by the camera placement mask processing unit 14a, and outputs an edited image E' in which the moving object has been masked (S4e).

[0111] The three-dimensional space generation unit 16 generates a three-dimensional space model M based on the original image I and the edited image E' (S5).

[0112] The moving object mask processing by the moving object mask processing unit 14e will be specifically described below.

[0113] The moving object mask processing unit 14e identifies moving objects that have a negative effect on the creation of the three-dimensional space model M by the three-dimensional space generation unit 16, and removes them from the processing target by masking. This processing is performed by semantic segmentation.

[0114] FIG. 28 is a diagram showing an example of an image including a moving object captured by a camera.

[0115] FIG. 29 is a diagram showing an example of a mask image.

[0116] FIG. 30 is a diagram showing an example of an image after the mask processing.

[0117] In the image to be processed, such as the example shown in Figure 28, a person is shown on the right side as a moving object U. If this person moves, the spatial position of the person will shift between still images taken in multiple frames at different times, causing problems in generating a three-dimensional spatial model.

[0118] Therefore, the moving object mask processing unit 14e performs processing to remove this person from the target. This processing can be performed by semantic segmentation, but various other methods have already been proposed.

[0119] For example, it is possible to apply a method such as Mask R-CNN disclosed in Non-Patent Document 1. According to this method, it is possible to perform mask processing on object regions that belong to the category of moving objects U, such as people and automobiles.

[0120] Specifically, as shown in Fig. 29, a masked image is prepared in advance by masking the moving object area V in addition to the area K where the passenger T is captured. Then, as shown in Fig. 30, an image is generated and used in which the moving object V is also avoided by masking in addition to the passenger T.

[0121] By performing such editing processing on each image, the accuracy of constructing a three-dimensional space using NeRF can be improved.

[0122] Note that Figure 26 shows a configuration in which the image editing unit 14 has an animal body mask processing unit 14e added after the camera placement mask processing unit 14a, but it is also possible to add the animal body mask processing unit 14e before the camera placement mask processing unit 14a and perform the camera placement mask processing after performing the animal body mask processing.

[0123] In addition, with moving object masking, the smaller the range of the masking process, the higher the accuracy of the reproduction of the three-dimensional space of NeRF. Therefore, it is possible to collect videos using a strategy that minimizes the masked area. For example, when there are multiple moving objects shooting video, it is possible for them to communicate with each other and change their movement paths so that the area of ​​each other's masked area is as small as possible. For example, by moving three or more moving objects shooting video in a straight line, the two moving objects at the front and middle overlap from the last moving object, and the two moving objects at the back and middle overlap from the front moving object's perspective, thereby achieving the effect of minimizing the masked area. In addition to moving objects on the ground, moving moving objects so that they overlap each other, such as when shooting with multiple aerial drones, can achieve the effect of reducing the masked area and improving the accuracy of spatial reproduction.

[0124] Seventh Embodiment FIG. 31 is a block diagram showing an example of the configuration of a three-dimensional space generation device to which a three-dimensional space generation method according to a seventh embodiment is applied.

[0125] FIG. 32 is a flowchart showing an example of the operation of a three-dimensional space generation device to which the three-dimensional space generation method according to the seventh embodiment is applied.

[0126] The three-dimensional space generation device 10G according to the seventh embodiment is a partial modification of the configuration of the three-dimensional space generation device 10A according to the first embodiment.

[0127] That is, the three-dimensional space generation device 10G has a configuration in which a shutter time identification unit 13 is added between the image acquisition unit 12 and the image editing unit 14 in the three-dimensional space generation device 10A, and further, the image editing unit 14 is provided with an image interpolation processing unit 14f instead of the camera placement mask processing unit 14a. Therefore, in Figure 31, the same parts as in Figure 1 are indicated by the same reference numerals, and duplicated explanations will be avoided, and only different parts will be described. Similarly, in Figure 32, the same steps as in Figure 2 are indicated by the same step numbers, and duplicated explanations will be avoided, and only different parts will be described.

[0128] Here, we consider generating a three-dimensional space, including moving objects, from time-series video captured by multiple cameras. When capturing images of an object with multiple cameras to generate a three-dimensional space at a specific instant in time, if the shutter timing of each camera is not synchronized, each camera will capture images of the moving object at a different position in three-dimensional space, making it difficult to generate the three-dimensional space. Therefore, we generate an image at the same time from multiple original images I captured by unsynchronized cameras.

[0129] The shutter time specifying unit 13 performs a process of specifying the shutter time for the original images I captured by the multiple cameras and sent from the image acquisition unit 12. That is, the shutter timing of each of the multiple cameras is specified (S31).

[0130] The image editing unit 14 edits the original image I, for which the shutter time has been specified, to generate an edited image E. Specifically, the image interpolation processing unit 14f interpolates the original image I based on the specified shutter time, and outputs the edited image E after the image interpolation (S32).

[0131] The three-dimensional space generation unit 16 generates a three-dimensional space model M based on the original image I and the edited image E (S5).

[0132] The processing by the shutter time identification unit 13 and the image interpolation processing unit 14f will be specifically described below.

[0133] FIG. 33 is a diagram showing an example of the first frame image captured by the first camera.

[0134] FIG. 34 is a diagram showing an example of the second frame image captured by the first camera.

[0135] FIG. 35 is a diagram showing an example of the first frame image captured by the second camera.

[0136] FIG. 36 is a diagram showing an example of the second frame image captured by the second camera.

[0137] As shown in FIGS. 33 to 36, in this example, the object of three-dimensional space reproduction is the road surface and a bicycle 30 traveling.

[0138] Here, an object whose movement characteristics are known in advance is included in the image. In this example, a fan F with two propellers is included in the image. For example, if the fan F rotates at about 10 to 20 Hz, the propellers of the fan F can be captured as an image by shooting at a frame rate of about 30 fps and using a shutter speed long enough to prevent blurring. From the angle of the propellers, the shutter time identification unit 13 can identify the degree of deviation between each frame from each camera.

[0139] Here, the rotation angle of the propeller may be simply estimated from a specific pixel value. Because the pixel values ​​of pixels near the propeller in the image change periodically, the shutter time identification unit 13 performs a frequency analysis such as a Fourier transform to find the phase near the frequency of the propeller's period, and from that deviation, it is possible to identify the deviation of the images from each camera.

[0140] The image interpolation processing unit 14f generates images of the same time from images captured by multiple cameras. Image interpolation is used to generate the images. For example, a frame interpolation method such as that described in Non-Patent Document 5 can be used for this image interpolation. According to this method, it is possible to obtain frames corresponding to the same time from the time difference between images and the time-series images of each camera by frame interpolation. The time at which frames are to be generated or the time interval at which interpolated images are to be generated are determined in advance.

[0141] FIG. 37 is a diagram showing an example of a frame interpolated image obtained from the frames of FIGS. 33 and 34. In FIG.

[0142] FIG. 38 is a diagram showing an example of a frame interpolated image obtained from the frames of FIGS. 35 and 36. In FIG.

[0143] By using interpolated images such as those shown in FIGS. 37 and 38, it is possible to generate a three-dimensional space even for a moving object.

[0144] Although the above describes an example in which frames at specific times are generated from images of each camera, the present invention is not limited to such a method. For example, it is also possible to generate a time-series space using frames at different times, or to use a method called Dynamic NeRF as described in Non-Patent Document 2.

[0145] As described above, according to the present invention, the quality of the reconstructed three-dimensional space can be improved by generating a set of images necessary for reconstructing the three-dimensional space, by using masking processing to remove the photographer from the video taken at the location to be reconstructed, or by using video taken under different conditions, such as at different times of day, to interpolate areas where the scenery or buildings are obstructed.

[0146] The present invention is not limited to the above-described embodiments, and can be embodied by modifying the components within the scope of the gist of the present invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0147] 10 3D space generation device 12 Image acquisition unit 13 Shutter time identification unit 14 Image editing unit 14a Camera placement mask processing unit 14b Color adjustment unit 14c Similar image mixing unit 14d Geometric transformation unit 14e Moving object mask processing unit 14f Image interpolation processing unit 15 Clustering unit 16 3D space generation unit 18 Image generation unit 20 Camera 30 Bicycle E Edited image F Fan I Original image P Image generation parameters R Generated image T Rider K Mask image (rider) U Moving object V Mask image (moving object)

Claims

1. A device for generating a three-dimensional space comprising: an image acquisition unit that acquires original images taken under at least two conditions of an object for which a three-dimensional space model is to be generated; an image editing unit that edits the original images to generate edited images; and a three-dimensional space generation unit that generates the three-dimensional space model based on the original images and the edited images.

2. The three-dimensional space generating device of claim 1, wherein the image editing unit includes at least one of: a camera arrangement mask processing unit that performs mask processing of the original image based on the camera arrangement of the camera used for the shooting, in order to edit the original image and generate the edited image; a color adjustment unit that performs color adjustment on the original image based on color adjustment information prepared in advance; a similar image mixing unit that mixes the original image and an image shot under conditions different from those of the original image based on similarity; and a geometric transformation unit that performs geometric transformation on the original image.

3. A method for generating a three-dimensional space, comprising: a processor of a three-dimensional space generation device acquires original images taken under at least two conditions of a target for which a three-dimensional space model is to be generated; edits the original images to generate edited images; and generates the three-dimensional space model based on the original images and the edited images.

4. A program for causing a processor to realize the following functions: acquiring original images taken under at least two conditions of a subject for which a three-dimensional spatial model is to be generated; editing the original images to generate edited images; and generating the three-dimensional spatial model based on the original images and the edited images.

Images