Information processing method and information processing device

Abstract

The present technology relates to an information processing method and an information processing device that make it possible to perform three-dimensional reconstruction using a captured image which includes a static obstacle and a dynamic obstacle. This information processing device generates 3D Gaussian on the basis of: a captured image from an N-viewpoint; and at least one of an obstacle category that is a category of an object that is considered to be an obstacle in the captured image from the N-viewpoint and a non-obstacle category that is a category of an object that is not considered to be an obstacle. The present technology can be applied to, for example, an information processing device that performs three-dimensional reconstruction using the captured image from the N-viewpoint transmitted from an imaging device.

Description

Information processing method and information processing device

[0001] The present technology relates to an information processing method and an information processing device, and more particularly to an information processing method and an information processing device that enable three-dimensional reconstruction using captured images that include static obstacles and dynamic obstacles.

[0002] In recent years, 3DGS (3D Gaussian Splatting) has been devised as a 3D reconstruction method, which has significantly improved the accuracy and speed of generating 2D images from any viewpoint based on 3D reconstruction results (Novel View Synthesis). However, the captured images used for 3DGS 3D reconstruction are generally assumed to be free of static and dynamic obstacles. This increases the cost of collecting the captured images used for 3D reconstruction. For example, when performing 3D reconstruction using images captured in large cities or tourist destinations, traffic restrictions must be implemented to prevent static and dynamic obstacles from being included in the captured images, which increases the cost of collecting the captured images. As a result, practical application of this method is difficult.

[0003] Therefore, a method has been devised to remove dynamic objects by separately learning static and dynamic scenes (see, for example, Non-Patent Document 1). A method has also been devised to detect obstacle pixels using a network that has learned time-series uncertainty and apply a mask (see, for example, Non-Patent Document 2). These methods enable 3D reconstruction using captured images that include dynamic obstacles.

[0004] Dongbin Zhang, Chuming Wang, Weitao Wang, Peihao Li, Minghan Qin, Haoqian Wang, “Gaussian in the Wild:3D Gaussian Splatting for Unconstrained Image Collections”, arXiv preprint, arXiv:2403.15704, 2024Weining Ren, Zihan Zhu, Boyang Sun, Jiaqi Chen, Marc Pollefeys, Songyou Peng, “NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild”, In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024 , pp.8931-8940

[0005] However, with the above-mentioned method, it is difficult to perform 3D reconstruction using captured images that include static obstacles such as standing people, etc. Therefore, there is a demand for a method of performing 3D reconstruction using captured images that include not only dynamic obstacles but also static obstacles, but such demand has not yet been fully met.

[0006] The present technology has been made in view of such circumstances, and makes it possible to perform three-dimensional reconstruction using captured images that include static and dynamic obstacles.

[0007] An information processing method according to one aspect of the present technology includes an information processing device generating a 3D Gaussian based on images captured from a plurality of viewpoints and at least one of an obstacle category, which is a category of objects that are obstacles in the images captured from the plurality of viewpoints, and a non-obstacle category, which is a category of objects that are not obstacles.

[0008] An information processing device according to one aspect of the present technology is an information processing device that includes a generation unit that generates a 3D Gaussian based on images captured from multiple viewpoints and at least one of an obstacle category, which is a category of objects that are obstacles in the images captured from the multiple viewpoints, and a non-obstacle category, which is a category of objects that are not obstacles.

[0009] In one aspect of the present technology, a 3D Gaussian is generated based on images captured from multiple viewpoints and at least one of an obstacle category, which is a category of objects that are obstacles in the images captured from the multiple viewpoints, and a non-obstacle category, which is a category of objects that are not obstacles.

[0010] An information processing device according to one aspect of the present technology may be an independent device or a module incorporated into another device.

[0011] Fig. 1 is a block diagram showing a configuration example of an embodiment of an information processing system to which the present technology is applied. Fig. 2 is a block diagram showing a configuration example of an information processing device. Fig. 3 is a first diagram showing an example of a processing result of each unit of the information processing device. Fig. 4 is a second diagram showing an example of a processing result of each unit of the information processing device. Fig. 5 is a diagram explaining an effect of the information processing device. Fig. 6 is a flowchart explaining 2D image generation processing. Fig. 7 is a block diagram showing a configuration example of computer hardware.

[0012] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. One embodiment 2. Description of a computer to which the present technology is applied

[0013] 1. One Embodiment Configuration Example of Information Processing System FIG. 1 is a block diagram showing a configuration example of an embodiment of an information processing system to which the present technology is applied.

[0014] 1 includes an image capturing device 11 and an information processing device 12. In the information processing system 10, the information processing device 12 performs 3D reconstruction using images captured by the image capturing device 11 from N viewpoints (N is a plural number).

[0015] Specifically, the image capturing device 11 is configured by, for example, an aerial photography device such as a drone. The image capturing device 11 captures images while moving and acquires images from N viewpoints. Therefore, the images from each viewpoint are captured at different times.

[0016] The image capturing device 11 may be configured with multiple aerial image capturing devices. In this case, the multiple aerial image capturing devices share the responsibility of capturing images from N viewpoints. The image capturing device 11 may also be configured with N fixed cameras fixed at predetermined positions. In this case, for example, the N fixed cameras simultaneously capture images to capture images from N viewpoints.

[0017] The image capturing device 11 transmits the captured images from the N viewpoints and the camera parameters to the information processing device 12. The camera parameters include internal parameters of the image capturing device 11 and external parameters that represent the position and orientation of the image capturing device 11 when capturing each of the captured images from the N viewpoints.

[0018] The information processing device 12 performs three-dimensional reconstruction by generating a 3D Gaussian using the captured images from N viewpoints and camera parameters transmitted from the image capturing device 11. The information processing device 12 generates a 2D image from any viewpoint that does not include obstacles using the 3D Gaussian.

[0019] At least a part of the photographing device 11 and the information processing device 12 may be integrated.

[0020] <Configuration Example of Information Processing Apparatus> FIG. 2 is a block diagram showing a configuration example of the information processing apparatus 12 of FIG.

[0021] The information processing device 12 in FIG. 2 includes an input unit 31 , a segmentation unit 32 , a first optimization unit 33 , a selection unit 34 , an inpainting unit 35 , a second optimization unit 36 ​​, and a 2D image generation unit 37 .

[0022] The input unit 31 accepts user operations. The user operates the input unit 31 to, for example, specify an obstacle category, which is a category of objects that are obstacles in the images captured from N viewpoints transmitted from the image capturing device 11 of FIG. 1 . In response to the operation, the input unit 31 inputs text information representing the obstacle category as a positive prompt to the segmentation unit 32. The user operates the input unit 31 to, for example, specify a non-obstacle category, which is a category of objects that are not obstacles in the images captured from N viewpoints. In response to the operation, the input unit 31 inputs text information representing the non-obstacle category as a negative prompt to the segmentation unit 32. Note that the user need only specify at least one of the obstacle category and the non-obstacle category, and is not required to specify both.

[0023] The segmentation unit 32 acquires captured images (RGB images) from N viewpoints and camera parameters transmitted from the image capture device 11. The segmentation unit 32 adds a mask to the captured images from N viewpoints using a text-based pre-trained segmentation model based on the captured images from N viewpoints, camera parameters, and positive and negative prompts. Specifically, the segmentation unit 32 detects, as an obstacle area, an area of ​​an object in the category represented by the positive prompt from an area in the captured image other than an area of ​​an object in the category represented by the negative prompt, and adds a mask to that area.

[0024] The text-based pre-trained segmentation model may be, for example, the lang-segment-anything module developed by Luca Medeiros et al. on GITHUB (registered trademark). The lang-segment-anything module combines object detection and a transformer-based segmentation model. The segmentation unit 32 supplies the N viewpoint images with masks and camera parameters to the first optimization unit 33 and the selection unit 34.

[0025] The first optimization unit 33 uses the captured images from N viewpoints and camera parameters supplied from the segmentation unit 32 to optimize the parameters of one or more 3D Gaussians using a 3DGS loss function modified to ignore masked pixels. As a result, the first optimization unit 33 generates a provisional 3D Gaussian. This provisional 3D Gaussian cannot adequately represent the masked region in the multi-viewpoint captured images. Therefore, a 2D image generated based on this provisional 3D Gaussian will have artifacts or defects around the masked region in the multi-viewpoint captured images. The first optimization unit 33 supplies the provisional 3D Gaussian to the selection unit 34 and the second optimization unit 36.

[0026] The selection unit 34 selects the i-th 3D Gaussian g from the top of the provisional 3D Gaussians supplied from the first optimization unit 33. ｉ The selection unit 34 selects the number C of viewpoints to which a mask is added in the area corresponding to the processing target in the photographed image, based on the photographed images of the N viewpoints and the camera parameters. mask(gi) and the number of viewpoints to which no mask is added, C unmask(gi) Count the numbers.

[0027] The selection unit 34 selects the number C mask(gi) and Math C unmask(gi) Based on at least one of the above, it is determined whether or not the area corresponding to the processing target is to be selected as the target area for inpainting by the inpainting unit 35.

[0028] For example, the selection unit 34 selects the number C mask(gi) is equal to or greater than the first threshold, and the number C unmask(gi) is equal to or less than the second threshold, the selection unit 34 determines that the area corresponding to the processing target is an area to which a mask is added in multiple viewpoints. Then, the selection unit 34 determines that the area should be selected as the target area for inpainting. mask(gi) is less than the first threshold value, or the number C unmask(gi)If the second threshold is greater than the first threshold, the selection unit 34 determines that the region corresponding to the processing target is not to be selected as the region to be inpainted. ｉ Any one of the following can be used as the 3D Gaussian g ｈ Select as.

[0029] The first threshold and the second threshold can be set to any number. For example, the first threshold and the second threshold can be set to 3, or the first threshold can be set to 4 and the second threshold can be set to 6.

[0030] The selection unit 34 selects the 3D Gaussian g ｈ A target image set is generated by arranging the regions corresponding to the four viewpoints in a grid, two in the row direction and two in the column direction. The four viewpoints are selected from N viewpoints so as to be as diverse as possible. unmask(gi) is not 0, the four viewpoints are assigned a 3D Gaussian g ｈ The area corresponding to the mask includes a viewpoint to which no mask is added.

[0031] The selection unit 34 supplies the target image set to the inpainting unit 35. The selection unit 34 uses the inpainted target image set supplied from the inpainting unit 35 as a result to update the four viewpoint captured images corresponding to the target image set, among the captured images from the N viewpoints.

[0032] The selection unit 34 then repeats (k-1) times the process of generating the above-mentioned target image set using the updated captured images from the N viewpoints and the process of updating the captured images from the N viewpoints using the inpainted target image set. The selection unit 34 supplies the captured images from the N viewpoints and camera parameters after updating a total of k times (k is an integer of 1 or greater) to the second optimization unit 36 ​​via the inpainting unit 35.

[0033] The inpainting unit 35 inpaints (interpolates) masked regions in the target image set using a pre-trained 2D inpainting diffusion model. For example, the 2D inpainting diffusion model may be the model described in Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky, “Resolution-robust Large Mask Inpainting with Fourier Convolutions,” arXiv preprint, arXiv:2109.07161, 2021.

[0034] In a 2D inpainting diffusion model, when images from multiple viewpoints are input in a grid pattern, inpainting can be performed with 3D consistency, compared to when each viewpoint image is input individually. This is described, for example, in Ethan Weber, Aleksander Holynski, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, and Angjoo Kanazawa, "Nerfiller: Completing scenes via generative 3D inpainting," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 20731-20741. Furthermore, excessive inpainting may impair 3D consistency.

[0035] As a result, the inpainting unit 35 calculates the 3D Gaussian g that needs to be inpainted in the captured images from the four viewpoints. ｈBy inpainting a target image set in which only the regions corresponding to the target image are arranged in a grid pattern, inpainting can be performed while ensuring 3D consistency. The inpainting unit 35 supplies the inpainted target image set to the selection unit 34.

[0036] The second optimization unit 36 ​​(generation unit) optimizes the parameters of the 3D Gaussian using the provisional 3D Gaussian as initial values, using the captured images from N viewpoints and the camera parameters supplied from the selection unit 34 via the inpainting unit 35. In this way, the second optimization unit 36 ​​generates a final 3D Gaussian and supplies it to the 2D image generation unit 37.

[0037] The 2D image generator 37 uses the final 3D Gaussian provided by the second optimizer 36 to generate an obstacle-free 2D image for any viewpoint.

[0038] <Example of Processing Results of Each Unit of Information Processing Apparatus> FIGS. 3 and 4 are diagrams showing examples of processing results of each unit of the information processing apparatus 12 of FIG.

[0039] When the image capturing device 11 captures an image of a desk with a cup placed on a dishcloth from N viewpoints, the image capturing device 11 transmits captured images 51-1 to 51-N from the N viewpoints in FIG. 3A and camera parameters.

[0040] In this case, when the user specifies a cup as the obstacle category, the segmentation unit 32 adds a mask to a region 52a of the cup in the photographed images 51-1 to 51-N, as shown in B of Fig. 3. This generates photographed images 52-1 to 52-N from N viewpoints to which the mask has been added.

[0041] The first optimization unit 33 optimizes the parameters of the 3D Gaussian using the captured images 52-1 to 52-N and the camera parameters, and the loss function of the 3DGS modified to ignore the pixels to which the mask is added, thereby generating, for example, three provisional 3D Gaussians 53-1 to 53-3 shown in FIG. 3C.

[0042] The selection unit 34 selects a 3D Gaussian g from the provisional 3D Gaussians 53-1 to 53-3 based on the captured images 52-1 to 52-N from N viewpoints to which the masks have been added and the camera parameters. ｈ The selection unit 34 selects the 3D Gaussian g ｈ The target image set 55 is generated by arranging the regions 54-1 to 54-4 corresponding to the masked regions 54a-1 to 54a-4 in a 2x2 grid.

[0043] The inpainting unit 35 inpaints regions 54a-1 to 54a-4 of the target image set 55 using a 2D inpainting diffusion model, generating an inpainted target image set 56. The selection unit 34 updates the captured images 52-1 to 52-N from the N viewpoints using the target image set 56. Then, the generation of the target image set by the selection unit 34, the inpainting by the inpainting unit 35, and the updating of the captured images from the N viewpoints using the inpainted target image set by the selection unit 34 are repeated.

[0044] After updating the captured images from N viewpoints a total of k times, the second optimization unit 36 ​​uses the updated captured images from N viewpoints and camera parameters to optimize the parameters of the 3D Gaussians 53-1 to 53-3, with the provisional 3D Gaussians 53-1 to 53-3 in C of Fig. 3 as initial values. As a result, the final 3D Gaussians 57-1 to 57-3 in C of Fig. 4 are generated.

[0045] <Explanation of Effects> FIG. 5 is a diagram for explaining the effects of the information processing device 12 of FIG.

[0046] In the example of Fig. 5, 3D reconstruction is performed using captured images from N viewpoints including the captured image 71 of Fig. 5A, and a 2D image from a viewpoint corresponding to the captured image 71 is generated. Also, the obstacle categories are "human", "person", and "belongings", and the non-obstacle categories are "billboard" and "signboard".

[0047] In this case, the 2D image based on the 3D reconstruction result by vanilla 3DGS is the 2D image 72 shown in Fig. 5B. The 2D image 72 contains artifacts and obstacles.

[0048] The 2D image generated using the provisional 3D Gaussian is the 2D image 73 in Fig. 5C. In the 2D image 73, defects and artifacts occur near the regions 73a and 73b where masks are added in the multi-viewpoint captured images.

[0049] In contrast, the 2D image generated by the information processing device 12 is a 2D image 74 shown in D of Fig. 5. In the 2D image 74, defects and artifacts near the masked regions 74a and 74b in the multi-viewpoint captured images are suppressed.

[0050] <Explanation of 2D Image Generation Processing> Fig. 6 is a flowchart illustrating the 2D image generation processing by the information processing device 12 in Fig. 2. This 2D image generation processing is started, for example, when captured images from N viewpoints and camera parameters are transmitted from the imaging device 11.

[0051] In step S11 of FIG. 6, the input unit 31 determines whether or not at least one of an obstacle category and a non-obstacle category has been designated by the user.

[0052] If it is determined in step S11 that at least one of the obstacle category and the non-obstacle category has not yet been designated, the process waits until it is designated.

[0053] On the other hand, if it is determined in step S11 that at least one of the obstacle category and the non-obstacle category has been designated, the input unit 31 inputs text information representing the obstacle category as a positive prompt to the segmentation unit 32. The input unit 31 inputs text information representing the non-obstacle category as a negative prompt to the segmentation unit 32. Then, the processing proceeds to step S12.

[0054] In step S12, the segmentation unit 32 adds masks to the captured images from the N viewpoints using the captured images from the N viewpoints, camera parameters, and positive and negative prompts transmitted from the image capture device 11. The segmentation unit 32 supplies the captured images from the N viewpoints and camera parameters to which the masks have been added to the first optimization unit 33 and the selection unit 34.

[0055] In step S13, the first optimization unit 33 generates a provisional 3D Gaussian using the captured images from the N viewpoints to which the masks have been added in step S12 and the camera parameters. The first optimization unit 33 supplies the provisional 3D Gaussian to the selection unit 34 and the second optimization unit 36.

[0056] In step S14, the information processing device 12 sets j to 1. In step S15, the selection unit 34 selects the provisional 3D Gaussian g generated in step S13 based on the captured images from the N viewpoints to which the masks have been added in step S12 and the camera parameters. ｉ Number of C mask(gi) and C unmask(gi) Count.

[0057] In step S16, the selection unit 34 selects the number C counted in step S15. mask(gi) and Math C unmask(gi) Based on this, the provisional 3D Gaussian g generated in step S13 is ｉ From 3D Gaussian g ｈ Select .

[0058] In step S17, the selection unit 34 selects the 3D Gaussian g ｈ The selection unit 34 generates a target image set by arranging the regions corresponding to the above in a 2×2 grid. The selection unit 34 supplies the target image set to the inpainting unit 35.

[0059] In step S18, the inpainting unit 35 inpaints the masked region of the target image set generated in step S17. The inpainting unit 35 supplies the inpainted target image set to the selection unit 34.

[0060] In step S19, the selection unit 34 updates the four viewpoint images corresponding to the target image set, among the N viewpoint images, using the target image set inpainted in step S18.

[0061] In step S20, the information processing device 12 determines whether j is k. If it is determined in step S20 that j is not k, the process proceeds to step S21. In step S21, the information processing device 12 increments j by 1, and the process returns to step S15, where the subsequent processes are repeated.

[0062] On the other hand, if it is determined in step S20 that j is k, that is, if the updating of the captured images from the N viewpoints has been repeated k times, the selection unit 34 supplies the captured images from the N viewpoints and the camera parameters updated in step S19 to the second optimization unit 36 ​​via the inpainting unit 35. Then, the processing proceeds to step S22.

[0063] In step S22, the second optimization unit 36 ​​generates a final 3D Gaussian using the captured images from the N viewpoints and the camera parameters supplied from the selection unit 34, and using the provisional 3D Gaussian generated in step S13 as an initial value. The second optimization unit 36 ​​supplies the final 3D Gaussian to the 2D image generation unit 37.

[0064] In step S23, the 2D image generator 37 generates an obstacle-free 2D image for an arbitrary viewpoint using the final 3D Gaussian generated in step S22, and the 2D image generation process then ends.

[0065] As described above, the information processing device 12 generates a 3D Gaussian based on the captured images from N viewpoints and at least one of the obstacle category and the non-obstacle category. Therefore, it is possible to perform 3D reconstruction using captured images that include static obstacles and dynamic obstacles.

[0066] The information processing device 12 also applies masks to the captured images from the N viewpoints based on at least one of the obstacle category and the non-obstacle category. The information processing device 12 then inpaints the captured images from the N viewpoints to which the masks have been applied, and generates a 3D Gaussian using the inpainted captured images from the N viewpoints. This allows for stable 3D reconstruction of areas such as floors that are frequently hidden by obstacles.

[0067] The information processing device also allows the user to specify the obstacle and non-obstacle categories that are used to generate the 3D Gaussians.

[0068] As a result, for example, by a user specifying people as an obstacle category, it is possible to perform 3D reconstruction of large cities or tourist destinations that do not contain people using images taken of such cities or tourist destinations that are densely populated with people and have few non-human areas.

[0069] 2. Description of a Computer to which the Present Technology is Applied The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, a program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0070] FIG. 7 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0071] In the computer, a processing circuit 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Access Memory) 903 are interconnected by a bus 904 .

[0072] An input / output interface 905 is also connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0073] The input unit 906 may include physical or virtual operating means, such as a keyboard, mouse, or touch panel, that a user operates to input information, as well as means for the user to input information by voice, gaze, or the like. The input unit 906 may also include sensors for inputting various physical quantities to the computer. For example, the input unit 906 may include sensors, such as a camera or microphone, that acquire physical quantities such as light (including infrared light other than visible light) and sound. The input unit 906 may also include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 907 may include means, such as a display, speaker, or haptic device, that present information to a user by stimulating the user's senses. The storage unit 908 may include a hard disk, non-volatile or volatile memory, or the like, and stores various information (including programs). The communication unit 909 may be a network interface or the like, and performs wired or wireless communication with the outside world. The drive 910 drives removable media 911, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0074] The processing circuit 901 includes a processor that executes a program, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The processing circuit 901 (the processor) performs the above-described series of processes by loading a program stored in a storage unit 908 into a RAM 903 via an input / output interface 905 and a bus 904 and executing the program. The processing circuit 901 can output the processing results of the series of processes from an output unit 907, for example, via the bus 904 and the input / output interface 905, as necessary. The processing circuit 901 can also store the processing results in the storage unit 908, transmit them from a communication unit 909, etc.

[0075] The program executed by the computer (processing circuitry 901) can be provided by being recorded on removable media 911 such as package media, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0076] In a computer, the program can be installed in the storage unit 908 via the input / output interface 905 by inserting the removable medium 911 into the drive 910. The program can also be received by the communication unit 909 from another device such as a server via a wired or wireless transmission medium and installed in the storage unit 908. Alternatively, the program can be installed in the ROM 902 or the storage unit 908 in advance.

[0077] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0078] The processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

[0079] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0080] When the above-described series of processes are performed by a computer executing a program, the input unit 906 functions as the input unit 31. When the processing circuit 901 (the processor thereof) executes the program, it functions as the segmentation unit 32, the first optimization unit 33, the selection unit 34, the inpainting unit 35, the second optimization unit 36, and the 2D image generation unit 37.

[0081] In this specification, a system refers to one component or a collection of multiple components (devices, modules (components), etc.). Therefore, one or multiple components of a computer, for example, only a processor, or a combination of a processor and memory (for example, only the processing circuit 901, or a combination of the processing circuit 901 and bus 904, etc.) is a system. With respect to a collection of multiple components, it does not matter whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, or a single device housed in a single housing with multiple modules, are both systems. Furthermore, for example, an entire computer, or a combination of a computer and another device such as a server (not shown) is also a system.

[0082] The components (blocks) of the devices illustrated in the present specification are conceptual blocks of functions, and the actual device does not necessarily have to have the configuration shown in the drawings. In other words, the device may have any configuration in which the functions of the illustrated components are divided and / or integrated in any unit, for example, a configuration having a single block in which the functions of all the components are integrated.

[0083] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0084] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0085] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0086] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0087] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0088] The present technology may have the following configurations. (1) An information processing method including: an information processing device generating a 3D Gaussian based on images captured from a plurality of viewpoints and at least one of an obstacle category, which is a category of objects that are obstacles in the images captured from the plurality of viewpoints, and a non-obstacle category, which is a category of objects that are not obstacles. (2) The information processing method according to (1), in which at least one of the obstacle category and the non-obstacle category is designated by a user. (3) The information processing method according to (1) or (2), in which the information processing device further includes: adding a mask to a region of the obstacle in the images captured from the plurality of viewpoints based on at least one of the obstacle category and the non-obstacle category, and the 3D Gaussian is generated based on the images captured from the plurality of viewpoints to which the mask has been added. (4) The information processing method according to (3), in which the information processing device further includes: inpainting the images captured from the plurality of viewpoints to which the mask has been added, and the 3D Gaussian is generated based on the inpainted images captured from the plurality of viewpoints. (5) The information processing method according to (4), further comprising: generating an image set in which the captured images from the plurality of viewpoints to which the masks have been added are arranged in a grid pattern, and the inpainting is performed on the image set. (6) The information processing method according to (5), further comprising: generating an image set in which the captured images from four viewpoints to which the masks have been added are arranged in a grid pattern in the image set.(7) The information processing method according to (4), further comprising: generating a provisional 3D Gaussian from the captured images of the multiple viewpoints to which the masks have been added; and determining whether to select the region corresponding to the provisional 3D Gaussian as a target region for inpainting based on at least one of the number of viewpoints to which the masks have been added to a region in the captured image corresponding to the provisional 3D Gaussian and the number of viewpoints to which the masks have not been added to a region in the captured image corresponding to the provisional 3D Gaussian, wherein the inpainting is performed on the target region in the captured images of the multiple viewpoints to which the masks have been added. (8) The information processing method according to (7), further comprising: the information processing device generating an image set in which the target regions in the captured images of the multiple viewpoints to which the masks have been added are arranged in a grid pattern, wherein the inpainting is performed on the image set. (9) The information processing method according to (8), wherein the target regions of four of the multiple viewpoints are arranged in a grid pattern in the image set. (10) The information processing method according to any one of (3) to (6), further including the information processing device generating a tentative 3D Gaussian from the captured images from the multiple viewpoints to which the mask has been added, wherein the 3D Gaussian is generated using the tentative 3D Gaussian as an initial value. (11) The information processing method according to any one of (1) to (10), further including the information processing device acquiring the captured images from the multiple viewpoints captured by an aerial photography device. (12) The information processing method according to any one of (1) to (11), further including the information processing device generating a 2D image from a predetermined viewpoint using the 3D Gaussian. (13) An information processing device comprising: a generation unit that generates a 3D Gaussian based on the captured images from the multiple viewpoints and at least one of an obstacle category that is a category of objects that are regarded as obstacles in the captured images from the multiple viewpoints and a non-obstacle category that is a category of objects that are not regarded as obstacles in the captured images from the multiple viewpoints.

[0089] REFERENCE SIGNS LIST 11 imaging device, 12 information processing device, 31 input unit, 32 segmentation unit, 33 first optimization unit, 34 selection unit, 35 inpainting unit, 36 second optimization unit, 37 2D image generation unit

Claims

1. An information processing method including an information processing device generating a 3D Gaussian based on images captured from multiple viewpoints and at least one of an obstacle category, which is a category of objects that are obstacles in the images captured from the multiple viewpoints, and a non-obstacle category, which is a category of objects that are not obstacles.

2. The information processing method according to claim 1, wherein at least one of the obstacle category and the non-obstacle category is designated by a user.

3. The information processing method of claim 1, further comprising: the information processing device adding a mask to the obstacle region in the captured images from the multiple viewpoints based on at least one of the obstacle category and the non-obstacle category; and the 3D Gaussian is generated based on the captured images from the multiple viewpoints to which the mask has been added.

4. The information processing method according to claim 3, further comprising: the information processing device inpainting the captured images from the multiple viewpoints to which the mask has been added; and the 3D Gaussian is generated based on the inpainted captured images from the multiple viewpoints.

5. The information processing method according to claim 4, further comprising: the information processing device generating an image set in which the captured images from the multiple viewpoints to which the mask has been added are arranged in a grid pattern; and the inpainting is performed on the image set.

6. The information processing method according to claim 5, wherein in the image set, the captured images with the masks added from four of the plurality of viewpoints are arranged in a grid pattern.

7. The information processing method of claim 4, further comprising: generating a provisional 3D Gaussian from the captured images of the multiple viewpoints to which the masks have been added; and determining whether or not to select the area corresponding to the provisional 3D Gaussian as a target area for inpainting based on at least one of the number of viewpoints to which the masks have been added to an area in the captured image corresponding to the provisional 3D Gaussian and the number of viewpoints to which the masks have not been added to an area in the captured image corresponding to the provisional 3D Gaussian, wherein the inpainting is performed on the target area in the captured images of the multiple viewpoints to which the masks have been added.

8. The information processing method according to claim 7, further comprising: the information processing device generating an image set in which the target areas in the captured images from the multiple viewpoints to which the masks have been added are arranged in a grid pattern; and the inpainting is performed on the image set.

9. The information processing method according to claim 8, wherein in the image set, the target regions of four of the plurality of viewpoints are arranged in a grid pattern.

10. The information processing method according to claim 3, further comprising: the information processing device generating a provisional 3D Gaussian from the captured images from the multiple viewpoints to which the mask has been added; and the 3D Gaussian being generated using the provisional 3D Gaussian as an initial value.

11. The information processing method according to claim 1, further comprising the information processing device acquiring the images from the multiple viewpoints taken by an aerial photography device.

12. The information processing method according to claim 1, further comprising: generating a 2D image of a predetermined viewpoint using the 3D Gaussian, said information processing device.

13. An information processing device having a generation unit that generates a 3D Gaussian based on images captured from multiple viewpoints and at least one of an obstacle category, which is a category of objects that are obstacles in the images captured from the multiple viewpoints, and a non-obstacle category, which is a category of objects that are not obstacles.

Images