Information processing method, information processing device, and program

Abstract

This information processing method, which is executed by one or a plurality of information processing devices, includes performing: camera attitude determination processing for determining, for each of a plurality of camera attitudes for image capture performed when generating a three-dimensional model of an object using a plurality of images obtained by imaging the object with said plurality of different camera attitudes, whether the camera attitude is in a state suitable for three-dimensional model generation; and presentation information generation processing for generating presentation information for a user on the basis of the results of the camera attitude determination processing.

Description

Information processing method, information processing device, and program

[0001] The present technology relates to an information processing method, an information processing device, and a program, and relates to a technology that can be applied in the field of three-dimensional model generation, for example.

[0002] One method for 3D modeling of 3D (three-dimensional) objects having a three-dimensional shape is called photogrammetry, which captures images of the 3D object from multiple directions and generates 3D data based on the resulting captured images. Another method is called real-time 3D modeling, which generates 3D data instantly (in real time) based on captured images, posture information, depth, and other information. Furthermore, in recent years, a method collectively known as neural rendering (e.g., NeRF (Neural Radiance Fields)) has been proposed, which constructs neural fields based on the posture of captured images and generates images or 3D models from any viewpoint.

[0003] The following Patent Document 1 discloses a technology for improving the accuracy of estimating distances to positions on images captured by a camera. Furthermore, the following Patent Document 2 describes a camera capable of fixed-point shooting that compares camera position information, shooting position information of previously captured images, and current and past camera positions to display the difference.

[0004] JP 2022-176839 A JP 2010-183150 A

[0005] When performing 3D modeling based on photographed images of a 3D object, it is desirable to take as many photographs of the 3D object as possible from as many different camera poses (positions and shooting directions) as possible in order to increase the accuracy of the 3D model. On the other hand, to reduce the workload and processing volume, it is desirable to perform photographs efficiently with as few photographs as possible. To perform photographs efficiently, the trajectory of the camera movement and the camera pose during the movement are set, but it is difficult for a user to check whether the actual photographs were taken according to the settings during or immediately after photographing. Therefore, after photographing, the user must wait a certain amount of time or leave the shooting location before checking the generated 3D model, and if the accuracy of the 3D model is insufficient, they must take photographs again. This makes it difficult to perform the photographing work efficiently.

[0006] Therefore, the present disclosure proposes a technology that enables a user to easily and efficiently capture an image of a 3D object using a camera.

[0007] The information processing method according to the present technology is an information processing method that executes, by one or more information processing devices, a camera attitude determination process for determining whether each of multiple camera attitudes taken during image capture is suitable for generating a 3D model of an object using multiple images of the object captured at multiple different camera attitudes, and a presentation information generation process for generating presentation information for a user based on the results of the camera attitude determination process. In the present disclosure, the camera attitude refers to the camera attitude defined by the shooting position and shooting direction. When generating a 3D model from multiple images captured at multiple camera attitudes, obtaining images at multiple appropriate camera attitudes allows for the generation of a highly accurate 3D model. Therefore, the method determines whether the camera attitude actually used for capturing the images is appropriate, and presents information corresponding to the result to the user.

[0008] 1 is an explanatory diagram of an overview of photogrammetry; an explanatory diagram of an overview of real-time 3D modeling; an explanatory diagram of the difference between photogrammetry and real-time 3D modeling; an explanatory diagram of an example of a device configuration according to an embodiment; an explanatory diagram of incremental 3D compositing according to an embodiment; an explanatory diagram of a 3D model generated by incremental 3D compositing according to an embodiment; a block diagram of an information processing device according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; a flowchart of incremental 3D compositing processing according to an embodiment; an explanatory diagram of an example of a display of camera posture according to an embodiment; an explanatory diagram of an example of the functional configuration of an incremental compositing calculation system according to an embodiment; a flowchart of incremental 3D compositing processing with camera posture determination added according to an embodiment. 1 is an explanatory diagram of abnormal camera pose detection according to an embodiment. FIG. 2 is an explanatory diagram of a threshold for abnormal camera pose detection according to an embodiment. FIG. 3 is an explanatory diagram of a transition to reshooting during shooting according to an embodiment. FIG. 4 is an explanatory diagram of a display example of a camera pose according to an embodiment. FIG. 5 is a flowchart of a camera pose presentation process during shooting according to an embodiment. FIG. 6 is a flowchart of a camera pose presentation process during shooting according to an embodiment. FIG. 7 is a flowchart of a camera pose presentation process during shooting according to an embodiment. FIG. 8 is an explanatory diagram of presentation of a camera pose and a shooting trajectory after shooting according to an embodiment. FIG. 9 is an explanatory diagram of presentation of a camera pose and a shooting trajectory after shooting according to an embodiment. FIG. 10 is an explanatory diagram of presentation of a camera pose and a shooting trajectory after shooting according to an embodiment. FIG. 11 is an explanatory diagram of transition of presentation of a camera pose and a shooting trajectory after shooting according to an embodiment. FIG. 12 is a flowchart of presentation of a camera pose and a shooting trajectory after shooting according to an embodiment.FIG. 1 is an explanatory diagram of a calculation example of camera attitude accuracy according to an embodiment. FIG. 2 is an explanatory diagram of a calculation process of camera attitude accuracy according to an embodiment. FIG. 3 is an explanatory diagram of a calculation example of camera attitude accuracy according to an embodiment. FIG. 4 is an explanatory diagram of a calculation process of camera attitude accuracy according to an embodiment. FIG. 5 is an explanatory diagram of a calculation example of camera attitude accuracy according to an embodiment. FIG. 6 is a flowchart of a calculation process of camera attitude accuracy according to an embodiment.

[0009] <1. 3D Modeling> <2. Device Configuration for Incremental 3D Synthesis> <3. System Configuration Example> <4. Incremental 3D Synthesis Processing Example> <5. Camera Attitude Display During Shooting> <6. Camera Attitude Display After Shooting> <7. Camera Attitude Calculation Example> <8. Summary and Modifications>

[0010] In this disclosure, the term "image" includes both still images and moving images. Furthermore, the term "image" refers not only to an image in a displayed state, but also to image data in a signal processing process or a transmission process.

[0011] <1. 3D Modeling> First, photogrammetry will be described. As a method for generating (reconstructing) a model of a three-dimensional shape of an object (also referred to as a 3D object), there is a method called photogrammetry, in which the 3D object is photographed from multiple directions and a 3D model (3D data) is generated based on the multiple captured images. In this disclosure, generating a model of the three-dimensional shape of a 3D object is also referred to as 3D modeling.

[0012] Photogrammetry is a technique for reconstructing a highly accurate three-dimensional model using the principles of triangulation from multiple images taken from various viewpoints. For example, a camera 150 shown in FIG. 1 captures images of a 3D object 100 from multiple viewpoints to obtain multiple captured images. These captured images are then used to perform processes called SfM (Structure from Motion) and MVS (Multi View Stereo), followed by post-processing such as meshing and texturing to generate a 3D model 105.

[0013] In this specification, the term "accuracy" for a 3D model refers not only to the reproducibility of the three-dimensional shape of the target 3D object, i.e., the accuracy and definition of the three-dimensional shape, but also to the reproducibility (accuracy, definition, etc.) of the texture applied to the surface of the 3D model.

[0014] In SfM, for example, corresponding points are searched for between captured images, the position and orientation of the camera are derived using epipolar constraints, and the positions of each corresponding point in three-dimensional space are identified by triangulation based on the camera position and orientation. In this disclosure, these points in three-dimensional space are also referred to as three-dimensional points. In other words, three-dimensional points corresponding to each corresponding point are identified. Then, the entire three-dimensional point group identified in this manner is optimized by bundle adjustment.

[0015] In MVS, for example, a 3D point group derived as described above is used to perform a more detailed search for corresponding points, and 3D points are added.

[0016] As described above, photogrammetry performs a global optimization calculation called bundle adjustment to minimize errors, which allows it to obtain very accurate results, but it also places a heavy burden on the calculations.Furthermore, because it is based on geometric calculations rather than physical measurements, in principle, the higher the resolution of the images used, the more accurate the model can be restored.

[0017] Next, real-time 3D modeling will be described. A different 3D modeling technique from the above-described photogrammetry is called real-time 3D modeling, which generates 3D data instantly (in real time) based on information such as captured images, posture information, and depth. In this technique, as shown in FIG. 2 , for example, a camera 150 captures an image of the 3D object 100 while moving around the periphery of the 3D object 100 as indicated by the dotted line 101. The camera 150 is equipped with not only an image sensor but also a Lidar (Light Detection And Ranging) scanner (dToF (Direct Time of Flight) module), which captures images and detects depth (the distance to the subject).

[0018] In recent years, dToF modules have become smaller and more sophisticated, enabling accurate depth measurement over relatively long distances (e.g., about 5 m) both indoors and outdoors. This has made it easy to experience real-time modeling and capturing at the consumer level.

[0019] Furthermore, the camera 150 is equipped with an inertial sensor that detects inertial information such as the acceleration and angular velocity of the camera 150 .

[0020] In real-time 3D modeling, a three-dimensional orientation calculation process called SLAM (Simultaneous Localization and Mapping) is performed to generate orientation information indicating the position and orientation of the camera 150. In addition, the orientation information and depth are used to update a truncated signed distance function (TSDF), and a 3D model 105 (mesh and texture) is generated by a process called MC (Marching Cubes).

[0021] In SLAM, for example, the position and orientation of the camera 150 are estimated based on captured images and inertial information (self-localization). In TSDF updates, depth is associated with voxels to detect volume. In MC, isosurfaces are calculated using adjacent voxels. With real-time orientation information from SLAM, the volume of a voxel can be detected without using a point cloud by overlaying depth (how far the light rays have reached) from multiple frames. Using a voxel representation makes it possible to estimate viewpoints that are in shadow and must be photographed (missing viewpoints). This makes it possible to detect perforated and protruding structures in the 3D object 100.

[0022] There is also a 3D modeling technique collectively known as neural rendering, which constructs neural fields based on the posture of a captured image and the captured image, and generates images and 3D models from any viewpoint. For example, NeRF (Neural Radiance Fields) has been proposed.

[0023] Each of these 3D modeling techniques has different characteristics, and no technique is superior in all respects. Figure 3 compares the characteristics of photogrammetry and real-time 3D modeling. As shown in Figure 3, photogrammetry uses SfM (including localization) and MVS, while real-time 3D modeling uses localization (SLAM) and TSDF. Furthermore, comparing the data used, photogrammetry uses only image data, while real-time 3D modeling uses depth and orientation data in addition to image data. Furthermore, comparing processing times, photogrammetry requires a long time, from several minutes to several tens of hours, while real-time 3D modeling allows for nearly instantaneous processing at 30 fps (frames per second).

[0024] In addition, when comparing the required computing power, photogrammetry requires computing power at the level of a high-end CPU (Central Processing Unit) and GPU (Graphics Processing Unit), while real-time 3D modeling requires computing power at the level of a mobile AP (Application Processor).In addition, when comparing the resolution of the generated models, photogrammetry is relatively high-resolution, depending on the resolution, number, and shooting method of the captured images, while real-time 3D modeling is relatively low-resolution, depending on the depth and self-position estimation accuracy, etc.

[0025] Furthermore, the internal representation of the generated 3D data is point cloud-based in photogrammetry, whereas it is voxel-based in real-time 3D modeling. Furthermore, while there are no restrictions on subject size or resolution in photogrammetry, real-time 3D modeling is sensor-dependent. Furthermore, when comparing the absolute accuracy of the model, photogrammetry is relatively high because it is optimized using bundle adjustment, while real-time 3D modeling is relatively low, depending on the sensor and self-localization accuracy. Furthermore, when comparing scale, photogrammetry is indefinite (the size is unknown), whereas real-time 3D modeling is unique (the absolute size is known).

[0026] Photogrammetry and real-time 3D modeling have the following differences in characteristics. In other words, applying real-time 3D modeling can reduce the workload and processing volume of 3D modeling compared to applying photogrammetry or neural rendering. However, applying photogrammetry or neural rendering can generate a more accurate 3D model than applying real-time 3D modeling. The technology described in this embodiment can be applied when photographing a 3D object 100 using these various 3D modeling methods.

[0027] 2. Device Configuration for Incremental 3D Composition> Figure 4 shows an example of various device configurations used for 3D modeling according to the embodiment. Incremental 3D composition as referred to in the present disclosure, which will be described in detail later, is a technique in which, in the process of photographing a 3D object 100, each time an image is acquired, the image is additionally input into a 3D modeling process to generate a 3D model.

[0028] In this embodiment, the device configuration for 3D modeling includes a photographing device 1, a UI (User Interface) device 2, and a 3D model generating device 3, as shown in FIG.

[0029] The photographing device 1 is a device that can photograph a 3D object 100 and output an image. A specific example of the photographing device 1 is a mobile body such as a drone 10 equipped with a camera 11. The mobile body equipped with the camera 11 is not limited to the drone 10, and may be other flying bodies, vehicles, ships, submarines, etc. Further, the photographing device 1 may also be a camera 11 that is handheld by a person to take pictures, or an information processing device such as a smartphone 12 equipped with a photographing function.

[0030] The UI device 2 is a device that provides various interfaces to the user. In this case, the interfaces include an input interface for operating the imaging device and an output interface for presenting a 3D model and imaging posture. The UI device 2 may be an information processing device such as a smartphone 12, a personal computer 13, or a tablet device (not shown). The UI device 2 may also be a control device for a moving object such as the drone 10.

[0031] The 3D model generation device 3 is a device that generates a 3D model using captured images. For example, in this embodiment, a device that performs incremental 3D composition is taken as an example. The 3D model generation device 3 can be any of various information processing devices, such as a server device 14 such as a cloud server, a personal computer 13, or a smartphone 12.

[0032] In such a configuration, the photographing device 1 photographs the 3D object 100, and the photographed images are transferred, for example, one by one, sequentially to the 3D model generation device 3. The transfer of images to the 3D model generation device 3 may be performed directly by the photographing device 1, or the photographing device 1 may transfer the images to the UI device 2, and the UI device 2 may transfer the images to the 3D model generation device 3.

[0033] The 3D model generation device 3 generates a 3D model based on the acquired images. In this case, the 3D model generation device 3 performs incremental 3D composition. The 3D model generation device 3 then transmits the 3D model generated at each point in time to the UI device 2. The UI device 2 can display the received 3D model. This allows the user of the UI device 2 to check the 3D model as it gradually grows during shooting.

[0034] 5 shows the difference between normal 3D composition and incremental 3D composition. In normal 3D composition, a large number of images G1, G2, ... G(n) are first taken as a photography task. After the photography is completed, the images G1, G2, ... G(n) are input into a 3D model generation process in a 3D model generation device 3. Depending on the size of the 3D object 100 and the required accuracy, this 3D model generation process takes about half a day to a day.

[0035] Thereafter, a 3D model Mc is generated based on the images G1, G2, ... G(n), and the user can visually confirm a display image dM of the 3D model Mc. In this case, the user cannot determine whether the photographing work was appropriate until checking the display image dM of the 3D model Mc. For example, if the user views the display image dM and finds that some parts are insufficient in accuracy, they may have to take the photograph again.

[0036] On the other hand, incremental 3D synthesis performs 3D synthesis sequentially in parallel with the shooting work. For example, as shown in the figure, images G1, G2, ... G(n) obtained during the shooting work are sequentially transferred to the 3D model generation device 3 and input into the 3D model generation process. This allows the 3D model generation device 3 to perform 3D modeling based on the multiple images acquired at each time point, and to generate 3D models M2, M3, ... M(n) at each time point. For example, as schematically shown in Figure 6, the 3D models M2, M3, M4, ... M(n) gradually grow as the 3D model grows.

[0037] Then, when the 3D model generation device 3 generates a 3D model M2, it transfers it to the UI device 2, allowing the user to check the 3D model M2 at that time using the display image dM2. Also, when the 3D model generation device 3 generates a 3D model M3, it transfers it to the UI device 2, allowing the user to check the 3D model M3 at that time using the display image dM3. After the shooting work is completed, a 3D model M(n) can be generated based on the images G1, G2, ... G(n), allowing the user to check the 3D model M(n) in its state after shooting is complete using the display image dM(n). Therefore, the user can check the progress and final state of the 3D model from during shooting to immediately after shooting. This provides information to determine whether the shooting up to that point was appropriate during or after shooting is completed.

[0038] The 3D modeling process as incremental 3D compositing may be performed in a somewhat simplified manner if it is only for confirmation during shooting, because the high-precision 3D model as the actual finished product can be created after shooting is completed in the same way as normal 3D compositing.

[0039] Incremental 3D compositing allows a user to check a 3D model during or immediately after shooting, allowing the user to immediately evaluate the shooting work and, if insufficient, to reshoot, thereby promoting the efficiency of the overall 3D model production work. However, even if the user actually checks the 3D model and determines that the accuracy is insufficient, it is difficult for the user to know what camera pose or shooting trajectory to use for reshooting. Therefore, in this embodiment, in addition to monitoring the 3D model using incremental 3D compositing, inappropriate camera poses among the camera poses captured are presented to the user as abnormal camera poses, and camera poses that the user wants to capture an image are presented as insufficient camera poses.

[0040] For example, the 3D model generation device 3 can generate information about the camera position and transfer it to the UI device 2, making it easier for the user to understand how to retake the image. This will be described in detail later.

[0041] Up to this point, incremental 3D composition has been described, but the actual device forms of the photographing device 1, UI device 2, and 3D model generation device 3 used to generate the 3D model illustrated in Figure 4 are diverse. The photographing device 1, UI device 2, and 3D model generation device 3 may each be separate devices, or a single device may serve as both. For example, the smartphone 12 may function as both the photographing device 1 and the UI device 2. Furthermore, the UI device 2 and the 3D model generation device 3 may be implemented as a single device, such as the smartphone 12 or personal computer 13. Furthermore, the photographing device 1, UI device 2, and 3D model generation device 3 may be implemented as a single device, such as a single smartphone 12.

[0042] That is, the photographing device 1 may be realized by a device having at least a photographing function, and the UI device 2 and the 3D model generation device 3 may be realized by an information processing device. The 3D model generation device 3 may be an information processing device such as a cloud server, or may be an information processing device capable of communicating with the UI device 2 or the photographing device 1 via a network such as a LAN.

[0043] Here, we will explain information processing devices that can be configured as the 3D model generation device 3 and the UI device 2. For example, various information processing devices such as a server device 14, a smartphone 12, a personal computer 13, or a tablet terminal or workstation (not shown) can be used as the 3D model generation device 3 or the UI device 2. The configuration of an information processing device 70 in these cases is shown in FIG.

[0044] 7 executes various processes in accordance with programs stored in a nonvolatile memory unit 74 such as a read-only memory (ROM) 72 or an electrically erasable programmable read-only memory (EEP-ROM), or programs loaded from a storage unit 79 to a random access memory (RAM) 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes, as appropriate.

[0045] The image processing unit 85 is configured as a processor that performs various types of image processing, such as 3D modeling processing, rendering, database processing, image processing including color and brightness adjustment processing and color conversion processing for captured images, encoding processing, decoding processing, image editing processing, and image analysis and detection processing, or a combination of these processes.

[0046] The image processing unit 85 can be realized by, for example, a CPU separate from the CPU 71, a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), an artificial intelligence (AI) processor, etc. The image processing unit 85 may be provided as a function within the CPU 71.

[0047] The CPU 71, ROM 72, RAM 73, nonvolatile memory unit 74, and image processing unit 85 are interconnected via a bus 83. To this bus 83, an input / output interface 75 is also connected.

[0048] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a trackball, a dial, a touch panel, a touchpad, or a remote controller. The input unit 76 detects a user operation, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0049] A display unit 77, which may be an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 78, which may be a speaker, are connected to the input / output interface 75 either integrally or separately. The display unit 77 displays various types of information. The display unit 77 may be, for example, a display device provided in the housing of the information processing device 70 or a separate display device connected to the information processing device 70. The display unit 77 displays various images, operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), on the display screen based on instructions from the CPU 71.

[0050] The input / output interface 75 may be connected to a storage unit 79 and a communication unit 80 configured from a hard disk drive (HDD) or a solid state drive (SSD).

[0051] The storage unit 79 can store various data and programs. A database can also be configured in the storage unit 79.

[0052] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices such as external DBs, editing devices, and information processing devices via wired / wireless communication, bus communication, and the like.

[0053] A drive 81 is also connected to the input / output interface 75 as needed, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately attached. The drive 81 allows video data, various computer programs, and the like to be read from the removable recording medium 82. The read data is stored in the storage unit 79, and the video and audio contained in the data are output on the display unit 77 and the audio output unit 78. In addition, the computer programs and the like read from the removable recording medium 82 are installed in the storage unit 79 as needed.

[0054] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0055] 3. System Configuration Examples In the present embodiment, various system configuration examples capable of performing incremental 3D compositing will be described. Note that in the configuration examples described below in FIGS. 8 to 19, the "system" shown in each block is not necessarily limited to a single device. The "system" refers to hardware, software, or an operation or processing function realized by a combination of these. For example, it is to be considered that the "systems" shown in FIGS. 8 to 19 are provided as each functional unit within any of the hardware devices such as the image capture device 1, UI device 2, or 3D model generation device 3 shown in FIG. 4.

[0056] 8 shows an example of a configuration in which incremental 3D compositing is performed on a cloud or on-premise server device 14, separate from the shooting site. The figure shows a moving system 40, a shooting system 41, a control system 42, an image transmission system 43, an image reception system 44, an incremental compositing calculation system 45, an information transmission system 46, an information reception system 47, and a display system 48.

[0057] The moving system 40 is a functional unit that moves the camera 11 during shooting, and is realized by a moving body such as the drone 10. The shooting system 41 is a functional unit that shoots the 3D object 100, and is realized by the camera 11, etc.

[0058] The control system 42 is a functional unit that controls the movement of the mobile system 40 and the photography of the photography system 41. The control system 42 transmits control signals to the mobile system 40 to instruct the trajectory to move for photography (photography trajectory) and to instruct movement, causing the mobile system 40 to perform the movement required for photography. For example, the control system 42 sets a flight plan as a photography trajectory for the drone 10 and controls flight according to the flight plan. The control system 42 also sequentially sends photography triggers to the photography system 41 in accordance with flight according to the flight plan, causing the photography system 41 to perform image photography.

[0059] The control system 42 may be provided in a smartphone 12 or the like as a UI device 2, or may be provided in a control device of the drone 10, or the like.

[0060] The photographing trigger may be given from the moving system 40 to the photographing system 41. Alternatively, the photographing system 41 may generate a photographing trigger internally.

[0061] The photographing system 41 photographs in response to a photographing trigger. The photographed image is transmitted and output by the image transmission system 43. At this time, the photographing system 41 or the image transmission system 43 may perform image reduction processing to reduce the amount of data to be transmitted or to simplify 3D modeling. The image transmission system 43 may be provided in a device present at the photographing site, such as the photographing device 1 or the UI device 2.

[0062] The captured images are sequentially transmitted from the shooting site to, for example, a cloud server by the image transmission system 43. Note that instead of transmitting the images, the images may be sent to the cloud or an on-premise information processing device by transferring a physical recording medium, such as a memory card, on which the images are recorded.

[0063] The image receiving system 44, the incremental synthesis calculation system 45, and the information transmitting system 46 are provided in the 3D model generating device 3, for example.

[0064] When the captured image is received by the image transmission system 43, the incremental synthesis calculation system 45 performs processing as incremental 3D synthesis, generating point cloud data and camera pose information as the 3D model up to that point. The camera pose information includes an estimated camera pose estimated as the camera pose at the time of capture, an abnormal camera pose which is an inappropriate pose among the estimated camera poses, an insufficient camera pose which is a camera pose that recommends additional capture to improve the accuracy of the 3D model, and an accuracy score which indicates the accuracy of each region of the 3D model.

[0065] The point cloud data, camera attitude information, etc. generated by the incremental synthesis calculation system 45 are transmitted by an information transmission system 46 and received by an information receiving system 47. Then, a display system 48 performs display based on the point cloud data, camera attitude information, etc., i.e., displays a 3D model, displays information related to the camera attitude, etc. The information receiving system 47 and display system 48 are provided in the UI device 2. The above is one example of a system configuration.

[0066] Differences from the previously described system configurations will be described from Figure 9 onwards. Like Figure 8, Figure 9 also shows an example configuration in which incremental 3D composition is performed on a cloud or on-premise server device 14, etc., separate from the shooting site. What differs from Figure 8 is the mobile system 40A. The mobile system 40A shows a configuration in which a person manually moves or changes the posture of the camera 11, or a configuration in which a person holds the camera 11 and takes pictures.

[0067] For example, a person holds the camera 11 in his / her hand and takes pictures while moving around the periphery of the 3D object 100. The rest is the same as in Fig. 8, and the taken images are transmitted by the image transmission system 43 and provided to the incremental synthesis calculation system 45 via the image receiving system 44.

[0068] 10 shows an example in which display is performed by a display system 48 on-premise. A moving system 40, a photographing system 41, a control system 42, and an image transmission system 43 are provided in the equipment at the photographing site, and photographed images are sequentially transmitted. An image receiving system 44, an incremental synthesis calculation system 45, and a display system 48 are provided on-premise. As a result, a display based on the point cloud data and camera attitude information calculated by the incremental synthesis calculation system 45 is displayed on the display system 48 provided in the on-premise UI device 2.

[0069] 11 shows an example in which, similar to FIG. 10 , when the camera 11 is manually moved as in FIG. 9 , display is performed, for example, by a display system 48 on-premise. A movement system 40A, an imaging system 41, and an image transmission system 43 are provided in the equipment at the imaging site, and captured images are transmitted sequentially. An image receiving system 44, an incremental synthesis calculation system 45, and a display system 48 are provided on-premise. As a result, a display based on the point cloud data and camera attitude information calculated by the incremental synthesis calculation system 45 is displayed on the display system 48 provided in the on-premise UI device 2.

[0070] 12 shows an example in which the incremental composition calculation system 45 is provided in an edge terminal, that is, an information processing device such as a terminal device provided at a photography site. Images captured by a moving system 40, a photography system 41, and a control system 42 are sequentially input to the incremental composition calculation system 45 of the on-site equipment. Note that either the photography system 41 or the incremental composition calculation system 45 may perform image reduction processing to simplify 3D modeling.

[0071] The incremental compositing calculation system 45 is provided with a calculation unit 50 that performs processes such as incremental compositing and camera attitude determination, and a display unit 60 that displays as a user interface. The calculation unit 50 performs incremental 3D compositing and the like to obtain point cloud data and camera attitude information, and the display unit 60 displays the data accordingly. This allows the 3D model and camera attitude to be checked while shooting at a shooting location, etc.

[0072] 13 shows an example in which, similar to FIG. 12, an incremental composition calculation system 45 is provided in an information processing device at the photography site as an edge terminal when the camera 11 is manually moved as in FIG. 9. Images captured by a moving system 40A and a photography system 41 are sequentially input to the incremental composition calculation system 45 of the on-site equipment. In the incremental composition calculation system 45, a calculation unit 50 performs incremental 3D composition and the like to obtain point cloud data and camera attitude information, which are then displayed on a display unit 60.

[0073] 14 shows an example in which an incremental synthesis calculation system 45 is provided on an edge terminal and a display system 48 is provided separately. Images captured by a mobile system 40, an imaging system 41, and a control system 42 are sequentially input to the incremental synthesis calculation system 45 of the field equipment. The incremental synthesis calculation system 45 generates point cloud data and camera attitude information. This information is transmitted by an information transmission system 46, received by an information receiving system 47, and transferred to a display system 48. The display system 48 displays the point cloud data and camera attitude information according to the data.

[0074] 15 shows an example in which, in a case where the camera 11 is moved manually as in FIG. 9, the incremental synthesis calculation system 45 is provided in an information processing device as an edge terminal as in FIG. 14, and a display system 48 is provided separately. Images captured by the moving system 40A and the imaging system 41 are sequentially input to the incremental synthesis calculation system 45 of the field equipment, and point cloud data and camera attitude information are generated. This information is transmitted by an information transmission system 46, received by an information receiving system 47, and transferred to a display system 48. The display system 48 displays the point cloud data and camera attitude information according to the point cloud data and camera attitude information.

[0075] 16 shows an example in which an incremental compositing calculation system 45 is provided in the cloud and linked to the edge side. In this case, an incremental compositing pre-processing calculation system 49 is provided in an information processing device on the edge side, for example, on the shooting site. The incremental compositing pre-processing calculation system 49 performs processing to extract information necessary for 3D modeling, such as feature amounts, from captured images.

[0076] In this case, the incremental synthesis preprocessing calculation system 49 sequentially calculates feature quantities and the like for the captured images. The calculation results are transferred by the information transmission system 58 to the incremental synthesis calculation system 45 via the information reception system 59 on the cloud side.

[0077] The incremental compositing calculation system 45 performs incremental 3D compositing based on the pre-processing calculation results, generating point cloud data and camera attitude information. This information is transmitted by an information transmission system 46, received by an information reception system 47, and transferred to an edge-side display system 48. The display system 48 displays the point cloud data and camera attitude information according to the data. By transmitting the pre-processing calculation results by the incremental compositing pre-processing calculation system 49 to the cloud side, the amount of data transferred can be reduced compared to when transmitting images.

[0078] 17 shows an example in which, in a case where the camera 11 is moved manually as in Fig. 9, an incremental composition calculation system 45 is provided in the cloud and cooperates with the edge side, similar to Fig. 16. The processing after the incremental composition pre-processing calculation system 49 is the same as in Fig. 16.

[0079] 18 is an example of a system in which the edge side and the cloud side cooperate with each other, similar to FIG. 16 , except that an information transmission system 58 from the edge side transmits both the calculation results of the incremental combining preprocessing calculation system 49 and the captured image to the cloud side. FIG. 19 is an example of a system in which the edge side and the cloud side cooperate with each other, similar to FIG. 17 , except that an information transmission system 58 from the edge side transmits both the calculation results of the incremental combining preprocessing calculation system 49 and the captured image to the cloud side. The information transmission system 58 may transmit the input calculation results and images after performing a process such as selecting and rejecting them based on predetermined criteria.

[0080] In this embodiment, various system configuration examples such as those described above are conceivable, and various system configurations other than those described above are also conceivable.

[0081] 4. Example of Incremental 3D Compositing Processing An example of incremental 3D compositing processing executed in the system of the embodiment will be described with reference to Fig. 20. Note that Fig. 20 shows the processing divided into phases PH1, PH2, PH3, and PH4.

[0082] The processing example in FIG. 20 is an example in accordance with the system configuration in FIG. 8, where phase PH1 is processing by the imaging system 41 provided in the imaging device 1, phase PH2 is processing by the image transmission system 43, phase PH3 is processing by the 3D model generation device 3 (image receiving system 44, incremental synthesis calculation system 45, information transmission system 46), and phase PH4 is processing by the UI device 2 (information receiving system 47 and display system 48).

[0083] In step S1, photography is performed by the photography system 41. During photography, the process of Fig. 20 is performed each time a photographed image is obtained. The photographed image is subjected to reduction processing, for example, by the image transmission system 43 in step S11, and then transmitted to the 3D model generation device 3 in step S12.

[0084] On the 3D model generating device 3 side, when the image receiving system 44 receives an image in step S20, processing is performed by the incremental synthesis calculation system 45. The incremental synthesis calculation system 45 extracts feature points from the image in step S21, and matches the feature points between the currently received image and images previously received in step S22.

[0085] In step S23, the incremental synthesis calculation system 45 estimates the camera orientation at the time of capturing the currently received image based on the matching results of the multiple images, and stores the estimated camera orientation at step S24. The estimated camera orientation is stored as the orientation of the camera 11 at the time of capturing the currently received image.

[0086] In step S25, the incremental synthesis calculation system 45 performs stereo matching of the multiple images including the current new image to estimate a depth image. In step S26, the incremental synthesis calculation system 45 performs integration processing of the depth values, and based on the integration processing, generates point cloud data having three-dimensional position information and pixel information, which is stored in step S27. This point cloud data becomes information for forming a 3D model based on the multiple images up to the currently received image.

[0087] The estimated camera attitude and point cloud data generated and stored as described above are transmitted to the UI device 2 by the information transmission system 46 in step S28.

[0088] On the UI device 2 side, when the estimated camera attitude and point cloud data are received by the information transmission system 46 in step S51, the display system 48 displays a 3D model and estimated camera attitude according to them in step S52.

[0089] Note that steps S25, S26, and S27 enclosed by dashed lines in the figure may not be executed in some cases, such as when the UI device 2 has low processing power and displays only the estimated camera orientation and does not display the 3D model.

[0090] 21 shows an example of a screen display by the display system 48 in the UI device 2. For example, the UI device 2 displays a live view of the image captured by the camera 11 during shooting, but the screen display can be switched to a preview screen 130 of the 3D model by a user operation or the like.

[0091] The preview screen 130 displays a 3D model image 131 based on the point cloud data generated by incremental 3D synthesis, as well as a frustum as an estimated camera pose 20. The estimated camera pose 20 is, for example, a quadrangular pyramid shape, and represents the camera position (three-dimensional position) and shooting direction when capturing the image. This preview screen 130 allows the user to check the camera poses that have been used to capture images and the generated 3D model.

[0092] Although not shown in Fig. 21, the surface of the 3D model image 131 can be colored to represent accuracy by the color. For example, the darker the green, the higher the score for image quality or information amount, and the darker the red, the lower the score.

[0093] In this embodiment, not only is the estimated camera pose 20 displayed, but camera poses that deviate from the original desirable pose are also displayed as abnormal camera poses 21. Furthermore, camera poses that are recommended for photographing in order to improve the accuracy of the 3D model are displayed as insufficient camera poses 23, and poses that are not the original poses but are acceptable are displayed as acceptable camera poses 22. The estimated camera pose 20, abnormal camera pose 21, acceptable camera pose 22, and insufficient camera pose 23 are displayed in different display modes, such as the color, brightness, size, and blinking of the frustum, so that the user can recognize each of them.

[0094] In the processing example of Figure 20, incremental 3D synthesis processing was explained without mentioning the processing for displaying abnormal camera posture 21, acceptable camera posture 22, and insufficient camera posture 23, but from here on, the processing for displaying these camera postures will be explained.

[0095] 5. Display of camera attitude during shooting> Incremental 3D composition allows a 3D model image 131 such as that shown in Fig. 21 to be displayed during or immediately after shooting, but by allowing the user to recognize at an early stage whether the estimated camera attitude is appropriate, reshooting can be performed efficiently. First, the display of the camera attitude during shooting will be described.

[0096] 22 shows the processing functions of the incremental synthesis calculation system 45. The incremental synthesis calculation system 45 includes an incremental synthesis calculation unit 51, a camera attitude determination unit 52, a presentation information generation unit 53, and a storage unit .

[0097] The incremental composition calculation unit 51 is a function that performs processing as the incremental 3D composition described in Fig. 20. The camera attitude determination unit 52 is a processing function that determines the estimated camera attitude 20 for capturing each image and also determines the abnormal camera attitude 21, the allowable camera attitude 22, and the insufficient camera attitude 23. The presentation information generation unit 53 is a function that generates camera attitude information for displaying the estimated camera attitude 20, the abnormal camera attitude 21, the allowable camera attitude 22, and the insufficient camera attitude 23 as presentation information to the user. The storage unit 54 is a function that stores various information during these various processes.

[0098] 22 is just an example, and the display system 48 may have the presentation information generation unit 53, for example. In terms of a specific device configuration, the 3D model generation device 3 in Fig. 4 may have the functions of Fig. 22, or the 3D model generation device 3 and the UI device 2 may each have some of the functions of Fig. 22.

[0099] Fig. 23 shows a processing example of the 3D model generation device 3 (incremental synthesis calculation system 45, etc.) as phase PH3 in Fig. 20. The processing from step S20 to step S27 in Fig. 20 can be said to be mainly processing by the incremental synthesis calculation unit 51 in Fig. 22, but the processing example in Fig. 23 is obtained by adding processing by the camera attitude determination unit 52 and the presentation information generation unit 53 to this.

[0100] The processes of steps S20 to S27 are as described above. In step S31, the incremental synthesis calculation system 45 generates and stores point cloud data during camera posture estimation. The point cloud data in this case is point cloud data having three-dimensional position information including distance information measured based on matching of feature points such as edge portions within the image. The point cloud data of step S27 described above is a point cloud for each pixel of the image, but the point cloud data of step S31 is a point cloud of feature points within the image, and can be said to be point cloud data that partially covers the image plane.

[0101] In step S32, the incremental synthesis calculation system 45 temporarily stores the data obtained in steps S25, S26, and S27 as intermediate data, that is, depth information within the image and point cloud data for each pixel.

[0102] In step S33, the incremental composition calculation system 45 may acquire information about a photographing trajectory. The photographing trajectory is information about a trajectory along which a moving body such as the drone 10 moves during photographing, which is equivalent to a flight plan in the case of the drone 10. The photographing trajectory, such as a flight plan, can be generated by the incremental composition calculation system 45 or another system based on user settings, or can be automatically generated.

[0103] In step S34, the incremental synthesis calculation system 45 sets various threshold information for processing. The thresholds will be described later with reference to FIG.

[0104] In step S30, the incremental composition calculation system 45 performs a camera attitude determination process. In this camera attitude determination process, the incremental composition calculation system 45 determines whether the estimated camera attitude (step S24) that estimated the camera attitude for the current captured image is appropriate. As a result of the appropriateness determination, the system determines an abnormal camera attitude, an insufficient camera attitude, or an acceptable camera attitude (an abnormal camera attitude but within an acceptable range), generates data on the determination results, and stores them in steps S35, S36, and S37.

[0105] Information on these abnormal camera postures, insufficient camera postures, and acceptable camera postures, as well as the estimated camera postures obtained in step S24, and the point cloud data in step S27, are transmitted to the UI device 2 by the information transmission system 46 in step S28 as information to be presented to the user.

[0106] On the UI device 2 side, the display system 48 displays the corresponding 3D model, estimated camera posture, abnormal camera posture, insufficient camera posture, and allowable camera posture in steps S51 and S52 shown in FIG.

[0107] The display state on the UI device 2 side during image capture, in which such processing is performed for each image, is explained using a schematic diagram. As shown in FIG. 24A , for example, a drone 10 flies along an image capture trajectory 26 indicated by a dashed line relative to a 3D object 100, and images are captured sequentially during the process. In this case, the estimated camera posture 20 at the time of capture for each image can be displayed using a frustum or the like, as shown in FIG. 24B . However, due to factors such as wind, it is not always possible to capture images from the position or direction of the planned image capture trajectory 26.

[0108] At this time, for an image that is off the shooting trajectory 26 and does not capture the target range of the 3D object 100 within the angle of view, the shooting posture can be displayed as an abnormal camera posture 21. Furthermore, for an image that is off the shooting trajectory 26 but is acceptable because the target range of the 3D object 100 is captured within the angle of view, the shooting posture can be displayed as an acceptable camera posture 22. Furthermore, for an abnormal camera posture 21, a camera posture that should be retaken can be displayed as an insufficient camera posture 23.

[0109] During shooting, the UI device 2 not only displays the estimated camera posture 20 normally as described above, but also displays in a different display mode those of the estimated camera posture 20 that have been determined to be abnormal camera postures 21 or acceptable camera postures 22, thereby making it possible to inform the user of the status of the shooting that is currently being performed. It is also preferable to display the insufficient camera posture 23 as needed.

[0110] The determination of whether the camera posture is appropriate for these displays in step S30 in FIG. 23 is made based on a threshold value set in step S34, and an example of this setting will be described with reference to FIG.

[0111] The camera posture determination elements include rotation (photographing direction) 91, position (three-dimensional position) 92, and overlap amount 93, and thresholds th1, th2, and th3 are set for each of these elements. The overlap amount is the amount of area where the photographed portion of the 3D object 100 (image within the angle of view) overlaps with images photographed at other times. In terms of the accuracy of the generated 3D model, it is necessary for each area of ​​the 3D object 100 to overlap with multiple images to a certain extent.

[0112] The parameters 90 for setting the threshold values ​​th1, th2, and th3 as the rotation 91, position 92, and overlap amount 93 are based on various pre-stored settings, for example. In other words, the threshold values ​​th1, th2, and th3 can be considered to be set according to the respective modes.

[0113] For example, in the incremental synthesis calculation system 45, threshold setting parameters are stored in advance as an inspection setting 95, a video production setting 96, and a general setting 97 for content production. One of the settings is selected based on the user's intended use, etc. For example, the inspection setting 95 is a setting with stricter thresholds for inspection. The video production setting is a setting for generating a highly accurate 3D model. The general setting is a setting with relatively relaxed accuracy requirements, etc. Note that the user may be able to adjust the threshold settings for the inspection setting 95, video production setting 96, general setting 97, etc., or save new custom user settings.

[0114] Parameters 90 are read as the selected settings, and thresholds th1, th2, and th3 are set for processing as rotation 91, position 92, and overlap amount 93, respectively. For the current estimated camera pose 20 detected in step S24 of Fig. 23, the incremental synthesis calculation system 45 compares the rotation direction value with threshold th1, the three-dimensional position value with threshold th2 (x, y, z position), and the overlap amount with threshold th3. As a result, it is determined whether the estimated camera pose is appropriate, abnormal, or acceptable.

[0115] 26A to 26D are schematic diagrams showing examples of displays on the UI device 2 during shooting. For example, assume that the live view screen 120 shown in Fig. 26A is displayed continuously by the camera 11 mounted on the drone 10 while the drone 10 is flying and shooting images, even when the shutter is not released. The figures show a state in which part of the side of the cylindrical 3D object 100, which is the subject of the image, is displayed.

[0116] If, at a certain point in time, the display system 48 of the UI device 2 receives information about an abnormal camera posture 21 from the incremental synthesis calculation system 45, a message 121 such as "There is an abnormality in the shooting" is displayed on the live view screen 120 as shown in the figure, informing the user that an abnormality has occurred.

[0117] For example, the user can operate a switching button 122 on the live view screen 120 to arbitrarily switch from the live view screen 120 to the preview screen 130. By operating the switching button 122 in response to the message 121 and switching to the live view screen 120, the user can check the details of the abnormality.

[0118] When the user operates the switching button 122, the display system 48 of the UI device 2 switches the screen to the preview screen 130 of Fig. 26B, and displays the 3D model 105, the estimated camera pose 20, the abnormal camera pose 21, the allowable camera pose 22, the insufficient camera pose 23, etc. Note that by displaying the current camera pose 24, that is, the latest estimated camera pose 20 detected in step S24 of Fig. 23, the user can grasp the current camera position.

[0119] In this state, when the user performs an operation to select an abnormal camera posture 21, the display system 48 displays a message 135 for recovery from the abnormal camera posture 21 and a shooting trajectory 26, as shown in Fig. 26C. The display system 48 displays, as the message 135, content asking the user whether or not to take a new photograph, and a response button for that question. There are also cases where the shooting trajectory from the current camera posture 24 is presented as the shooting trajectory 26 to be corrected for a new photograph.

[0120] The user sees this display, decides whether or not to retake some of the images, and responds to message 135. If the user responds that they will retake the images, control system 42 in UI device 2 outputs a control signal to drone 10 (movement system 40) to correct the shooting trajectory, and display system 48 in UI device 2 displays message 136 and current camera attitude 24 during movement, as shown in FIG. 26D .

[0121] After that, movement and photography are actually performed along the corrected photography trajectory 26. For example, photography is performed with the insufficient camera attitude 23, and then the camera returns to the original photography trajectory, and subsequent photography is continued. These situations are displayed on the preview screen 130.

[0122] In the above description, when the user specifies a certain abnormal camera posture 21 on the preview screen 130 in FIG. 26B, the message 135 in FIG. 26C is displayed and the user is given an opportunity to proceed with retaking the image. However, for example, the message 135 in FIG. 26C may be displayed in response to the operation of the switch button 122 on the live view screen, and the user may be given an opportunity to proceed with retaking the image.

[0123] A description will be given of a display example of the missing camera pose 23 and the 3D model image 131 on the preview screen 130. Fig. 27A shows an example in which the estimated camera pose 20 is displayed around the 3D model image 131 and the missing camera pose 23 is also displayed.

[0124] The 3D model image 131 is displayed with regions differentiated by color (or density) as high-precision regions 28, low-precision regions 29, etc. The high-precision regions 28 represent regions with a high precision index, i.e., regions with high precision of the 3D model. The low-precision regions 29 represent regions with a low precision index, i.e., regions with low precision of the 3D model. A low-precision region 29 occurs when there are few images of that region, when the images of that region are not taken with the camera in an appropriate position, or when the overlap rate of the images of that region is low, etc.

[0125] By showing the high-precision areas 28 and low-precision areas 29, the accuracy of the current 3D model and areas where photography is insufficient can be presented to the user. It is also desirable to display the insufficient camera pose 23 and prompt the user to take a photograph (retake) at that camera pose. In other words, showing the accuracy of each area by differences in color and density in the 3D model image 131 and displaying the insufficient camera pose 23 is advantageous for helping the user decide whether to retake the photograph.

[0126] As shown in Figure 27B, when a shooting trajectory 26 for retaking a picture of the missing camera posture 23 is displayed, the accuracy improvement of the 3D model resulting from retaking the picture using this shooting trajectory 26 may be displayed. For example, in Figure 27B, the area that was a low-accuracy area 29 in Figure 27A has become a high-accuracy area 28. By displaying such an improvement in accuracy together with the shooting trajectory 26, the user can be provided with appropriate information for making a decision.

[0127] 27C also shows a case where the user selects the insufficient camera pose 23. For example, by the user specifying one of the two insufficient camera poses 23, the accuracy change caused by capturing an image at that insufficient camera pose 23 is presented. That is, in FIG. 27C, part of the area that was the low-accuracy area 29 in FIG. 27A has become the high-accuracy area 28. Note that, although the case where the insufficient camera pose 23 is selected has been described here, if the user specifies the abnormal camera pose 21 or the allowable camera pose 22, the change in accuracy of the 3D model caused by capturing an image at the alternative insufficient camera pose 23 may be similarly displayed.

[0128] As described above, by displaying the change in accuracy of the 3D model together with regard to retaking the image, the user can more easily determine whether or not to retake the image.

[0129] An example of processing by the UI device 2 (display system 48) for a UI such as that shown in Figures 26A to 26D will be described with reference to Figure 28. Note that Figure 28 shows processing performed when an abnormal camera posture 21 is detected while the live view screen 120 is being displayed on the UI device 2. The display system 48 repeatedly executes the processing shown in Figure 28 during image capture.

[0130] In step S200, the display system 48 determines whether or not an abnormality has occurred during image capture. Specifically, the display system 48 determines whether or not information about the abnormal camera posture 21 has been transmitted from the incremental synthesis calculation system 45.

[0131] If information on the abnormal camera posture 21 has been received, the display system 48 proceeds to step S201 and displays an alert. For example, the display system 48 displays the message 121 in FIG. 26A to notify the user that an abnormality has occurred in the image capture. In step S202, the display system 48 checks a display request for a 3D screen. For example, the display system 48 monitors the user's operation of the switching button 122.

[0132] When a request to display a 3D screen is recognized as an operation of the switching button 122, the display system 48 proceeds from step S2021 to step S204, and determines whether multiple abnormal camera postures 21 have been transmitted from the incremental synthesis calculation system 45 up to this point, and branches the processing accordingly.

[0133] If there is currently one abnormal camera pose 21, the display system 48 proceeds to step S221, transitions the display to the preview screen 130, and displays the 3D model image 131 and the camera poses. For example, the estimated camera pose 20 and the current camera pose 24 are displayed, and also the abnormal camera pose 21, the allowable camera pose 22, and the insufficient camera pose 23 are displayed.

[0134] In step S222, the display system 48 monitors the user's operation of specifying the abnormal camera posture 21, etc., as described in Fig. 26B. If the user performs another operation on the preview screen 130, such as an operation to rotate the 3D model image 131, another prepared operation, or an operation to return to the live view screen 120, the process proceeds from step S225 to a process corresponding to the operation. If the preview screen 130 is maintained, the process corresponding to the operation is performed, and then the process returns to step S222.

[0135] If the user has performed an operation to specify the abnormal camera posture 21, the display system 48 proceeds to step S223, where it displays a message 135 asking whether or not to retake the image, as shown in Fig. 26C. Then, the display system 48 confirms the user's response of whether to retake the image or to cancel in step S224. If the user has selected to cancel, the display system 48 proceeds to step S225.

[0136] If the user instructs the display system 48 to retake the image, the display system 48 proceeds to step S230 and updates the display to allow the image to be retaken. For example, the display system 48 displays the message 136 described in FIG. 26D to notify the user that the image should be retaken. In this case, the control system 42 in the UI device 2 transmits a control signal to the drone 10 to instruct the drone 10 to take a new image capture trajectory 26 and cause the drone 10 to retake the image.

[0137] If there are multiple abnormal camera postures 21, the process of the display system 48 proceeds from step S204 to step S205, where the display is transitioned to the preview screen 130, and the 3D model image 131 and the camera postures are displayed. This display is similar to that of step S221.

[0138] In step S206, the display system 48 monitors the user's operation of specifying the abnormal camera posture 21, etc., as described in Fig. 26B. If the user performs another operation on the preview screen 130, such as an operation to rotate the 3D model image 131, another prepared operation, or an operation to return to the live view screen 120, the process proceeds from step S209 to a process corresponding to the operation. If the preview screen 130 is maintained, the process corresponding to the operation is performed, and then the process returns to step S206.

[0139] If the user performs an operation to specify an abnormal camera posture 21, the display system 48 proceeds to step S207 and displays a message 135 asking whether or not to retake the image, as shown in Fig. 26C. In this case, the display system 48 displays a shooting trajectory 26 for retaking the image of the plurality of abnormal camera postures 21, for example, a flight plan of the drone 10.

[0140] Note that the imaging trajectory 26 may not be displayed in response to the specification of one abnormal camera pose 21, but may be displayed when the user subsequently specifies other abnormal camera poses 21, resulting in a state in which a plurality of abnormal camera poses 21 have been specified. Alternatively, the imaging trajectory 26 may be displayed in response to the specification of one abnormal camera pose 21, covering all of the existing abnormal camera poses 21.

[0141] In step S208, the display system 48 confirms whether the user has answered "OK" or "Cancel" to the message 135. If the answer is "Cancel", the process proceeds to step S209.

[0142] If the user instructs the display system 48 to retake the image, the display system 48 proceeds to step S230 and updates the display for retaking the image. The control system 42 also transmits a control signal to the drone 10 to instruct the drone 10 to take a new image trajectory 26 and retake the image.

[0143] In the example of FIG. 28 above, when there are multiple abnormal camera attitudes 21 and multiple locations to be re-shot, the shooting trajectory 26 (flight plan) is displayed in step S207. When there is only one location to be re-shot, the user can roughly predict the shooting trajectory because they only need to move from the position indicated by the current camera attitude 24 to the location to be re-shot. Therefore, by not displaying the shooting trajectory 26 in step S223, the display can be simplified. However, the shooting trajectory 26 may also be displayed when there is only one location to be re-shot. In this case, there is no need to branch the process in step S204, and the process may always proceed from step S202 to step S205.

[0144] Fig. 29 shows another example of processing. Note that the same steps as in Fig. 28 are assigned the same step numbers and their explanations are omitted. Fig. 29 shows an example of transitioning to the image in Fig. 26C without passing through the image in Fig. 26B.

[0145] If the process proceeds from step S200 to step S204 and proceeds to step S221, message 135 in Fig. 26C is displayed in step S223 and a user operation is awaited. Also, if the process proceeds to step S205, message 135 in Fig. 26C is displayed in step S207 and a user operation is awaited. The rest is the same as in Fig. 28. According to the processing example in Fig. 29, the user operation when retaking a photo can be simplified.

[0146] Fig. 30 shows yet another example of processing. In Fig. 30, steps S206 and S222 in Fig. 28 are replaced with steps S240 and S241. Other processing is the same as in Fig. 28.

[0147] In steps S240 and S241, the display system 48 determines whether or not to retake the image at this time. For example, if the accuracy score of the 3D model is equal to or greater than a predetermined value at this time, the display system 48 determines not to retake the image, and proceeds to step S209 or step S225.

[0148] If the accuracy score of the current 3D model is less than the predetermined value at the time of step S240 or step S241, the display system 48 determines that recapture is necessary, and proceeds to step S207 or step S223.

[0149] In this way, it is possible to automatically determine whether or not reshooting is necessary, and to perform reshooting as necessary. Note that, although the display system 48 has been described as determining whether or not reshooting is necessary, for example, the incremental synthesis calculation system 45 may determine whether or not reshooting is necessary and notify the display system 48 of the determination result. In this case, in step S240 or step S241, the process may be branched based on the notification.

[0150] 6. Display of Camera Posture After Shooting Next, the display of the camera posture and the suitability determination after shooting, for example, immediately after shooting while staff are still present at the shooting location, will be described.

[0151] 31 shows an example of the display of 3D model images 131 and camera poses on the preview screen 130 after shooting along the set shooting trajectory has been completed and the required number of images has been taken. Along with the 3D model image 131, estimated camera poses 20 at each location where the images were taken are displayed, some of which are displayed as abnormal camera poses 21, and missing camera poses 23 to be used in place of the abnormal camera poses 21 are also displayed.

[0152] In such a case, for example, the display system 48 displays shooting trajectories 26A and 26B for reshooting, as shown in Fig. 32. The shooting trajectories 26A and 26B can be generated by a system that acquires camera pose information such as the estimated camera pose 20, the abnormal camera pose 21, and the missing camera pose 23. For example, they can be generated by the display system 48 or the incremental synthesis calculation system 45. For example, these systems determine a movement course that enables the missing camera poses 23 to be photographed in place of all the abnormal camera poses 21, and generate the shooting trajectories 26A and 26B.

[0153] Only one shooting trajectory 26 may be presented to the user, but it is preferable to present multiple shooting trajectories 26 to the user and allow the user to select one, as shown in Fig. 32. For example, the user can select between shooting trajectory 26A, which is a course that requires a long time for reshooting but improves the accuracy of the 3D model, and shooting trajectory 26B, which requires a short time for reshooting but improves the accuracy of the 3D model to a lesser extent.

[0154] In this case, when one of the shooting trajectories is recommended to the user, the shooting trajectory may be emphasized by making the line of the shooting trajectory thicker or by using a more conspicuous color. Also, multiple shooting trajectories 26 may be displayed as inspection settings, video production settings, general settings for content production, etc.

[0155] 33 shows an example in which a user arbitrarily creates a shooting trajectory 26C. For example, the user sets the shooting trajectory 26C by drawing a trajectory on the screen so as to cover a part of the missing camera pose 23. Alternatively, the user may select a missing camera pose 23 that he or she wishes to reshoot, and the shooting trajectory 26C may be automatically generated so that shooting at that missing camera pose 23 becomes possible.

[0156] When the user arbitrarily creates the shooting trajectory 26C in this way, an abnormal camera pose 21 in which shooting is not performed along the shooting trajectory 26C and a corresponding missing camera pose 23 may occur. Therefore, such abnormal camera poses 21 and missing camera poses 23 may be highlighted by, for example, indicating them with a marker 137, or by displaying the abnormal camera poses 21 and missing camera poses 23 themselves in a conspicuous color or brightness or by blinking them. This allows the user to notice that the shooting trajectory 26C should be corrected, or to check areas that do not require reshooting.

[0157] 34A, 34B, and 34C show examples of presentation after imaging. For example, the display system 48 automatically transitions to the preview screen 130 of FIG. 34A after imaging is completed. Then, imaging trajectories 26A and 26B are calculated and displayed as shown in FIG. 34B. If the user selects imaging trajectory 26B, imaging trajectory 26B is displayed as shown in FIG. 34C, and re-imaging can be performed based on this.

[0158] 35 shows an example of processing performed by the UI device 2 (display system 48, control system 42) when image capture is completed. When image capture is completed, the display system 48 automatically transitions to the preview screen 130 in step S300, where the camera postures are displayed. For example, as shown in FIG. 34A , all estimated camera postures 20, abnormal camera postures 21 and allowable camera postures 22 among them, and insufficient camera postures 23 are displayed.

[0159] In step S301, the display system 48 acquires each missing camera pose. For example, all missing camera poses 23 transmitted from the incremental synthesis calculation system 45 during shooting and stored are acquired as processing targets for calculating the shooting trajectory 26.

[0160] In step S302, the display system 48 calculates the shooting trajectory 26. For example, the display system 48 calculates a plurality of shooting trajectories 26 using indexes such as accuracy and shooting time based on the missing camera attitude 23. For example, the display system 48 generates a plurality of shooting trajectories 26, such as a flight plan that prioritizes accuracy, a flight plan that prioritizes shooting time, and a recommended flight plan.

[0161] In step S303, the display system 48 displays the generated multiple shooting trajectories 26 (26A, 26B) as shown in FIG. 34B. In step S304, the display system 48 waits for the user to make a selection. When the user selects a shooting trajectory 26, the display system 48 proceeds to step S305 and updates the display so that the selected shooting trajectory 26 (shooting trajectory 26B in the figure) is displayed as shown in FIG. 34C. The control system 42 also transmits a control signal to the drone 10, instructing it to use the selected shooting trajectory 26 and perform reshooting.

[0162] 36A, 36B, and 36C show other examples of presentation after shooting. For example, the display system 48 automatically transitions to the preview screen 130 of FIG. 36A after shooting is completed. Thereafter, the shooting trajectory 26C is calculated in response to a user operation, and the shooting trajectory 26C is displayed as shown in FIG. 36B. Furthermore, as shown in FIG. 36C, the missing camera posture 23 that is not covered by the shooting trajectory 26C is highlighted. Thereafter, the user issues an instruction to execute reshooting, which allows the reshooting to proceed.

[0163] 37 shows an example of processing performed by the UI device 2 (display system 48, control system 42) when image capture ends. When image capture ends, the display system 48 automatically transitions to the preview screen 130 in step S300, and displays the camera attitude, for example, as shown in FIG. 36A.

[0164] In step S323, the display system 48 performs processing to create a shooting trajectory 26C in response to a user input. For example, in response to an operation by the user to specify one or more missing camera postures 23, the display system 48 creates a shooting trajectory 26C as a course for shooting at the missing camera postures 23. Alternatively, in response to an operation by the user to draw a trajectory on the screen, the display system 48 may generate a shooting trajectory 26C that follows the trajectory.

[0165] Furthermore, in step S324, the display system 48 determines the missing camera poses 23 (and the abnormal camera poses 21) that cannot be photographed by the photographing trajectory 26C. Then, in step S325, the display system 48 highlights the missing camera poses 23 that cannot be photographed by the photographing trajectory 26C.

[0166] The display system 48 waits for a user operation in step S326. If the user performs an operation to re-create the shooting trajectory 26, the display system 48 returns to step S323.

[0167] If the user performs an operation to execute reshooting, the display system 48 proceeds to step S327, and updates the display to indicate that reshooting is to be performed. The control system 42 also transmits a control signal to the drone 10, instructing the drone 10 to follow the selected shooting trajectory 26C and execute reshooting.

[0168] As described above, when shooting is completed, a shooting trajectory 26 is generated corresponding to the missing camera pose 23 in the series of shooting. Alternatively, a shooting trajectory 26 is generated for the missing camera pose 23 selected by the user. Then, the user can arbitrarily perform reshooting using these shooting trajectories 26.

[0169] 7. Example of Calculation of Camera Attitude A specific example of the camera attitude determination process in step S30 of FIG. 23 will be described.

[0170] First, Fig. 38 shows an example in which a shooting trajectory 26 is input to the camera posture determination process (step S33 in Fig. 23). Fig. 38 shows the set shooting trajectory 26 and its diagram. It also shows the actual shooting trajectory 26' at the time of shooting and the estimated camera posture 20, and further shows a diagram in which these two are superimposed. As a result of superimposing the positions in this way, one estimated camera posture 20 deviates significantly from the expected camera posture (expected camera posture 27), and it is possible to determine that this estimated camera posture 20 is an abnormal camera posture 21.

[0171] Figure 39 shows steps S24, S33, and S30 of Figure 23. In step S33, the shooting trajectory 26 and planned camera posture 27 are input to the camera posture determination process. The estimated camera posture 20 stored in step S24 is also input to the camera posture determination process.

[0172] In the camera posture determination process of step S30, the incremental synthesis calculation system 45 first performs a positioning process in step S33a to align the planned camera posture 27 with the estimated camera posture 20. In this case, translation and rotation matrices for the positioning are algebraically determined from the posture correspondence information. By using RANSAC (RANdom Sample Consensus), the accuracy of the positioning calculation can be improved by excluding estimated camera postures 20 that deviate significantly from the planned camera posture 27 on the shooting trajectory 26.

[0173] In the abnormal camera pose detection in step S30b, the incremental synthesis calculation system 45 detects an estimated camera pose 20 that deviates significantly from the planned camera pose 27 from the aligned planned camera pose 27 and estimated camera pose 20, and sets it as an abnormal camera pose 21.

[0174] Next, a specific example of the camera attitude determination process performed in step S30 of FIG. 23 when the shooting trajectory 26 is not input will be described.

[0175] 40 shows an estimated camera pose 20 and a normal camera pose 20av obtained from the movement tendencies of a plurality of estimated camera poses 20. An estimated camera pose 20 that deviates significantly from the normal camera pose 20av obtained from the movement tendencies of a plurality of estimated camera poses 20 can be determined to be an abnormal camera pose 21.

[0176] Fig. 41 shows an example of the camera attitude determination process in step S30 in Fig. 23. In step S30c, the incremental synthesis calculation system 45 selects viewpoint positions before and after the shooting viewpoint. That is, for a certain estimated camera attitude 20, for example, estimated camera attitudes 20 at one or more previous shooting timings and estimated camera attitudes 20 at one or more subsequent shooting timings are selected as processing targets.

[0177] In step S30d, the incremental synthesis calculation system 45 calculates the normal camera pose 20av from the statistics of the viewpoint position. That is, it calculates the normal camera pose 20av estimated from the movement tendency of the front and rear estimated camera poses 20. For example, in the case of a circular shooting trajectory as shown in Figure 40, the front and rear estimated camera poses 20 are fitted to a circular trajectory model, and the normal camera pose 20av is obtained on that trajectory model. Note that if the shooting trajectory is not circular but linear, it is sufficient to fit it to a linear trajectory model. In other words, it is sufficient to apply a trajectory model that matches the shooting trajectory according to the shape of the subject.

[0178] In step S30e, the incremental synthesis calculation system 45 compares the difference between the normal camera pose and the estimated camera pose 20. Specifically, it is sufficient to detect the degree to which the target estimated camera pose 20 deviates from the above-mentioned trajectory model. In step S30f, if the comparison result is greater than a threshold, the incremental synthesis calculation system 45 determines that the estimated camera pose 20 is an abnormal camera pose 21.

[0179] That is, the above processing refers to the estimated camera pose 20 at viewpoint positions before and after a certain shooting viewpoint, and if the estimated camera pose 20 at a certain shooting viewpoint deviates significantly from the expected camera pose, it is determined to be an abnormal camera pose 21. For example, 10 consecutive frames from the past and future are selected, the average movement amount μ and standard deviation σ between each frame are calculated, and frames with movement amounts that deviate from the average by about 3σ are determined to be abnormal.

[0180] Furthermore, as incremental synthesis progresses, the accuracy of the previously estimated camera position may improve and no longer be abnormal. In such cases, it is a good idea to mark and display the updated position so that users can see it.

[0181] Next, another example of the camera attitude determination process performed in step S30 of Fig. 23 when there is no input of the shooting trajectory 26 will be described. Fig. 42 shows an example in which the above-mentioned high-precision region 28 and low-precision region 29 are displayed in a 3D model image 131 displayed on the preview screen 130. In addition, the estimated camera attitude 20 and the insufficient camera attitude 23 are displayed around the 3D model image 131.

[0182] For example, the following elements, or a combination thereof, serve as indices (accuracy indices) for determining the accuracy of each region of the 3D model, and high-accuracy regions 28 and low-accuracy regions 29 are determined. Whether or not a certain region in the shape of the 3D object 100, which is the subject, has been photographed as two or more images in the first place Amount of parallax between images Number of images photographing the same region Light angle of the image Shooting distance

[0183] The area determined to be a low-accuracy area 29 based on the accuracy index described above is an area where more images should be taken to increase the number of images, or an image taken in a more appropriate pose is required. In other words, the insufficient camera pose 23 can be calculated based on the accuracy of each area of ​​the 3D model or the accuracy index described above that calculates the accuracy.

[0184] The following are conceivable specific examples of calculating the missing camera pose 23: Calculating the missing camera pose so that the image is captured from at least two cameras. Calculating the missing camera pose so that the image is captured from at least two cameras with a parallax of at least a certain amount. Calculating the missing camera pose so that the image is captured from at least two cameras with the cameras facing the subject directly at a certain amount.

[0185] A flowchart of the incremental synthesis calculation system 45 when performing such processing is shown in Figure 43. This is a modified example of the incremental synthesis calculation system 45 shown in Figure 23. Note that the same processing as in Figure 23 is given the same step number to avoid duplicate explanation, and steps S40, S41, and S42 will be mainly explained here.

[0186] In step S40, the incremental synthesis calculation system 45 performs shape data generation processing based on the point cloud data obtained during camera posture estimation. Specifically, it performs noise reduction, mesh generation, and the like.

[0187] In step S41, the incremental synthesis calculation system 45 calculates an accuracy index. To this end, the shape data of the 3D model calculated in step S40 and the estimated camera pose 20 calculated in step S24 are input, and the accuracy index is calculated.

[0188] In step S42, the incremental synthesis calculation system 45 calculates the missing camera pose 23 based on the accuracy index. The calculated missing camera pose 23 becomes one of the pieces of information transmitted in step S28. That is, in step S28, the information transmission system 46 transmits information on the missing camera pose 23 determined in the above-described process in addition to the point cloud data of step S27 and the estimated camera pose 20 of step S24.

[0189] Figure 44 is a modified example of Figure 43. In the example of Figure 44, shape data generation processing is performed in step S44 from the point cloud data obtained in step S27. As a result, more precise shape data is obtained and input into the calculation of the accuracy index in step S41. In addition, the intermediate data in step S32 that saved the processing results of step S26 and the point cloud data during the camera pose estimation in step S31 are also input for the calculation of the accuracy index in step S41. As a result, a more diverse or more precise accuracy index can be obtained, and the missing camera pose 23 is calculated accordingly in step S42.

[0190] In step S28, the information transmission system 46 transmits the point cloud data of step S27, the estimated camera pose 20 of step S24, and information on the missing camera pose 23 determined in the above process.

[0191] 8. Summary and Modifications According to the above embodiment, the following effects can be obtained.

[0192] In this embodiment, it is assumed that 3D modeling is performed to generate a 3D model of a 3D object 100 using multiple images of the 3D object 100 captured at multiple different camera poses. In this case, one or multiple information processing devices execute a camera pose determination process (S30) that determines whether each of the multiple camera poses is suitable for 3D model generation, and a presentation information generation process (S35, S36, S37) that generates presentation information for the user based on the results of the camera pose determination process. After capturing an image, the user wants to confirm whether the camera 11's shooting pose was appropriate for 3D model generation. However, even by referring to the camera position information output during incremental synthesis, for example, it is not easy for the user to determine whether the image was captured at an appropriate shooting pose. Therefore, the estimated camera pose 20, which is estimated from the actual camera pose during capture, is compared with the desired pose to determine whether the pose is appropriate, and the result is displayed. This allows the user to easily recognize whether the actual capture was appropriate.

[0193] In the embodiment, while the 3D object 100 is being photographed, the photographed images are sequentially and incrementally processed to generate a 3D model, thereby forming the 3D model portion by portion during the photographing. By inputting sequentially photographed images into the incremental composition calculation system 45 during the photographing and compositing the 3D model, it is possible to generate a 3D model that grows after the photographing starts, as described in Fig. 6. This is also a process in which the entire 3D model is generated when the photographing is completed.

[0194] In the embodiment, processing is performed to present the generated 3D model to the user while shooting the 3D object 100. When a 3D model is generated during shooting using incremental synthesis, the growing 3D model is presented to the user as a 3D model image 131 on a preview screen 130. This allows the user to check the progress of shooting and whether any shots have been missed while on location, etc.

[0195] In the embodiment, an example of the presentation information generation process is given in which presentation information is generated to display a generated 3D model along with multiple camera poses used to capture images of the 3D model in accordance with the results of the camera pose determination process. For example, when a 3D model image 131 is displayed on a preview screen 130 as shown in FIG. 21 , the camera poses are displayed using frustum. At this time, each frustum has the meaning of an estimated camera pose 20, an abnormal camera pose 21, an acceptable camera pose 22, an insufficient camera pose 23, or the like, depending on the display mode. By executing such a display, the user can intuitively recognize what kind of pose the abnormal camera pose 21 or the insufficient camera pose 23 is relative to the 3D object 100.

[0196] In the embodiment, while photographing the 3D object 100, a camera posture determination process and a presentation information generation process for the photograph are executed, and abnormal camera postures 21 among the camera postures photographed up to that point and insufficient camera postures 23, which are camera postures recommended for photographing, are displayed (see FIGS. 26, 28, 29, and 30). For example, at each point during photographing, it is determined whether each of the estimated camera postures up to that point is appropriate, and the abnormal camera postures 21 and insufficient camera postures 23 are displayed as the determination results. This allows the user to check the appropriateness of the camera postures up to that point while photographing. Note that instead of displaying both the abnormal camera posture 21 and the insufficient camera posture 23, only one of them may be displayed.

[0197] In the embodiment, an example has been given in which, during shooting, a camera attitude determination process and a presentation information generation process for the shooting are executed, an abnormal camera attitude 21 or an insufficient camera attitude 23 among the camera attitudes shot up to that point is displayed, and control processing is executed in response to a user instruction to reshoot (see FIGS. 26, 28, 29, and 30). For example, when a user identifies an abnormal camera attitude 21 or an insufficient camera attitude 23 during shooting, the user can instruct the system to reshoot. This allows for efficient reshooting. For example, when a mobile system 40 such as a drone 10 is used, control processing in response to reshooting includes transmitting a shooting trajectory 26 (e.g., a flight plan) that has been modified to take the reshoot into account. For example, when a person is shooting with a handheld camera 11, processing such as displaying a shooting guide leading to the insufficient camera attitude 23 is performed.

[0198] In the embodiment, an example has been given in which, during shooting, a camera attitude determination process and a presentation information generation process for the shooting are executed, and abnormal camera attitudes 21 or insufficient camera attitudes 23 among the camera attitudes shot up to that point are displayed, along with a shooting trajectory 26 for reshooting (see FIGS. 26, 28, 29, and 30). This allows the user to check the shooting trajectory and decide whether or not to take a reshoot.

[0199] In the embodiment, an example has been given in which, during shooting, a camera posture determination process and a presentation information generation process for the shooting are executed, and abnormal camera postures 21 or insufficient camera postures 23 up to that point, time information related to reshooting, and information on changes in accuracy of the 3D model due to reshooting are displayed (see FIGS. 26C and 27). For example, during shooting, the user can check the abnormal camera postures 21 and insufficient camera postures 23, and time information related to reshooting and information on changes in accuracy of the 3D model due to reshooting are displayed. This makes it easier for the user to decide whether to reshoot.

[0200] In the embodiment, an example has been given in which, after photographing the 3D object 100, the abnormal camera pose 21 or the missing camera pose 23 is displayed based on the camera pose determination process and the presentation information generation process for the photographing (see FIGS. 31 to 37). By enabling the user to check the abnormal camera pose 21 or the missing camera pose 23 after photographing, the user can check immediately after photographing at the photographing site, and if an abnormal camera pose is found, the user can immediately proceed with re-photographing. Note that even after photographing, it is also possible to display either the abnormal camera pose 21 or the missing camera pose 23 rather than displaying both.

[0201] In the embodiment, an example has been given in which the abnormal camera posture 21 or the insufficient camera posture 23 is displayed after shooting, and control processing is executed in response to a user instruction to reshoot (see FIGS. 31 to 37). By allowing the user to instruct reshooting when they confirm the abnormal camera posture 21 or the insufficient camera posture 23 after shooting, reshooting can be performed immediately after shooting at the shooting site, thereby making the entire shooting operation more efficient.

[0202] In the embodiment, an example has been given in which the abnormal camera posture 21 or insufficient camera posture 23 and the shooting trajectory 26 for reshooting are displayed after shooting (see FIGS. 32, 34, and 35). For example, after shooting, the abnormal camera posture 21 or insufficient camera posture 23 is presented to the user, and a shooting trajectory 26 (26A, 26B) for reshooting that takes these into consideration is displayed. This allows the user to check the shooting trajectory 26 for reshooting and then determine whether or not to reshoot.

[0203] In the embodiment, an example has been given in which the abnormal camera posture 21 or the missing camera posture 23 is displayed after shooting, and a process for generating a shooting trajectory 26C for reshooting is executed in response to a user operation (see FIGS. 33, 36, and 37). For example, after shooting, the user is allowed to recognize the abnormal camera posture 21 or the missing camera posture 23 and generate the shooting trajectory 26C for reshooting. This allows the user to determine for themselves whether or not reshooting is necessary for each missing camera posture 23, and then set the shooting trajectory 26C for reshooting.

[0204] In the embodiment, an example has been given in which missing camera postures 23 for which reshooting is not performed according to the shooting trajectory 26C generated in response to a user operation are highlighted (see FIGS. 33, 36, and 37). When the user arbitrarily sets the shooting trajectory 26, it is not necessarily possible to shoot for all missing camera postures 23. Therefore, missing camera postures 23 for which shooting is not possible are highlighted to make it easier for the user to check.

[0205] In the embodiment, the camera attitude determination process is exemplified by determining an abnormal camera attitude among the multiple camera attitudes that have taken images, or an insufficient camera attitude that is a camera attitude that is recommended for image capture, based on a comparison between the multiple camera attitudes that have taken images and the camera attitude according to the set image capture trajectory (see FIGS. 38 and 39). When an image capture trajectory 26 is set as a flight plan or the like, and an ideal camera estimate is set, an abnormal camera attitude 21 or an insufficient camera attitude 23 can be determined by comparing the estimated camera attitude with the ideal camera estimate.

[0206] In the embodiment, the camera attitude determination process uses a camera attitude determined from the movement tendency of the multiple camera attitudes that have taken images to determine an abnormal camera attitude among the multiple camera attitudes that have taken images, or an insufficient camera attitude that is a camera attitude that is recommended for image capture (see FIGS. 40 and 41). In the absence of information on the image capture trajectory 26, such as a flight plan, if the image capture attitude at a certain point in time is significantly different from the image capture attitude expected in terms of the image capture attitudes before and after that point, it can be considered an abnormal camera attitude 21. This also makes it possible to determine an insufficient camera attitude 23.

[0207] In the embodiment, the camera posture determination process exemplified a case in which an insufficient camera posture is determined based on the accuracy index for each region of a 3D model generated from multiple images captured at each shooting posture (see FIGS. 42, 43, and 44). By examining the accuracy score for each part of the generated 3D model, it can be determined that parts with a score indicating low accuracy require more shooting. Therefore, an insufficient camera posture 23 can be determined based on the score for each part of the 3D model.

[0208] In this embodiment, the information processing device 70 serving as the 3D model generation device 3 includes a control unit (CPU 71) that executes a camera attitude determination process that determines whether each of a plurality of camera attitudes used in image capture performed when generating a 3D model of the 3D object 100 using a plurality of images of the object captured at a plurality of different camera attitudes is in a state suitable for 3D model generation, and a presentation information generation process that generates presentation information for the user based on the results of the camera attitude determination process. This allows the user to easily check the appropriateness of the camera attitude during or after image capture, and allows for efficient re-shooting based on the results.

[0209] Note that the information processing device 70 serving as the 3D model generation device 3 may perform the camera attitude determination process, and the information processing device 70 serving as the UI device 2 may perform the presentation information generation process. Alternatively, the information processing device 70 serving as the UI device 2 may perform the camera attitude determination process and the presentation information generation process. In other words, the camera attitude determination process and the presentation information generation process may be performed by either the incremental synthesis calculation system 45 or the display system 48.

[0210] Furthermore, although the preview screen 130 is configured to display the estimated camera pose 20 and the like together with the 3D model image 131, the abnormal camera pose 21, the insufficient camera pose 23, and the like may be displayed without displaying the 3D model image 131. For this purpose, steps S25, S26, and S27 in FIG. 23 may not be executed during or immediately after shooting. This reduces the processing load. For example, assuming a system configuration such as that shown in FIGS. 12 and 13 , even if the device equipped with the incremental synthesis calculation system 45 is an information processing device 70 with relatively low processing power, such as a smartphone 12, it becomes easy to enable the user to check the camera pose during shooting, etc.

[0211] Furthermore, although the embodiment has been described with reference to a case where incremental 3D composition is performed, the technology of the present disclosure can also be applied when incremental 3D composition is not performed. For example, even when the captured images are collectively transmitted to a server device 14 or the like after capture and 3D modeling is performed, the camera posture determination process and presentation information generation process can be executed in parallel to present the abnormal camera posture 21 and insufficient camera posture 23 to the user. Even with this process, the user can check the appropriateness of each camera posture at an early stage after capture.

[0212] Furthermore, the camera posture determination process and presentation information generation process can be executed for multiple images that have been taken and have undergone 3D modeling, and abnormal camera postures 21 and insufficient camera postures 23 can be presented to the user. Such a display allows the user to analyze, for example, the relationship between the accuracy of the 3D model and the camera posture.

[0213] Furthermore, the technology disclosed herein is applicable to various 3D modeling techniques, not limited to photogrammetry.

[0214] The program of the embodiment is a program that causes a processor such as a CPU or DSP, or a device including these, to execute the processes shown in Figures 23, 28, 29, 30, 43, and 44. That is, the program of the embodiment is a program that causes the information processing device 70 to execute a camera attitude determination process that determines whether or not each of a plurality of camera attitudes used in photographing, which was performed when generating a 3D model of the 3D object 100 using a plurality of images obtained by photographing the 3D object 100 at a plurality of different camera attitudes, is in a state suitable for 3D model generation, and a presentation information generation process that generates presentation information for the user based on the result of the camera attitude determination process.

[0215] By using such a program, the 3D model generating device 3 and the UI device 2 according to the embodiment can be realized by various information processing devices 70.

[0216] The programs of the above-described embodiments can be pre-recorded on a hard disk drive (HDD) or a ROM in a microcomputer having a CPU, which serves as a recording medium built into a computer or other device. Such programs can also be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) discs, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memories, and memory cards. Such removable recording media can be provided as so-called packaged software. Such programs can be installed on a personal computer or the like from removable recording media, or downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0217] Furthermore, such a program is suitable for widely providing the 3D model generation device 3 and UI device 2 of the embodiment. For example, by downloading the program to a personal computer, a communication device, a mobile terminal device such as a smartphone or tablet, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the 3D model generation device 3 and UI device 2 of the present disclosure.

[0218] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0219] The present technology can also be configured as follows. (1) An information processing method, performed by one or more information processing devices, comprising: a camera attitude determination process for determining whether each of a plurality of camera attitudes used in photographing an object when generating a three-dimensional model of the object using a plurality of images of the object photographed at a plurality of different camera attitudes is suitable for generating a three-dimensional model; and a presentation information generation process for generating presentation information for a user based on the result of the camera attitude determination process. (2) The information processing method described in (1) above, in which a process for forming a three-dimensional model portion by portion during photographing of the object is performed by generating a three-dimensional model using photographed images sequentially and additionally as processing targets during photographing of the object. (3) The information processing method described in (2) above, in which a process for presenting the generated three-dimensional model to a user is performed during photographing of the object. (4) The information processing method described in any of (1) to (3) above, in which the presentation information generation process generates presentation information for displaying the generated three-dimensional model together with the plurality of camera attitudes used to photograph the three-dimensional model in accordance with the result of the camera attitude determination process. (5) The information processing method according to any one of (1) to (4) above, wherein, while photographing an object, the camera attitude determination process and the presentation information generation process for the photographing are executed to display at least one of an abnormal camera attitude among the camera attitudes photographed up to that point, or an insufficient camera attitude that is a camera attitude that is recommended for photographing. (6) The information processing method according to any one of (1) to (5) above, wherein, while photographing an object, the camera attitude determination process and the presentation information generation process for the photographing are executed to display at least one of an abnormal camera attitude among the camera attitudes photographed up to that point, or an insufficient camera attitude that is a camera attitude that is recommended for photographing, and executes control processing in accordance with a user instruction to reshoot.(7) The information processing method according to any one of (1) to (6) above, wherein, during photographing of an object, the camera attitude determination process and the presentation information generation process for the photographing are executed to display at least one of an abnormal camera attitude among the camera attitudes taken up to that point in time or an insufficient camera attitude that is a camera attitude that recommends photographing, and a photographing trajectory for reshooting. (8) The information processing method according to any one of (1) to (7) above, wherein, during photographing of an object, the camera attitude determination process and the presentation information generation process for the photographing are executed to display at least one of an abnormal camera attitude among the camera attitudes taken up to that point in time or an insufficient camera attitude that is a camera attitude that recommends photographing, and at least one of time information regarding reshooting or information on a change in accuracy of the three-dimensional model due to reshooting. (9) The information processing method according to any one of (1) to (8) above, after photographing of an object, at least one of an abnormal camera attitude among the multiple camera attitudes used in photographing or an insufficient camera attitude that is a camera attitude that recommends photographing, based on the camera attitude determination process and the presentation information generation process for the photographing. (10) The information processing method according to any one of (1) to (9) above, further comprising: after an object is photographed, displaying at least one of an abnormal camera posture among the plurality of camera postures at which the photograph was taken or an insufficient camera posture that is a camera posture that is recommended for photographing, based on the camera posture determination process and the presentation information generation process for the photographing, and executing control processing in accordance with a user instruction to reshoot. (11) The information processing method according to any one of (1) to (10) above, further comprising: after an object is photographed, displaying at least one of an abnormal camera posture among the plurality of camera postures at which the photograph was taken or an insufficient camera posture that is a camera posture that is recommended for photographing, and a photographing trajectory for reshooting, based on the camera posture determination process and the presentation information generation process for the photographing.(12) The information processing method according to any one of (1) to (11) above, further comprising: after an object is photographed, displaying at least one of an abnormal camera posture among the multiple camera postures that performed the photographing or an insufficient camera posture that is a camera posture that is recommended for photographing based on the camera posture determination process and the presentation information generation process for the photographing, and executing a process of generating a photographing trajectory for reshooting in response to a user operation. (13) The information processing method according to (12) above, further comprising: highlighting an insufficient camera posture that does not allow reshooting based on the photographing trajectory generated in response to a user operation. (14) The information processing method according to any one of (1) to (13) above, further comprising: determining an abnormal camera posture among the multiple camera postures that performed the photographing or an insufficient camera posture that is a camera posture that is recommended for photographing based on a comparison between the multiple camera postures that performed the photographing and a camera posture corresponding to a set photographing trajectory, (15) The information processing method according to any one of (1) to (13), wherein the camera attitude determination process determines an abnormal camera attitude among the multiple camera attitudes that took the images, or an insufficient camera attitude that is a camera attitude that is recommended for image capture, using a camera attitude determined from movement tendencies of the multiple camera attitudes that took the images. (16) The information processing method according to any one of (1) to (13), wherein the camera attitude determination process determines an insufficient camera attitude based on an accuracy index for each region of a three-dimensional model generated from multiple images obtained by image capture at each image capture attitude. (17) An information processing device comprising: a control unit that executes a camera attitude determination process that determines whether each of multiple camera attitudes in image capture taken when generating a three-dimensional model of an object using multiple images of the object captured at multiple different camera attitudes is in a state suitable for generating a three-dimensional model, and a presentation information generation process that generates presentation information to be presented to a user based on a result of the camera attitude determination process.(18) A program that causes an information processing device to execute a camera attitude determination process that determines whether each of a plurality of camera attitudes taken during photography, when generating a three-dimensional model of an object using a plurality of images of the object taken at a plurality of different camera attitudes, is in a state suitable for generating a three-dimensional model, and a presentation information generation process that generates presentation information for a user based on the results of the camera attitude determination process.

[0220] REFERENCE SIGNS LIST 1 Photographing device 2 UI device 3 3D model generating device 20 Estimated camera pose 21 Abnormal camera pose 22 Allowable camera pose 23 Missing camera pose 24 Current camera pose 26 Photographing trajectory 42 Control system 45 Incremental synthesis calculation system 48 Display system 51 Incremental synthesis calculation unit 52 Camera pose determination unit 53 Transmission information generation unit 54 Storage unit 70 Information processing device 71 CPU

Claims

1. An information processing method, performed by one or more information processing devices, which comprises a camera attitude determination process for determining whether or not each of a plurality of camera attitudes taken during photographing, when generating a three-dimensional model of an object using a plurality of images taken of the object with a plurality of different camera attitudes, is in a state suitable for generating a three-dimensional model, and a presentation information generation process for generating presentation information for a user based on the results of the camera attitude determination process.

2. The information processing method according to claim 1, further comprising the step of generating a three-dimensional model by sequentially and incrementally processing the captured images during the photographing of an object, thereby forming a three-dimensional model portion by portion during the photographing.

3. The information processing method according to claim 2, further comprising the step of: performing processing for presenting a generated three-dimensional model to a user while photographing an object.

4. The information processing method according to claim 1, wherein the presentation information generation process generates presentation information for displaying the generated three-dimensional model together with multiple camera postures at which the three-dimensional model was photographed, in accordance with the results of the camera posture determination process.

5. The information processing method according to claim 1, further comprising: executing the camera attitude determination process and the presentation information generation process for the photograph while photographing an object, and displaying at least one of an abnormal camera attitude among the camera attitudes photographed up to that point, or an insufficient camera attitude which is a camera attitude that is recommended for photographing.

6. The information processing method according to claim 1, further comprising: executing the camera attitude determination process and the presentation information generation process for the photograph during photographing of an object, displaying at least one of an abnormal camera attitude among the camera attitudes photographed up to that point, or an insufficient camera attitude which is a camera attitude that is recommended for photographing, and executing a control process in response to a user instruction to take a new photograph.

7. The information processing method according to claim 1, further comprising: executing the camera attitude determination process and the presentation information generation process for the photograph during photographing of an object, and displaying at least one of an abnormal camera attitude among the camera attitudes photographed up to that point, or an insufficient camera attitude which is a camera attitude that is recommended for photographing, and a photographing trajectory for reshooting.

8. The information processing method according to claim 1, wherein, during photographing of an object, the camera attitude determination process and the presentation information generation process for the photograph are executed to display at least one of abnormal camera attitudes among the camera attitudes photographed up to that point in time, or insufficient camera attitudes which are camera attitudes that are recommended for photographing, and at least one of time information regarding reshooting or information regarding changes in accuracy of the three-dimensional model due to reshooting.

9. The information processing method according to claim 1, further comprising: after photographing an object, displaying at least one of an abnormal camera posture among a plurality of camera postures used to photograph the object, or an insufficient camera posture which is a camera posture recommended for photographing, based on the camera posture determination process and the presentation information generation process for the photograph.

10. The information processing method according to claim 1, further comprising: after photographing an object, displaying at least one of an abnormal camera posture among a plurality of camera postures used to photograph the object, or an insufficient camera posture which is a camera posture recommended for photographing, based on the camera posture determination process and the presentation information generation process for the photographing, and executing a control process in response to a user instruction to take a new photograph.

11. The information processing method according to claim 1, further comprising: after photographing an object, displaying at least one of an abnormal camera posture among a plurality of camera postures used to photograph the object, or an insufficient camera posture which is a camera posture that is recommended for photographing, and a photographing trajectory for reshooting, based on the camera posture determination process and the presentation information generation process for the photographing.

12. The information processing method according to claim 1, further comprising: after photographing an object, displaying at least one of an abnormal camera posture among a plurality of camera postures used to photograph the object, or an insufficient camera posture which is a camera posture recommended for photographing, based on the camera posture determination process and the presentation information generation process for the photographing, and executing a process of generating a photographing trajectory for reshooting in response to user operation.

13. The information processing method according to claim 12, further comprising highlighting an insufficient camera posture for which reshooting is not performed according to a shooting trajectory generated in response to a user operation.

14. The information processing method according to claim 1, wherein the camera attitude determination process determines an abnormal camera attitude among the multiple camera attitudes that took the photograph, or an insufficient camera attitude that is a camera attitude that is recommended for photographing, based on a comparison between the multiple camera attitudes that took the photograph and a camera attitude corresponding to a set photographing trajectory.

15. The information processing method according to claim 1, wherein the camera attitude determination process uses a camera attitude calculated from the movement tendencies of the multiple camera attitudes that took the photographs to determine an abnormal camera attitude among the multiple camera attitudes that took the photographs, or an insufficient camera attitude that is a camera attitude that is recommended for photographing.

16. The information processing method according to claim 1, wherein the camera attitude determination process determines an insufficient camera attitude based on an accuracy index for each region of a three-dimensional model generated from a plurality of images obtained by photographing at each photographing attitude.

17. An information processing device having a control unit that executes a camera attitude determination process that determines whether or not each of a plurality of camera attitudes taken during shooting, when generating a three-dimensional model of an object using a plurality of images of the object taken with a plurality of different camera attitudes, is in a state suitable for generating a three-dimensional model, and a presentation information generation process that generates presentation information for a user based on the results of the camera attitude determination process.

18. A program that causes an information processing device to execute a camera attitude determination process that determines whether or not each of a plurality of camera attitudes taken during photographing, when generating a three-dimensional model of an object using a plurality of images taken of the object with a plurality of different camera attitudes, is in a state suitable for generating a three-dimensional model, and a presentation information generation process that generates presentation information for a user based on the results of the camera attitude determination process.

Images